[dpdk-dev] [PATCH 00/22] Add DLB2 PMD

[dpdk-dev] [PATCH 00/22] Add DLB2 PMD

[Timothy McDaniel]

The following patch series adds support for a new eventdev PMD. The DLB2
PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
hardware.
The DLB2 is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB2 hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB2 ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a
directed queue. Additionally, even though LDB ports may link multiple queues,
the number of queues that may be linked is limited by hardware.

While reviewing the code, please be aware that this PMD has full
control over the DLB2 hardware. Intel will be extending the DLB2 PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.
Note that the DLB2 hardware is a successor of the DLB hardware, and
as such is structured similarly, both in terms of code layout and
implementation.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Depends-on: patch-77466 ("eventdev: add PCI probe named convenience function")
Depends-on: series-12160 ("Eventdev ABI changes")
Depends-on: patch-77460 ("eal: add umonitor umwait to x86 cpuflags")

Timothy McDaniel (22):
  event/dlb2: add meson build infrastructure
  event/dlb2: add dynamic logging
  event/dlb2: add private data structures and constants
  event/dlb2: add definitions shared with LKM or shared code
  event/dlb2: add inline functions
  event/dlb2: add probe
  event/dlb2: add xstats
  event/dlb2: add infos get and configure
  event/dlb2: add queue and port default conf
  event/dlb2: add queue setup
  event/dlb2: add port setup
  event/dlb2: add port link
  event/dlb2: add port unlink and port unlinks in progress
  event/dlb2: add eventdev start
  event/dlb2: add enqueue and its burst variants
  event/dlb2: add dequeue and its burst variants
  event/dlb2: add eventdev stop and close
  event/dlb2: add PMD's token pop public interface
  event/dlb2: add PMD self-tests
  event/dlb2: add queue and port release
  event/dlb2: add timeout ticks entry point
  doc: add new DLB2 eventdev driver to relnotes

 app/test/test_eventdev.c                          |    9 +
 config/rte_config.h                               |    7 +
 doc/guides/rel_notes/release_20_11.rst            |    5 +
 drivers/event/dlb2/dlb2.c                         | 4046 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c                   |   88 +
 drivers/event/dlb2/dlb2_iface.h                   |   75 +
 drivers/event/dlb2/dlb2_inline_fns.h              |   85 +
 drivers/event/dlb2/dlb2_log.h                     |   25 +
 drivers/event/dlb2/dlb2_priv.h                    |  619 +++
 drivers/event/dlb2/dlb2_selftest.c                | 1570 ++++++
 drivers/event/dlb2/dlb2_user.h                    |  883 +++
 drivers/event/dlb2/dlb2_xstats.c                  | 1269 +++++
 drivers/event/dlb2/meson.build                    |   16 +
 drivers/event/dlb2/pf/base/dlb2_hw_types.h        |  367 ++
 drivers/event/dlb2/pf/base/dlb2_mbox.h            |  596 ++
 drivers/event/dlb2/pf/base/dlb2_osdep.h           |  248 +
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h    |  447 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h      |  131 +
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h     |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h            | 2527 +++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c        | 6023 +++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h        | 1913 +++++++
 drivers/event/dlb2/pf/dlb2_main.c                 |  692 +++
 drivers/event/dlb2/pf/dlb2_main.h                 |  107 +
 drivers/event/dlb2/pf/dlb2_pf.c                   |  734 +++
 drivers/event/dlb2/rte_pmd_dlb2.c                 |   39 +
 drivers/event/dlb2/rte_pmd_dlb2.h                 |   59 +
 drivers/event/dlb2/rte_pmd_dlb2_event_version.map |    9 +
 drivers/event/meson.build                         |    4 +
 29 files changed, 22624 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
 create mode 100644 drivers/event/dlb2/dlb2_log.h
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c
 create mode 100644 drivers/event/dlb2/dlb2_user.h
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map

-- 
2.6.4

[dpdk-dev] [PATCH 01/22] event/dlb2: add meson build infrastructure

[Timothy McDaniel]

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux X86 platforms at this time.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 config/rte_config.h                               | 7 +++++++
 drivers/event/dlb2/meson.build                    | 7 +++++++
 drivers/event/dlb2/rte_pmd_dlb2_event_version.map | 3 +++
 drivers/event/meson.build                         | 4 ++++
 4 files changed, 21 insertions(+)
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map

diff --git a/config/rte_config.h b/config/rte_config.h
index 0bae630..fd1b3c3 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -131,4 +131,11 @@
 /* QEDE PMD defines */
 #define RTE_LIBRTE_QEDE_FW ""
 
+/* DLB2 defines */
+#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
+#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
+#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
+#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
+
 #endif /* _RTE_CONFIG_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
new file mode 100644
index 0000000..d4fd39f
--- /dev/null
+++ b/drivers/event/dlb2/meson.build
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2020 Intel Corporation
+
+sources = files(
+)
+
+deps += ['mbuf', 'mempool', 'ring', 'bus_vdev', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
new file mode 100644
index 0000000..299ae63
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
@@ -0,0 +1,3 @@
+DPDK_21.0 {
+	local: *;
+};
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index ebe76a7..f73fcb9 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -10,6 +10,10 @@ if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
 endif
+if ((dpdk_conf.has('RTE_ARCH_X86_64') or dpdk_conf.has('RTE_ARCH_X86')) and
+        is_linux)
+        drivers += 'dlb2'
+endif
 std_deps = ['eventdev', 'kvargs']
 config_flag_fmt = 'RTE_LIBRTE_@0@_EVENTDEV_PMD'
 driver_name_fmt = 'rte_pmd_@0@_event'
-- 
2.6.4

Re: [dpdk-dev] [PATCH 01/22] event/dlb2: add meson build infrastructure

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> To: Richardson, Bruce <bruce.richardson@intel.com>; Ray Kinsella
> <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 01/22] event/dlb2: add meson build infrastructure
> 
> Adds the meson build infrastructure, which includes
> compile-time constants in rte_config.h. DLB2 is
> only supported on Linux X86 platforms at this time.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  config/rte_config.h                               | 7 +++++++
>  drivers/event/dlb2/meson.build                    | 7 +++++++
>  drivers/event/dlb2/rte_pmd_dlb2_event_version.map | 3 +++
>  drivers/event/meson.build                         | 4 ++++
>  4 files changed, 21 insertions(+)
>  create mode 100644 drivers/event/dlb2/meson.build
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> 
> diff --git a/config/rte_config.h b/config/rte_config.h
> index 0bae630..fd1b3c3 100644
> --- a/config/rte_config.h
> +++ b/config/rte_config.h
> @@ -131,4 +131,11 @@
>  /* QEDE PMD defines */
>  #define RTE_LIBRTE_QEDE_FW ""
> 
> +/* DLB2 defines */
> +#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
> +#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
> +#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
> +#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
> +#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
> +
>  #endif /* _RTE_CONFIG_H_ */
> diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
> new file mode 100644
> index 0000000..d4fd39f
> --- /dev/null
> +++ b/drivers/event/dlb2/meson.build
> @@ -0,0 +1,7 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2019-2020 Intel Corporation
> +
> +sources = files(
> +)
> +
> +deps += ['mbuf', 'mempool', 'ring', 'bus_vdev', 'pci', 'bus_pci']

I don't believe the code in this series depends on bus_vdev, is that right?

With that resolved (if need be):
Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v2 00/22] Add DLB2 PMD

[Timothy McDaniel]

The following patch series adds support for a new eventdev PMD. The DLB2
PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
hardware.
The DLB2 is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB2 hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB2 ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a
directed queue. Additionally, even though LDB ports may link multiple queues,
the number of queues that may be linked is limited by hardware.

While reviewing the code, please be aware that this PMD has full
control over the DLB2 hardware. Intel will be extending the DLB2 PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.
Note that the DLB2 hardware is a successor of the DLB hardware, and
as such is structured similarly, both in terms of code layout and
implementation.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Changes since V1:
- implement changes requested in code reviews by Gage Eads and Mike Chen
- fix a memzone leak
- convert to use eal rte-cpuflags patch from Liang Ma

Known Issues:
- the documentation contained in dlb2.rst is not complete, and should be
updated to include command line parameters (class of service, etc ...). 

Depends-on: patch-79539 ("eal: add new x86 cpuid support for WAITPKG")

Timothy McDaniel (22):
  event/dlb2: add documentation and meson build infrastructure
  event/dlb2: add dynamic logging
  event/dlb2: add private data structures and constants
  event/dlb2: add definitions shared with LKM or shared code
  event/dlb2: add inline functions
  event/dlb2: add probe
  event/dlb2: add xstats
  event/dlb2: add infos get and configure
  event/dlb2: add queue and port default conf
  event/dlb2: add queue setup
  event/dlb2: add port setup
  event/dlb2: add port link
  event/dlb2: add port unlink and port unlinks in progress
  event/dlb2: add eventdev start
  event/dlb2: add enqueue and its burst variants
  event/dlb2: add dequeue and its burst variants
  event/dlb2: add eventdev stop and close
  event/dlb2: add PMD's token pop public interface
  event/dlb2: add PMD self-tests
  event/dlb2: add queue and port release
  event/dlb2: add timeout ticks entry point
  doc: add new DLB2 eventdev driver to relnotes

 MAINTAINERS                                       |    5 +
 app/test/test_eventdev.c                          |    7 +
 config/rte_config.h                               |    7 +
 doc/api/doxy-api-index.md                         |    1 +
 doc/guides/eventdevs/dlb2.rst                     |  330 ++
 doc/guides/eventdevs/index.rst                    |    1 +
 doc/guides/rel_notes/release_20_11.rst            |    5 +
 drivers/event/dlb2/dlb2.c                         | 3955 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c                   |   74 +
 drivers/event/dlb2/dlb2_iface.h                   |   75 +
 drivers/event/dlb2/dlb2_inline_fns.h              |   81 +
 drivers/event/dlb2/dlb2_log.h                     |   25 +
 drivers/event/dlb2/dlb2_priv.h                    |  578 ++
 drivers/event/dlb2/dlb2_selftest.c                | 1570 ++++++
 drivers/event/dlb2/dlb2_user.h                    |  679 +++
 drivers/event/dlb2/dlb2_xstats.c                  | 1240 +++++
 drivers/event/dlb2/meson.build                    |   16 +
 drivers/event/dlb2/pf/base/dlb2_hw_types.h        |  367 ++
 drivers/event/dlb2/pf/base/dlb2_mbox.h            |  596 ++
 drivers/event/dlb2/pf/base/dlb2_osdep.h           |  247 +
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h    |  440 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h      |  131 +
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h     |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h            | 2527 +++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c        | 6024 +++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h        | 1913 +++++++
 drivers/event/dlb2/pf/dlb2_main.c                 |  679 +++
 drivers/event/dlb2/pf/dlb2_main.h                 |  104 +
 drivers/event/dlb2/pf/dlb2_pf.c                   |  729 +++
 drivers/event/dlb2/rte_pmd_dlb2.c                 |   39 +
 drivers/event/dlb2/rte_pmd_dlb2.h                 |   68 +
 drivers/event/dlb2/rte_pmd_dlb2_event_version.map |    9 +
 drivers/event/meson.build                         |    3 +
 33 files changed, 22556 insertions(+)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
 create mode 100644 drivers/event/dlb2/dlb2_log.h
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c
 create mode 100644 drivers/event/dlb2/dlb2_user.h
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map

-- 
2.6.4

[dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[Timothy McDaniel]

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux X86 platforms at this time.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 config/rte_config.h                               |   7 +
 doc/guides/eventdevs/dlb2.rst                     | 330 ++++++++++++++++++++++
 doc/guides/eventdevs/index.rst                    |   1 +
 drivers/event/dlb2/meson.build                    |   7 +
 drivers/event/dlb2/rte_pmd_dlb2_event_version.map |   3 +
 drivers/event/meson.build                         |   3 +
 6 files changed, 351 insertions(+)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map

diff --git a/config/rte_config.h b/config/rte_config.h
index 0bae630..fd1b3c3 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -131,4 +131,11 @@
 /* QEDE PMD defines */
 #define RTE_LIBRTE_QEDE_FW ""
 
+/* DLB2 defines */
+#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
+#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
+#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
+#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
+
 #endif /* _RTE_CONFIG_H_ */
diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
new file mode 100644
index 0000000..fa8d6dd
--- /dev/null
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -0,0 +1,330 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Driver for the Intel® Dynamic Load Balancer (DLB2)
+==================================================
+
+The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
+
+Prerequisites
+-------------
+
+Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
+the basic DPDK environment.
+
+Configuration
+-------------
+
+The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
+device access. To use this operation mode, the PCIe PF device must be bound
+to a DPDK-compatible VFIO driver, such as vfio-pci.
+
+Eventdev API Notes
+------------------
+
+The DLB2 provides the functions of a DPDK event device; specifically, it
+supports atomic, ordered, and parallel scheduling events from queues to ports.
+However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
+features are abstracted by the PMD such as directed ports.
+
+In general the dlb PMD is designed for ease-of-use and does not require a
+detailed understanding of the hardware, but these details are important when
+writing high-performance code. This section describes the places where the
+eventdev API and DLB2 misalign.
+
+Scheduling Domain Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are 32 scheduling domainis the DLB2.
+When one is configured, it allocates load-balanced and
+directed queues, ports, credits, and other hardware resources. Some
+resource allocations are user-controlled -- the number of queues, for example
+-- and others, like credit pools (one directed and one load-balanced pool per
+scheduling domain), are not.
+
+The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
+setup argument and the per-port ``new_event_threshold`` argument apply as
+defined in the eventdev header file. The limit is applied to all enqueues,
+regardless of whether it will consume a directed or load-balanced credit.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
+Flow ID
+~~~~~~~
+
+The flow ID field is preserved in the event when it is scheduled in the
+DLB2.
+
+Hardware Credits
+~~~~~~~~~~~~~~~~
+
+DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
+event storage, with each unit of storage represented by a credit. A port spends
+a credit to enqueue an event, and hardware refills the ports with credits as the
+events are scheduled to ports. Refills come from credit pools, and each port is
+a member of a load-balanced credit pool and a directed credit pool. The
+load-balanced credits are used to enqueue to load-balanced queues, and directed
+credits are used for directed queues.
+
+A DLB2 eventdev contains one load-balanced and one directed credit pool. These
+pools' sizes are controlled by the nb_events_limit field in struct
+rte_event_dev_config. The load-balanced pool is sized to contain
+nb_events_limit credits, and the directed pool is sized to contain
+nb_events_limit/4 credits. The directed pool size can be overridden with the
+num_dir_credits vdev argument, like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,num_dir_credits=<value>
+
+This can be used if the default allocation is too low or too high for the
+specific application needs. The PMD also supports a vdev arg that limits the
+max_num_events reported by rte_event_dev_info_get():
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,max_num_events=<value>
+
+By default, max_num_events is reported as the total available load-balanced
+credits. If multiple DLB2-based applications are being used, it may be desirable
+to control how many load-balanced credits each application uses, particularly
+when application(s) are written to configure nb_events_limit equal to the
+reported max_num_events.
+
+Each port is a member of both credit pools. A port's credit allocation is
+defined by its low watermark, high watermark, and refill quanta. These three
+parameters are calculated by the dlb PMD like so:
+
+- The load-balanced high watermark is set to the port's enqueue_depth.
+  The directed high watermark is set to the minimum of the enqueue_depth and
+  the directed pool size divided by the total number of ports.
+- The refill quanta is set to half the high watermark.
+- The low watermark is set to the minimum of 16 and the refill quanta.
+
+When the eventdev is started, each port is pre-allocated a high watermark's
+worth of credits. For example, if an eventdev contains four ports with enqueue
+depths of 32 and a load-balanced credit pool size of 4096, each port will start
+with 32 load-balanced credits, and there will be 3968 credits available to
+replenish the ports. Thus, a single port is not capable of enqueueing up to the
+nb_events_limit (without any events being dequeued), since the other ports are
+retaining their initial credit allocation; in short, all ports must enqueue in
+order to reach the limit.
+
+If a port attempts to enqueue and has no credits available, the enqueue
+operation will fail and the application must retry the enqueue. Credits are
+replenished asynchronously by the DLB2 hardware.
+
+Software Credits
+~~~~~~~~~~~~~~~~
+
+The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
+credit scheme on top of the hardware credit scheme in order to comply with
+the per-port backpressure described in the eventdev API.
+
+The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
+credit when it enqueues to a queue, and credits are later replenished after the
+events are dequeued and released.
+
+In the software credit scheme, a credit is consumed when a new (.op =
+RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
+replenished when the event is released from the system (either explicitly with
+RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
+
+In this model, an event is "in the system" from its first enqueue into eventdev
+until it is last dequeued. If the event goes through multiple event queues, it
+is still considered "in the system" while a worker thread is processing it.
+
+A port will fail to enqueue if the number of events in the system exceeds its
+``new_event_threshold`` (specified at port setup time). A port will also fail
+to enqueue if it lacks enough hardware credits to enqueue; load-balanced
+credits are used to enqueue to a load-balanced queue, and directed credits are
+used to enqueue to a directed queue.
+
+The out-of-credit situations are typically transient, and an eventdev
+application using the DLB2 ought to retry its enqueues if they fail.
+If enqueue fails, DLB2 PMD sets rte_errno as follows:
+
+- -ENOSPC: Credit exhaustion (either hardware or software)
+- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
+
+Depending on the pipeline the application has constructed, it's possible to
+enter a credit deadlock scenario wherein the worker thread lacks the credit
+to enqueue an event, and it must dequeue an event before it can recover the
+credit. If the worker thread retries its enqueue indefinitely, it will not
+make forward progress. Such deadlock is possible if the application has event
+"loops", in which an event in dequeued from queue A and later enqueued back to
+queue A.
+
+Due to this, workers should stop retrying after a time, release the events it
+is attempting to enqueue, and dequeue more events. It is important that the
+worker release the events and don't simply set them aside to retry the enqueue
+again later, because the port has limited history list size (by default, twice
+the port's dequeue_depth).
+
+Priority
+~~~~~~~~
+
+The DLB2 supports event priority and per-port queue service priority, as
+described in the eventdev header file. The DLB2 does not support 'global' event
+queue priority established at queue creation time.
+
+DLB2 supports 8 event and queue service priority levels. For both priority
+types, the PMD uses the upper three bits of the priority field to determine the
+DLB2 priority, discarding the 5 least significant bits. The 5 least significant
+event priority bits are not preserved when an event is enqueued.
+
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
+
+Reconfiguration
+~~~~~~~~~~~~~~~
+
+The Eventdev API allows one to reconfigure a device, its ports, and its queues
+by first stopping the device, calling the configuration function(s), then
+restarting the device. The DLB2 does not support configuring an individual queue
+or port without first reconfiguring the entire device, however, so there are
+certain reconfiguration sequences that are valid in the eventdev API but not
+supported by the PMD.
+
+Specifically, the PMD supports the following configuration sequence:
+1. Configure and start the device
+2. Stop the device
+3. (Optional) Reconfigure the device
+4. (Optional) If step 3 is run:
+
+   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
+   b. The reconfigured port(s) lose their previous queue links.
+
+5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
+6. Restart the device. If the device is reconfigured in step 3 but one or more
+   of its ports or queues are not, the PMD will apply their previous
+   configuration (including port->queue links) at this time.
+
+The PMD does not support the following configuration sequences:
+1. Configure and start the device
+2. Stop the device
+3. Setup queue or setup port
+4. Start the device
+
+This sequence is not supported because the event device must be reconfigured
+before its ports or queues can be.
+
+Deferred Scheduling
+~~~~~~~~~~~~~~~~~~~
+
+The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
+dequeued event before returning from rte_event_dequeue_burst(). This frees the
+corresponding entries in the CQ, which enables the DLB2 to schedule more events
+to it.
+
+To support applications seeking finer-grained scheduling control -- for example
+deferring scheduling to get the best possible priority scheduling and
+load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
+the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
+call. This mode only applies to load-balanced event ports with dequeue depth of
+1.
+
+To enable deferred scheduling, use the defer_sched vdev argument like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,defer_sched=on
+
+Atomic Inflights Allocation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
+inflight event in a temporary buffer that is divided among load-balanced
+queues. If a queue's atomic buffer storage fills up, this can result in
+head-of-line-blocking. For example:
+
+- An LDB queue allocated N atomic buffer entries
+- All N entries are filled with events from flow X, which is pinned to CQ 0.
+
+Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
+scheduled. The likelihood of this case depends on the eventdev configuration,
+traffic behavior, event processing latency, potential for a worker to be
+interrupted or otherwise delayed, etc.
+
+By default, the PMD allocates 16 buffer entries for each load-balanced queue,
+which provides an even division across all 128 queues but potentially wastes
+buffer space (e.g. if not all queues are used, or aren't used for atomic
+scheduling).
+
+The PMD provides a dev arg to override the default per-queue allocation. To
+increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,atm_inflights=64
+
diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
index bb66a5e..55d5ba8 100644
--- a/doc/guides/eventdevs/index.rst
+++ b/doc/guides/eventdevs/index.rst
@@ -18,3 +18,4 @@ application through the eventdev API.
     octeontx
     octeontx2
     opdl
+    dlb2
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
new file mode 100644
index 0000000..54ba2c8
--- /dev/null
+++ b/drivers/event/dlb2/meson.build
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2020 Intel Corporation
+
+sources = files(
+)
+
+deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
new file mode 100644
index 0000000..299ae63
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
@@ -0,0 +1,3 @@
+DPDK_21.0 {
+	local: *;
+};
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index ebe76a7..01d3b59 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -10,6 +10,9 @@ if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
 endif
+if (dpdk_conf.has('RTE_ARCH_X86_64') and is_linux)
+        drivers += 'dlb2'
+endif
 std_deps = ['eventdev', 'kvargs']
 config_flag_fmt = 'RTE_LIBRTE_@0@_EVENTDEV_PMD'
 driver_name_fmt = 'rte_pmd_@0@_event'
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[Jerin Jacob]

On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> Adds the meson build infrastructure, which includes
> compile-time constants in rte_config.h. DLB2 is
> only supported on Linux X86 platforms at this time.
>
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> Reviewed-by: Gage Eads <gage.eads@intel.com>
> ---
>  config/rte_config.h                               |   7 +
>  doc/guides/eventdevs/dlb2.rst                     | 330 ++++++++++++++++++++++
>  doc/guides/eventdevs/index.rst                    |   1 +
>  drivers/event/dlb2/meson.build                    |   7 +
>  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |   3 +
>  drivers/event/meson.build                         |   3 +
>  6 files changed, 351 insertions(+)
>  create mode 100644 doc/guides/eventdevs/dlb2.rst
>  create mode 100644 drivers/event/dlb2/meson.build
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map
>
> diff --git a/config/rte_config.h b/config/rte_config.h
> index 0bae630..fd1b3c3 100644
> --- a/config/rte_config.h
> +++ b/config/rte_config.h
> @@ -131,4 +131,11 @@
>  /* QEDE PMD defines */
>  #define RTE_LIBRTE_QEDE_FW ""
>
> +/* DLB2 defines */
> +#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
> +#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
> +#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
> +#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
> +#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256

We are going way with GLOBAL compile time options for drivers.
I think,  we should move to driver-specific folder not pollute top
level config file.
@Richardson, Bruce  @Thomas Monjalon  Thoughts?


> +
>  #endif /* _RTE_CONFIG_H_ */
> diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
> new file mode 100644
> index 0000000..fa8d6dd
> --- /dev/null
> +++ b/doc/guides/eventdevs/dlb2.rst
> @@ -0,0 +1,330 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2020 Intel Corporation.
> +
> +Driver for the Intel® Dynamic Load Balancer (DLB2)
> +==================================================
> +
> +The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
> +
> +Prerequisites
> +-------------
> +
> +Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
> +the basic DPDK environment.
> +
> +Configuration
> +-------------
> +
> +The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
> +device access. To use this operation mode, the PCIe PF device must be bound
> +to a DPDK-compatible VFIO driver, such as vfio-pci.
> +
> +Eventdev API Notes
> +------------------
> +
> +The DLB2 provides the functions of a DPDK event device; specifically, it
> +supports atomic, ordered, and parallel scheduling events from queues to ports.
> +However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
> +features are abstracted by the PMD such as directed ports.
> +
> +In general the dlb PMD is designed for ease-of-use and does not require a
> +detailed understanding of the hardware, but these details are important when
> +writing high-performance code. This section describes the places where the
> +eventdev API and DLB2 misalign.
> +
> +Scheduling Domain Configuration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +There are 32 scheduling domainis the DLB2.
> +When one is configured, it allocates load-balanced and
> +directed queues, ports, credits, and other hardware resources. Some
> +resource allocations are user-controlled -- the number of queues, for example
> +-- and others, like credit pools (one directed and one load-balanced pool per
> +scheduling domain), are not.
> +
> +The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
> +setup argument and the per-port ``new_event_threshold`` argument apply as
> +defined in the eventdev header file. The limit is applied to all enqueues,
> +regardless of whether it will consume a directed or load-balanced credit.
> +
> +Load-balanced and Directed Ports
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
> +does not have the same concept, but it has a similar one: ports and queues that
> +are singly-linked (i.e. linked to a single queue or port, respectively).
> +
> +The ``rte_event_dev_info_get()`` function reports the number of available
> +event ports and queues (among other things). For the DLB2 PMD, max_event_ports
> +and max_event_queues report the number of available load-balanced ports and
> +queues, and max_single_link_event_port_queue_pairs reports the number of
> +available directed ports and queues.
> +
> +When a scheduling domain is created in ``rte_event_dev_configure()``, the user
> +specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
> +control the total number of ports (load-balanced and directed) and the number
> +of directed ports. Hence, the number of requested load-balanced ports is
> +``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
> +specifies the total number of queues (load-balanced and directed). The number
> +of directed queues comes from ``nb_single_link_event_port_queues``, since
> +directed ports and queues come in pairs.
> +
> +When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
> +whether it should be configured as a directed (the flag is set) or a
> +load-balanced (the flag is unset) port. Similarly, the
> +``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
> +whether it is a directed or load-balanced queue.
> +
> +Load-balanced ports can only be linked to load-balanced queues, and directed
> +ports can only be linked to directed queues. Furthermore, directed ports can
> +only be linked to a single directed queue (and vice versa), and that link
> +cannot change after the eventdev is started.
> +
> +The eventdev API does not have a directed scheduling type. To support directed
> +traffic, the dlb PMD detects when an event is being sent to a directed queue
> +and overrides its scheduling type. Note that the originally selected scheduling
> +type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
> +will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
> +port.
> +
> +Flow ID
> +~~~~~~~
> +
> +The flow ID field is preserved in the event when it is scheduled in the
> +DLB2.
> +
> +Hardware Credits
> +~~~~~~~~~~~~~~~~
> +
> +DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
> +event storage, with each unit of storage represented by a credit. A port spends
> +a credit to enqueue an event, and hardware refills the ports with credits as the
> +events are scheduled to ports. Refills come from credit pools, and each port is
> +a member of a load-balanced credit pool and a directed credit pool. The
> +load-balanced credits are used to enqueue to load-balanced queues, and directed
> +credits are used for directed queues.
> +
> +A DLB2 eventdev contains one load-balanced and one directed credit pool. These
> +pools' sizes are controlled by the nb_events_limit field in struct
> +rte_event_dev_config. The load-balanced pool is sized to contain
> +nb_events_limit credits, and the directed pool is sized to contain
> +nb_events_limit/4 credits. The directed pool size can be overridden with the
> +num_dir_credits vdev argument, like so:
> +
> +    .. code-block:: console
> +
> +       --vdev=dlb1_event,num_dir_credits=<value>
> +
> +This can be used if the default allocation is too low or too high for the
> +specific application needs. The PMD also supports a vdev arg that limits the
> +max_num_events reported by rte_event_dev_info_get():
> +
> +    .. code-block:: console
> +
> +       --vdev=dlb1_event,max_num_events=<value>
> +
> +By default, max_num_events is reported as the total available load-balanced
> +credits. If multiple DLB2-based applications are being used, it may be desirable
> +to control how many load-balanced credits each application uses, particularly
> +when application(s) are written to configure nb_events_limit equal to the
> +reported max_num_events.
> +
> +Each port is a member of both credit pools. A port's credit allocation is
> +defined by its low watermark, high watermark, and refill quanta. These three
> +parameters are calculated by the dlb PMD like so:
> +
> +- The load-balanced high watermark is set to the port's enqueue_depth.
> +  The directed high watermark is set to the minimum of the enqueue_depth and
> +  the directed pool size divided by the total number of ports.
> +- The refill quanta is set to half the high watermark.
> +- The low watermark is set to the minimum of 16 and the refill quanta.
> +
> +When the eventdev is started, each port is pre-allocated a high watermark's
> +worth of credits. For example, if an eventdev contains four ports with enqueue
> +depths of 32 and a load-balanced credit pool size of 4096, each port will start
> +with 32 load-balanced credits, and there will be 3968 credits available to
> +replenish the ports. Thus, a single port is not capable of enqueueing up to the
> +nb_events_limit (without any events being dequeued), since the other ports are
> +retaining their initial credit allocation; in short, all ports must enqueue in
> +order to reach the limit.
> +
> +If a port attempts to enqueue and has no credits available, the enqueue
> +operation will fail and the application must retry the enqueue. Credits are
> +replenished asynchronously by the DLB2 hardware.
> +
> +Software Credits
> +~~~~~~~~~~~~~~~~
> +
> +The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
> +credit scheme on top of the hardware credit scheme in order to comply with
> +the per-port backpressure described in the eventdev API.
> +
> +The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
> +credit when it enqueues to a queue, and credits are later replenished after the
> +events are dequeued and released.
> +
> +In the software credit scheme, a credit is consumed when a new (.op =
> +RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
> +replenished when the event is released from the system (either explicitly with
> +RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
> +
> +In this model, an event is "in the system" from its first enqueue into eventdev
> +until it is last dequeued. If the event goes through multiple event queues, it
> +is still considered "in the system" while a worker thread is processing it.
> +
> +A port will fail to enqueue if the number of events in the system exceeds its
> +``new_event_threshold`` (specified at port setup time). A port will also fail
> +to enqueue if it lacks enough hardware credits to enqueue; load-balanced
> +credits are used to enqueue to a load-balanced queue, and directed credits are
> +used to enqueue to a directed queue.
> +
> +The out-of-credit situations are typically transient, and an eventdev
> +application using the DLB2 ought to retry its enqueues if they fail.
> +If enqueue fails, DLB2 PMD sets rte_errno as follows:
> +
> +- -ENOSPC: Credit exhaustion (either hardware or software)
> +- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
> +
> +Depending on the pipeline the application has constructed, it's possible to
> +enter a credit deadlock scenario wherein the worker thread lacks the credit
> +to enqueue an event, and it must dequeue an event before it can recover the
> +credit. If the worker thread retries its enqueue indefinitely, it will not
> +make forward progress. Such deadlock is possible if the application has event
> +"loops", in which an event in dequeued from queue A and later enqueued back to
> +queue A.
> +
> +Due to this, workers should stop retrying after a time, release the events it
> +is attempting to enqueue, and dequeue more events. It is important that the
> +worker release the events and don't simply set them aside to retry the enqueue
> +again later, because the port has limited history list size (by default, twice
> +the port's dequeue_depth).
> +
> +Priority
> +~~~~~~~~
> +
> +The DLB2 supports event priority and per-port queue service priority, as
> +described in the eventdev header file. The DLB2 does not support 'global' event
> +queue priority established at queue creation time.
> +
> +DLB2 supports 8 event and queue service priority levels. For both priority
> +types, the PMD uses the upper three bits of the priority field to determine the
> +DLB2 priority, discarding the 5 least significant bits. The 5 least significant
> +event priority bits are not preserved when an event is enqueued.
> +
> +Load-Balanced Queues
> +~~~~~~~~~~~~~~~~~~~~
> +
> +A load-balanced queue can support atomic and ordered scheduling, or atomic and
> +unordered scheduling, but not atomic and unordered and ordered scheduling. A
> +queue's scheduling types are controlled by the event queue configuration.
> +
> +If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
> +``nb_atomic_order_sequences`` determines the supported scheduling types.
> +With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
> +and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
> +supported by scheduling those events as ordered events.  Note that when the
> +event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
> +``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
> +unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
> +
> +If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
> +dictates the queue's scheduling type.
> +
> +The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
> +queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
> +group is configured to contain either 1 queue with 1024 reorder entries, 2
> +queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
> +
> +When a load-balanced queue is created, the PMD will configure a new sequence
> +number group on-demand if num_sequence_numbers does not match a pre-existing
> +group with available reorder buffer entries. If all sequence number groups are
> +in use, no new group will be created and queue configuration will fail. (Note
> +that when the PMD is used with a virtual DLB2 device, it cannot change the
> +sequence number configuration.)
> +
> +The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
> +the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
> +load-balanced queues can use the full 16-bit flow ID range.
> +
> +Reconfiguration
> +~~~~~~~~~~~~~~~
> +
> +The Eventdev API allows one to reconfigure a device, its ports, and its queues
> +by first stopping the device, calling the configuration function(s), then
> +restarting the device. The DLB2 does not support configuring an individual queue
> +or port without first reconfiguring the entire device, however, so there are
> +certain reconfiguration sequences that are valid in the eventdev API but not
> +supported by the PMD.
> +
> +Specifically, the PMD supports the following configuration sequence:
> +1. Configure and start the device
> +2. Stop the device
> +3. (Optional) Reconfigure the device
> +4. (Optional) If step 3 is run:
> +
> +   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
> +   b. The reconfigured port(s) lose their previous queue links.
> +
> +5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
> +6. Restart the device. If the device is reconfigured in step 3 but one or more
> +   of its ports or queues are not, the PMD will apply their previous
> +   configuration (including port->queue links) at this time.
> +
> +The PMD does not support the following configuration sequences:
> +1. Configure and start the device
> +2. Stop the device
> +3. Setup queue or setup port
> +4. Start the device
> +
> +This sequence is not supported because the event device must be reconfigured
> +before its ports or queues can be.
> +
> +Deferred Scheduling
> +~~~~~~~~~~~~~~~~~~~
> +
> +The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
> +dequeued event before returning from rte_event_dequeue_burst(). This frees the
> +corresponding entries in the CQ, which enables the DLB2 to schedule more events
> +to it.
> +
> +To support applications seeking finer-grained scheduling control -- for example
> +deferring scheduling to get the best possible priority scheduling and
> +load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
> +the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
> +call. This mode only applies to load-balanced event ports with dequeue depth of
> +1.
> +
> +To enable deferred scheduling, use the defer_sched vdev argument like so:
> +
> +    .. code-block:: console
> +
> +       --vdev=dlb1_event,defer_sched=on
> +
> +Atomic Inflights Allocation
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
> +inflight event in a temporary buffer that is divided among load-balanced
> +queues. If a queue's atomic buffer storage fills up, this can result in
> +head-of-line-blocking. For example:
> +
> +- An LDB queue allocated N atomic buffer entries
> +- All N entries are filled with events from flow X, which is pinned to CQ 0.
> +
> +Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
> +scheduled. The likelihood of this case depends on the eventdev configuration,
> +traffic behavior, event processing latency, potential for a worker to be
> +interrupted or otherwise delayed, etc.
> +
> +By default, the PMD allocates 16 buffer entries for each load-balanced queue,
> +which provides an even division across all 128 queues but potentially wastes
> +buffer space (e.g. if not all queues are used, or aren't used for atomic
> +scheduling).
> +
> +The PMD provides a dev arg to override the default per-queue allocation. To
> +increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
> +
> +    .. code-block:: console
> +
> +       --vdev=dlb1_event,atm_inflights=64
> +


Please move the doc changes to whichever patch that introduces this change,


> diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
> index bb66a5e..55d5ba8 100644
> --- a/doc/guides/eventdevs/index.rst
> +++ b/doc/guides/eventdevs/index.rst
> @@ -18,3 +18,4 @@ application through the eventdev API.
>      octeontx
>      octeontx2
>      opdl
> +    dlb2
> diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
> new file mode 100644
> index 0000000..54ba2c8
> --- /dev/null
> +++ b/drivers/event/dlb2/meson.build
> @@ -0,0 +1,7 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2019-2020 Intel Corporation
> +
> +sources = files(
> +)
> +
> +deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
> diff --git a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> new file mode 100644
> index 0000000..299ae63
> --- /dev/null
> +++ b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> @@ -0,0 +1,3 @@
> +DPDK_21.0 {
> +       local: *;
> +};
> diff --git a/drivers/event/meson.build b/drivers/event/meson.build"
> index ebe76a7..01d3b59 100644
> --- a/drivers/event/meson.build
> +++ b/drivers/event/meson.build
> @@ -10,6 +10,9 @@ if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
>         dpdk_conf.has('RTE_ARCH_ARM64'))
>         drivers += 'octeontx'
>  endif
> +if (dpdk_conf.has('RTE_ARCH_X86_64') and is_linux)
> +        drivers += 'dlb2'
> +endif

Please add the message in "Content Skipped" section,
Reference: grep "reason" in drivers/vdpa/mlx5/meson.build





>  std_deps = ['eventdev', 'kvargs']
>  config_flag_fmt = 'RTE_LIBRTE_@0@_EVENTDEV_PMD'
>  driver_name_fmt = 'rte_pmd_@0@_event'
> --
> 2.6.4
>

Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[Bruce Richardson]

On Sun, Oct 18, 2020 at 02:18:32PM +0530, Jerin Jacob wrote:
> On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > Adds the meson build infrastructure, which includes
> > compile-time constants in rte_config.h. DLB2 is
> > only supported on Linux X86 platforms at this time.
> >
> > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > ---
> >  config/rte_config.h                               |   7 +
> >  doc/guides/eventdevs/dlb2.rst                     | 330 ++++++++++++++++++++++
> >  doc/guides/eventdevs/index.rst                    |   1 +
> >  drivers/event/dlb2/meson.build                    |   7 +
> >  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |   3 +
> >  drivers/event/meson.build                         |   3 +
> >  6 files changed, 351 insertions(+)
> >  create mode 100644 doc/guides/eventdevs/dlb2.rst
> >  create mode 100644 drivers/event/dlb2/meson.build
> >  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> >
> > diff --git a/config/rte_config.h b/config/rte_config.h
> > index 0bae630..fd1b3c3 100644
> > --- a/config/rte_config.h
> > +++ b/config/rte_config.h
> > @@ -131,4 +131,11 @@
> >  /* QEDE PMD defines */
> >  #define RTE_LIBRTE_QEDE_FW ""
> >
> > +/* DLB2 defines */
> > +#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
> > +#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
> > +#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
> > +#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
> > +#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
> 
> We are going way with GLOBAL compile time options for drivers.
> I think,  we should move to driver-specific folder not pollute top
> level config file.
> @Richardson, Bruce  @Thomas Monjalon  Thoughts?
>
Yes, we do need a better way to pass driver-specific build config options.
How likely is it that the user may want to tweak these values? If it's not
likely having them just as regular defines in the driver folder may be
better, though they are not particularly problematic in rte_config.h given
the number of other values already there.
 
> 
> > +
<snip>
> > --- a/drivers/event/meson.build
> > +++ b/drivers/event/meson.build
> > @@ -10,6 +10,9 @@ if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
> >         dpdk_conf.has('RTE_ARCH_ARM64'))
> >         drivers += 'octeontx'
> >  endif
> > +if (dpdk_conf.has('RTE_ARCH_X86_64') and is_linux)
> > +        drivers += 'dlb2'
> > +endif
> 
> Please add the message in "Content Skipped" section,
> Reference: grep "reason" in drivers/vdpa/mlx5/meson.build
> 

The octeontx case is also wrong in this file, IMHO. Rather than checking
things in the event level and adding things to the list, the list should
just be static. If something should be optionally compiled, then check the
conditions in the driver meson.build file itself and add "build=false" to
disable, setting "reason" to the cause of it being disabled. This keeps all
the logic about a driver in the other file, rather than someone having to
look in multiple places for why something is or isn't getting built
properly.

Regards,
/Bruce

Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[McDaniel, Timothy]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Monday, October 19, 2020 3:34 AM
> To: Jerin Jacob <jerinjacobk@gmail.com>
> Cc: McDaniel, Timothy <timothy.mcdaniel@intel.com>; Mcnamara, John
> <john.mcnamara@intel.com>; Kovacevic, Marko
> <Marko.Kovacevic@intel.com>; Ray Kinsella <mdr@ashroe.eu>; Neil Horman
> <nhorman@tuxdriver.com>; dpdk-dev <dev@dpdk.org>; Carrillo, Erik G
> <Erik.G.Carrillo@intel.com>; Eads, Gage <gage.eads@intel.com>; Van Haaren,
> Harry <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>;
> Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and
> meson build infrastructure
> 
> On Sun, Oct 18, 2020 at 02:18:32PM +0530, Jerin Jacob wrote:
> > On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
> > <timothy.mcdaniel@intel.com> wrote:
> > >
> > > Adds the meson build infrastructure, which includes
> > > compile-time constants in rte_config.h. DLB2 is
> > > only supported on Linux X86 platforms at this time.
> > >
> > > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > > ---
> > >  config/rte_config.h                               |   7 +
> > >  doc/guides/eventdevs/dlb2.rst                     | 330 ++++++++++++++++++++++
> > >  doc/guides/eventdevs/index.rst                    |   1 +
> > >  drivers/event/dlb2/meson.build                    |   7 +
> > >  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |   3 +
> > >  drivers/event/meson.build                         |   3 +
> > >  6 files changed, 351 insertions(+)
> > >  create mode 100644 doc/guides/eventdevs/dlb2.rst
> > >  create mode 100644 drivers/event/dlb2/meson.build
> > >  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> > >
> > > diff --git a/config/rte_config.h b/config/rte_config.h
> > > index 0bae630..fd1b3c3 100644
> > > --- a/config/rte_config.h
> > > +++ b/config/rte_config.h
> > > @@ -131,4 +131,11 @@
> > >  /* QEDE PMD defines */
> > >  #define RTE_LIBRTE_QEDE_FW ""
> > >
> > > +/* DLB2 defines */
> > > +#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
> > > +#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
> > > +#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
> > > +#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
> > > +#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
> >
> > We are going way with GLOBAL compile time options for drivers.
> > I think,  we should move to driver-specific folder not pollute top
> > level config file.
> > @Richardson, Bruce  @Thomas Monjalon  Thoughts?
> >
> Yes, we do need a better way to pass driver-specific build config options.
> How likely is it that the user may want to tweak these values? If it's not
> likely having them just as regular defines in the driver folder may be
> better, though they are not particularly problematic in rte_config.h given
> the number of other values already there.
> 
> >
> > > +
> <snip>
> > > --- a/drivers/event/meson.build
> > > +++ b/drivers/event/meson.build
> > > @@ -10,6 +10,9 @@ if not (toolchain == 'gcc' and
> cc.version().version_compare('<4.8.6') and
> > >         dpdk_conf.has('RTE_ARCH_ARM64'))
> > >         drivers += 'octeontx'
> > >  endif
> > > +if (dpdk_conf.has('RTE_ARCH_X86_64') and is_linux)
> > > +        drivers += 'dlb2'
> > > +endif
> >
> > Please add the message in "Content Skipped" section,
> > Reference: grep "reason" in drivers/vdpa/mlx5/meson.build
> >
> 
> The octeontx case is also wrong in this file, IMHO. Rather than checking
> things in the event level and adding things to the list, the list should
> just be static. If something should be optionally compiled, then check the
> conditions in the driver meson.build file itself and add "build=false" to
> disable, setting "reason" to the cause of it being disabled. This keeps all
> the logic about a driver in the other file, rather than someone having to
> look in multiple places for why something is or isn't getting built
> properly.
> 
> Regards,
> /Bruce

Due to time constraints, I would prefer to take these issues up in a future release.

Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[Thomas Monjalon]

20/10/2020 17:17, McDaniel, Timothy:
> From: Bruce Richardson <bruce.richardson@intel.com>
> > On Sun, Oct 18, 2020 at 02:18:32PM +0530, Jerin Jacob wrote:
> > > On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
> > > <timothy.mcdaniel@intel.com> wrote:
> > > >
> > > > Adds the meson build infrastructure, which includes
> > > > compile-time constants in rte_config.h. DLB2 is
> > > > only supported on Linux X86 platforms at this time.
> > > >
> > > > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > > > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > > > ---
> > > > --- a/drivers/event/meson.build
> > > > +++ b/drivers/event/meson.build
> > > > @@ -10,6 +10,9 @@ if not (toolchain == 'gcc' and
> > cc.version().version_compare('<4.8.6') and
> > > >         dpdk_conf.has('RTE_ARCH_ARM64'))
> > > >         drivers += 'octeontx'
> > > >  endif
> > > > +if (dpdk_conf.has('RTE_ARCH_X86_64') and is_linux)
> > > > +        drivers += 'dlb2'
> > > > +endif
> > >
> > > Please add the message in "Content Skipped" section,
> > > Reference: grep "reason" in drivers/vdpa/mlx5/meson.build
> > >
> > 
> > The octeontx case is also wrong in this file, IMHO. Rather than checking
> > things in the event level and adding things to the list, the list should
> > just be static. If something should be optionally compiled, then check the
> > conditions in the driver meson.build file itself and add "build=false" to
> > disable, setting "reason" to the cause of it being disabled. This keeps all
> > the logic about a driver in the other file, rather than someone having to
> > look in multiple places for why something is or isn't getting built
> > properly.
> > 
> > Regards,
> > /Bruce
> 
> Due to time constraints, I would prefer to take these issues up in a future release.

If you have strong time constraints to complete everything,
then it is better to postpone the feature to the next release.

Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[McDaniel, Timothy]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday, October 20, 2020 10:21 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>; Eads, Gage
> <gage.eads@intel.com>
> Cc: Richardson, Bruce <bruce.richardson@intel.com>; Jerin Jacob
> <jerinjacobk@gmail.com>; Mcnamara, John <john.mcnamara@intel.com>;
> Kovacevic, Marko <marko.kovacevic@intel.com>; Ray Kinsella
> <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>; dpdk-dev
> <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>;
> david.marchand@redhat.com
> Subject: Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and
> meson build infrastructure
> 
> 20/10/2020 17:17, McDaniel, Timothy:
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > > On Sun, Oct 18, 2020 at 02:18:32PM +0530, Jerin Jacob wrote:
> > > > On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
> > > > <timothy.mcdaniel@intel.com> wrote:
> > > > >
> > > > > Adds the meson build infrastructure, which includes
> > > > > compile-time constants in rte_config.h. DLB2 is
> > > > > only supported on Linux X86 platforms at this time.
> > > > >
> > > > > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > > > > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > > > > ---
> > > > > --- a/drivers/event/meson.build
> > > > > +++ b/drivers/event/meson.build
> > > > > @@ -10,6 +10,9 @@ if not (toolchain == 'gcc' and
> > > cc.version().version_compare('<4.8.6') and
> > > > >         dpdk_conf.has('RTE_ARCH_ARM64'))
> > > > >         drivers += 'octeontx'
> > > > >  endif
> > > > > +if (dpdk_conf.has('RTE_ARCH_X86_64') and is_linux)
> > > > > +        drivers += 'dlb2'
> > > > > +endif
> > > >
> > > > Please add the message in "Content Skipped" section,
> > > > Reference: grep "reason" in drivers/vdpa/mlx5/meson.build
> > > >
> > >
> > > The octeontx case is also wrong in this file, IMHO. Rather than checking
> > > things in the event level and adding things to the list, the list should
> > > just be static. If something should be optionally compiled, then check the
> > > conditions in the driver meson.build file itself and add "build=false" to
> > > disable, setting "reason" to the cause of it being disabled. This keeps all
> > > the logic about a driver in the other file, rather than someone having to
> > > look in multiple places for why something is or isn't getting built
> > > properly.
> > >
> > > Regards,
> > > /Bruce
> >
> > Due to time constraints, I would prefer to take these issues up in a future
> release.
> 
> If you have strong time constraints to complete everything,
> then it is better to postpone the feature to the next release.
> 

You read too much into my comment. 
My point is that there does not seem to be community consensus on this issue,
so why are we debating it at this point in the release cycle.  There are plenty of examples
doing exactly what I am doing.

Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[Bruce Richardson]

On Tue, Oct 20, 2020 at 04:33:42PM +0100, McDaniel, Timothy wrote:
> 
> 
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Tuesday, October 20, 2020 10:21 AM
> > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>; Eads, Gage
> > <gage.eads@intel.com>
> > Cc: Richardson, Bruce <bruce.richardson@intel.com>; Jerin Jacob
> > <jerinjacobk@gmail.com>; Mcnamara, John <john.mcnamara@intel.com>;
> > Kovacevic, Marko <marko.kovacevic@intel.com>; Ray Kinsella
> > <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>; dpdk-dev
> > <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Van Haaren, Harry
> > <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>;
> > david.marchand@redhat.com
> > Subject: Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and
> > meson build infrastructure
> >
> > 20/10/2020 17:17, McDaniel, Timothy:
> > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > > On Sun, Oct 18, 2020 at 02:18:32PM +0530, Jerin Jacob wrote:
> > > > > On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
> > > > > <timothy.mcdaniel@intel.com> wrote:
> > > > > >
> > > > > > Adds the meson build infrastructure, which includes
> > > > > > compile-time constants in rte_config.h. DLB2 is
> > > > > > only supported on Linux X86 platforms at this time.
> > > > > >
> > > > > > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > > > > > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > > > > > ---
> > > > > > --- a/drivers/event/meson.build
> > > > > > +++ b/drivers/event/meson.build
> > > > > > @@ -10,6 +10,9 @@ if not (toolchain == 'gcc' and
> > > > cc.version().version_compare('<4.8.6') and
> > > > > >         dpdk_conf.has('RTE_ARCH_ARM64'))
> > > > > >         drivers += 'octeontx'
> > > > > >  endif
> > > > > > +if (dpdk_conf.has('RTE_ARCH_X86_64') and is_linux)
> > > > > > +        drivers += 'dlb2'
> > > > > > +endif
> > > > >
> > > > > Please add the message in "Content Skipped" section,
> > > > > Reference: grep "reason" in drivers/vdpa/mlx5/meson.build
> > > > >
> > > >
> > > > The octeontx case is also wrong in this file, IMHO. Rather than checking
> > > > things in the event level and adding things to the list, the list should
> > > > just be static. If something should be optionally compiled, then check the
> > > > conditions in the driver meson.build file itself and add "build=false" to
> > > > disable, setting "reason" to the cause of it being disabled. This keeps all
> > > > the logic about a driver in the other file, rather than someone having to
> > > > look in multiple places for why something is or isn't getting built
> > > > properly.
> > > >
> > > > Regards,
> > > > /Bruce
> > >
> > > Due to time constraints, I would prefer to take these issues up in a future
> > release.
> >
> > If you have strong time constraints to complete everything,
> > then it is better to postpone the feature to the next release.
> >
> 
> You read too much into my comment.
> My point is that there does not seem to be community consensus on this issue,

There is consensus here, and the way the octeon event driver is doing
things is wrong, and was just missed in previous reviews.

> so why are we debating it at this point in the release cycle.  There are plenty of examples
> doing exactly what I am doing.
> 
Looking through the driver/*/meson.build files this appears to be the only
incidence where a driver is optionally adding itself to the list. All other
drivers are always processed so that when enabled/disabled they can be
properly listed in the configuration summary and a reason given.

Regards,
/Bruce

Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[Bruce Richardson]

On Tue, Oct 20, 2020 at 04:17:13PM +0100, McDaniel, Timothy wrote:
> 
> 
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Monday, October 19, 2020 3:34 AM
> > To: Jerin Jacob <jerinjacobk@gmail.com>
> > Cc: McDaniel, Timothy <timothy.mcdaniel@intel.com>; Mcnamara, John
> > <john.mcnamara@intel.com>; Kovacevic, Marko
> > <Marko.Kovacevic@intel.com>; Ray Kinsella <mdr@ashroe.eu>; Neil Horman
> > <nhorman@tuxdriver.com>; dpdk-dev <dev@dpdk.org>; Carrillo, Erik G
> > <Erik.G.Carrillo@intel.com>; Eads, Gage <gage.eads@intel.com>; Van Haaren,
> > Harry <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>;
> > Thomas Monjalon <thomas@monjalon.net>
> > Subject: Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and
> > meson build infrastructure
> >
> > On Sun, Oct 18, 2020 at 02:18:32PM +0530, Jerin Jacob wrote:
> > > On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
> > > <timothy.mcdaniel@intel.com> wrote:
> > > >
> > > > Adds the meson build infrastructure, which includes
> > > > compile-time constants in rte_config.h. DLB2 is
> > > > only supported on Linux X86 platforms at this time.
> > > >
> > > > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > > > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > > > ---
> > > >  config/rte_config.h                               |   7 +
> > > >  doc/guides/eventdevs/dlb2.rst                     | 330 ++++++++++++++++++++++
> > > >  doc/guides/eventdevs/index.rst                    |   1 +
> > > >  drivers/event/dlb2/meson.build                    |   7 +
> > > >  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |   3 +
> > > >  drivers/event/meson.build                         |   3 +
> > > >  6 files changed, 351 insertions(+)
> > > >  create mode 100644 doc/guides/eventdevs/dlb2.rst
> > > >  create mode 100644 drivers/event/dlb2/meson.build
> > > >  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> > > >
> > > > diff --git a/config/rte_config.h b/config/rte_config.h
> > > > index 0bae630..fd1b3c3 100644
> > > > --- a/config/rte_config.h
> > > > +++ b/config/rte_config.h
> > > > @@ -131,4 +131,11 @@
> > > >  /* QEDE PMD defines */
> > > >  #define RTE_LIBRTE_QEDE_FW ""
> > > >
> > > > +/* DLB2 defines */
> > > > +#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
> > > > +#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
> > > > +#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
> > > > +#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
> > > > +#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
> > >
> > > We are going way with GLOBAL compile time options for drivers.
> > > I think,  we should move to driver-specific folder not pollute top
> > > level config file.
> > > @Richardson, Bruce  @Thomas Monjalon  Thoughts?
> > >
> > Yes, we do need a better way to pass driver-specific build config options.
> > How likely is it that the user may want to tweak these values? If it's not
> > likely having them just as regular defines in the driver folder may be
> > better, though they are not particularly problematic in rte_config.h given
> > the number of other values already there.
> >
> > >
> > > > +
> > <snip>
> > > > --- a/drivers/event/meson.build
> > > > +++ b/drivers/event/meson.build
> > > > @@ -10,6 +10,9 @@ if not (toolchain == 'gcc' and
> > cc.version().version_compare('<4.8.6') and
> > > >         dpdk_conf.has('RTE_ARCH_ARM64'))
> > > >         drivers += 'octeontx'
> > > >  endif
> > > > +if (dpdk_conf.has('RTE_ARCH_X86_64') and is_linux)
> > > > +        drivers += 'dlb2'
> > > > +endif
> > >
> > > Please add the message in "Content Skipped" section,
> > > Reference: grep "reason" in drivers/vdpa/mlx5/meson.build
> > >
> >
> > The octeontx case is also wrong in this file, IMHO. Rather than checking
> > things in the event level and adding things to the list, the list should
> > just be static. If something should be optionally compiled, then check the
> > conditions in the driver meson.build file itself and add "build=false" to
> > disable, setting "reason" to the cause of it being disabled. This keeps all
> > the logic about a driver in the other file, rather than someone having to
> > look in multiple places for why something is or isn't getting built
> > properly.
> >
> > Regards,
> > /Bruce
> 
> Due to time constraints, I would prefer to take these issues up in a future release.
>
I'm not suggesting you need to fix the existing octeon case, but it's easy
enough to add dlb2 to the existing list and add :

if not dpdk_conf.has('RTE_ARCH_X86_64') or not is_linux
	build = false
	reason = "..."
endif

to the dlb2 meson.build file.

Regards,
/Bruce 

Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[McDaniel, Timothy]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Tuesday, October 20, 2020 10:35 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: Jerin Jacob <jerinjacobk@gmail.com>; Mcnamara, John
> <john.mcnamara@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>; Ray Kinsella <mdr@ashroe.eu>; Neil Horman
> <nhorman@tuxdriver.com>; dpdk-dev <dev@dpdk.org>; Carrillo, Erik G
> <erik.g.carrillo@intel.com>; Eads, Gage <gage.eads@intel.com>; Van Haaren,
> Harry <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>;
> Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and
> meson build infrastructure
> 
> On Tue, Oct 20, 2020 at 04:17:13PM +0100, McDaniel, Timothy wrote:
> >
> >
> > > -----Original Message-----
> > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > Sent: Monday, October 19, 2020 3:34 AM
> > > To: Jerin Jacob <jerinjacobk@gmail.com>
> > > Cc: McDaniel, Timothy <timothy.mcdaniel@intel.com>; Mcnamara, John
> > > <john.mcnamara@intel.com>; Kovacevic, Marko
> > > <Marko.Kovacevic@intel.com>; Ray Kinsella <mdr@ashroe.eu>; Neil
> Horman
> > > <nhorman@tuxdriver.com>; dpdk-dev <dev@dpdk.org>; Carrillo, Erik G
> > > <Erik.G.Carrillo@intel.com>; Eads, Gage <gage.eads@intel.com>; Van
> Haaren,
> > > Harry <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>;
> > > Thomas Monjalon <thomas@monjalon.net>
> > > Subject: Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation
> and
> > > meson build infrastructure
> > >
> > > On Sun, Oct 18, 2020 at 02:18:32PM +0530, Jerin Jacob wrote:
> > > > On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
> > > > <timothy.mcdaniel@intel.com> wrote:
> > > > >
> > > > > Adds the meson build infrastructure, which includes
> > > > > compile-time constants in rte_config.h. DLB2 is
> > > > > only supported on Linux X86 platforms at this time.
> > > > >
> > > > > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > > > > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > > > > ---
> > > > >  config/rte_config.h                               |   7 +
> > > > >  doc/guides/eventdevs/dlb2.rst                     | 330
> ++++++++++++++++++++++
> > > > >  doc/guides/eventdevs/index.rst                    |   1 +
> > > > >  drivers/event/dlb2/meson.build                    |   7 +
> > > > >  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |   3 +
> > > > >  drivers/event/meson.build                         |   3 +
> > > > >  6 files changed, 351 insertions(+)
> > > > >  create mode 100644 doc/guides/eventdevs/dlb2.rst
> > > > >  create mode 100644 drivers/event/dlb2/meson.build
> > > > >  create mode 100644
> drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> > > > >
> > > > > diff --git a/config/rte_config.h b/config/rte_config.h
> > > > > index 0bae630..fd1b3c3 100644
> > > > > --- a/config/rte_config.h
> > > > > +++ b/config/rte_config.h
> > > > > @@ -131,4 +131,11 @@
> > > > >  /* QEDE PMD defines */
> > > > >  #define RTE_LIBRTE_QEDE_FW ""
> > > > >
> > > > > +/* DLB2 defines */
> > > > > +#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
> > > > > +#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
> > > > > +#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
> > > > > +#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
> > > > > +#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
> > > >
> > > > We are going way with GLOBAL compile time options for drivers.
> > > > I think,  we should move to driver-specific folder not pollute top
> > > > level config file.
> > > > @Richardson, Bruce  @Thomas Monjalon  Thoughts?
> > > >
> > > Yes, we do need a better way to pass driver-specific build config options.
> > > How likely is it that the user may want to tweak these values? If it's not
> > > likely having them just as regular defines in the driver folder may be
> > > better, though they are not particularly problematic in rte_config.h given
> > > the number of other values already there.
> > >
> > > >
> > > > > +
> > > <snip>
> > > > > --- a/drivers/event/meson.build
> > > > > +++ b/drivers/event/meson.build
> > > > > @@ -10,6 +10,9 @@ if not (toolchain == 'gcc' and
> > > cc.version().version_compare('<4.8.6') and
> > > > >         dpdk_conf.has('RTE_ARCH_ARM64'))
> > > > >         drivers += 'octeontx'
> > > > >  endif
> > > > > +if (dpdk_conf.has('RTE_ARCH_X86_64') and is_linux)
> > > > > +        drivers += 'dlb2'
> > > > > +endif
> > > >
> > > > Please add the message in "Content Skipped" section,
> > > > Reference: grep "reason" in drivers/vdpa/mlx5/meson.build
> > > >
> > >
> > > The octeontx case is also wrong in this file, IMHO. Rather than checking
> > > things in the event level and adding things to the list, the list should
> > > just be static. If something should be optionally compiled, then check the
> > > conditions in the driver meson.build file itself and add "build=false" to
> > > disable, setting "reason" to the cause of it being disabled. This keeps all
> > > the logic about a driver in the other file, rather than someone having to
> > > look in multiple places for why something is or isn't getting built
> > > properly.
> > >
> > > Regards,
> > > /Bruce
> >
> > Due to time constraints, I would prefer to take these issues up in a future
> release.
> >
> I'm not suggesting you need to fix the existing octeon case, but it's easy
> enough to add dlb2 to the existing list and add :
> 
> if not dpdk_conf.has('RTE_ARCH_X86_64') or not is_linux
> 	build = false
> 	reason = "..."
> endif
> 
> to the dlb2 meson.build file.
> 
> Regards,
> /Bruce

Very well.  I used the existing event/meson.build and conditional inclusion of the octeon driver
as my guide for conditionally including DLB and DLB2.  It's easy enough to change, so I will do it.

Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[McDaniel, Timothy]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Monday, October 19, 2020 3:34 AM
> To: Jerin Jacob <jerinjacobk@gmail.com>
> Cc: McDaniel, Timothy <timothy.mcdaniel@intel.com>; Mcnamara, John
> <john.mcnamara@intel.com>; Kovacevic, Marko
> <Marko.Kovacevic@intel.com>; Ray Kinsella <mdr@ashroe.eu>; Neil Horman
> <nhorman@tuxdriver.com>; dpdk-dev <dev@dpdk.org>; Carrillo, Erik G
> <Erik.G.Carrillo@intel.com>; Eads, Gage <gage.eads@intel.com>; Van Haaren,
> Harry <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>;
> Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and
> meson build infrastructure
> 
> On Sun, Oct 18, 2020 at 02:18:32PM +0530, Jerin Jacob wrote:
> > On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
> > <timothy.mcdaniel@intel.com> wrote:
> > >
> > > Adds the meson build infrastructure, which includes
> > > compile-time constants in rte_config.h. DLB2 is
> > > only supported on Linux X86 platforms at this time.
> > >
> > > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > > ---
> > >  config/rte_config.h                               |   7 +
> > >  doc/guides/eventdevs/dlb2.rst                     | 330 ++++++++++++++++++++++
> > >  doc/guides/eventdevs/index.rst                    |   1 +
> > >  drivers/event/dlb2/meson.build                    |   7 +
> > >  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |   3 +
> > >  drivers/event/meson.build                         |   3 +
> > >  6 files changed, 351 insertions(+)
> > >  create mode 100644 doc/guides/eventdevs/dlb2.rst
> > >  create mode 100644 drivers/event/dlb2/meson.build
> > >  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> > >
> > > diff --git a/config/rte_config.h b/config/rte_config.h
> > > index 0bae630..fd1b3c3 100644
> > > --- a/config/rte_config.h
> > > +++ b/config/rte_config.h
> > > @@ -131,4 +131,11 @@
> > >  /* QEDE PMD defines */
> > >  #define RTE_LIBRTE_QEDE_FW ""
> > >
> > > +/* DLB2 defines */
> > > +#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
> > > +#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
> > > +#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
> > > +#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
> > > +#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
> >
> > We are going way with GLOBAL compile time options for drivers.
> > I think,  we should move to driver-specific folder not pollute top
> > level config file.
> > @Richardson, Bruce  @Thomas Monjalon  Thoughts?
> >
> Yes, we do need a better way to pass driver-specific build config options.
> How likely is it that the user may want to tweak these values? If it's not
> likely having them just as regular defines in the driver folder may be
> better, though they are not particularly problematic in rte_config.h given
> the number of other values already there.
> 

We were careful to only place those options that were likely to be modified by the end user
here.

> >
> > > +
> <snip>
> > > --- a/drivers/event/meson.build
> > > +++ b/drivers/event/meson.build
> > > @@ -10,6 +10,9 @@ if not (toolchain == 'gcc' and
> cc.version().version_compare('<4.8.6') and
> > >         dpdk_conf.has('RTE_ARCH_ARM64'))
> > >         drivers += 'octeontx'
> > >  endif
> > > +if (dpdk_conf.has('RTE_ARCH_X86_64') and is_linux)
> > > +        drivers += 'dlb2'
> > > +endif
> >
> > Please add the message in "Content Skipped" section,
> > Reference: grep "reason" in drivers/vdpa/mlx5/meson.build
> >
> 
> The octeontx case is also wrong in this file, IMHO. Rather than checking
> things in the event level and adding things to the list, the list should
> just be static. If something should be optionally compiled, then check the
> conditions in the driver meson.build file itself and add "build=false" to
> disable, setting "reason" to the cause of it being disabled. This keeps all
> the logic about a driver in the other file, rather than someone having to
> look in multiple places for why something is or isn't getting built
> properly.
> 
> Regards,
> /Bruce

Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Sunday, October 18, 2020 3:49 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: Richardson, Bruce <bruce.richardson@intel.com>; Mcnamara, John
> <john.mcnamara@intel.com>; Kovacevic, Marko
> <Marko.Kovacevic@intel.com>; Ray Kinsella <mdr@ashroe.eu>; Neil Horman
> <nhorman@tuxdriver.com>; dpdk-dev <dev@dpdk.org>; Carrillo, Erik G
> <Erik.G.Carrillo@intel.com>; Eads, Gage <gage.eads@intel.com>; Van Haaren,
> Harry <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>;
> Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and
> meson build infrastructure
> 
> On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > Adds the meson build infrastructure, which includes
> > compile-time constants in rte_config.h. DLB2 is
> > only supported on Linux X86 platforms at this time.
> >
> > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > ---
> >  config/rte_config.h                               |   7 +
> >  doc/guides/eventdevs/dlb2.rst                     | 330 ++++++++++++++++++++++
> >  doc/guides/eventdevs/index.rst                    |   1 +
> >  drivers/event/dlb2/meson.build                    |   7 +
> >  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |   3 +
> >  drivers/event/meson.build                         |   3 +
> >  6 files changed, 351 insertions(+)
> >  create mode 100644 doc/guides/eventdevs/dlb2.rst
> >  create mode 100644 drivers/event/dlb2/meson.build
> >  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> >
> > diff --git a/config/rte_config.h b/config/rte_config.h
> > index 0bae630..fd1b3c3 100644
> > --- a/config/rte_config.h
> > +++ b/config/rte_config.h
> > @@ -131,4 +131,11 @@
> >  /* QEDE PMD defines */
> >  #define RTE_LIBRTE_QEDE_FW ""
> >
> > +/* DLB2 defines */
> > +#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
> > +#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
> > +#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
> > +#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
> > +#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
> 
> We are going way with GLOBAL compile time options for drivers.
> I think,  we should move to driver-specific folder not pollute top
> level config file.
> @Richardson, Bruce  @Thomas Monjalon  Thoughts?
> 
> 
> > +
> >  #endif /* _RTE_CONFIG_H_ */
> > diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
> > new file mode 100644
> > index 0000000..fa8d6dd
> > --- /dev/null
> > +++ b/doc/guides/eventdevs/dlb2.rst
> > @@ -0,0 +1,330 @@
> > +..  SPDX-License-Identifier: BSD-3-Clause
> > +    Copyright(c) 2020 Intel Corporation.
> > +
> > +Driver for the Intel® Dynamic Load Balancer (DLB2)
> > +==================================================
> > +
> > +The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
> > +
> > +Prerequisites
> > +-------------
> > +
> > +Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
> > +the basic DPDK environment.
> > +
> > +Configuration
> > +-------------
> > +
> > +The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
> > +device access. To use this operation mode, the PCIe PF device must be bound
> > +to a DPDK-compatible VFIO driver, such as vfio-pci.
> > +
> > +Eventdev API Notes
> > +------------------
> > +
> > +The DLB2 provides the functions of a DPDK event device; specifically, it
> > +supports atomic, ordered, and parallel scheduling events from queues to
> ports.
> > +However, the DLB2 hardware is not a perfect match to the eventdev API.
> Some DLB2
> > +features are abstracted by the PMD such as directed ports.
> > +
> > +In general the dlb PMD is designed for ease-of-use and does not require a
> > +detailed understanding of the hardware, but these details are important
> when
> > +writing high-performance code. This section describes the places where the
> > +eventdev API and DLB2 misalign.
> > +
> > +Scheduling Domain Configuration
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +There are 32 scheduling domainis the DLB2.
> > +When one is configured, it allocates load-balanced and
> > +directed queues, ports, credits, and other hardware resources. Some
> > +resource allocations are user-controlled -- the number of queues, for
> example
> > +-- and others, like credit pools (one directed and one load-balanced pool per
> > +scheduling domain), are not.
> > +
> > +The DLB2 is a closed system eventdev, and as such the ``nb_events_limit``
> device
> > +setup argument and the per-port ``new_event_threshold`` argument apply as
> > +defined in the eventdev header file. The limit is applied to all enqueues,
> > +regardless of whether it will consume a directed or load-balanced credit.
> > +
> > +Load-balanced and Directed Ports
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +DLB2 ports come in two flavors: load-balanced and directed. The eventdev
> API
> > +does not have the same concept, but it has a similar one: ports and queues
> that
> > +are singly-linked (i.e. linked to a single queue or port, respectively).
> > +
> > +The ``rte_event_dev_info_get()`` function reports the number of available
> > +event ports and queues (among other things). For the DLB2 PMD,
> max_event_ports
> > +and max_event_queues report the number of available load-balanced ports
> and
> > +queues, and max_single_link_event_port_queue_pairs reports the number of
> > +available directed ports and queues.
> > +
> > +When a scheduling domain is created in ``rte_event_dev_configure()``, the
> user
> > +specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
> > +control the total number of ports (load-balanced and directed) and the
> number
> > +of directed ports. Hence, the number of requested load-balanced ports is
> > +``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues``
> field
> > +specifies the total number of queues (load-balanced and directed). The
> number
> > +of directed queues comes from ``nb_single_link_event_port_queues``, since
> > +directed ports and queues come in pairs.
> > +
> > +When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag
> determines
> > +whether it should be configured as a directed (the flag is set) or a
> > +load-balanced (the flag is unset) port. Similarly, the
> > +``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
> > +whether it is a directed or load-balanced queue.
> > +
> > +Load-balanced ports can only be linked to load-balanced queues, and
> directed
> > +ports can only be linked to directed queues. Furthermore, directed ports can
> > +only be linked to a single directed queue (and vice versa), and that link
> > +cannot change after the eventdev is started.
> > +
> > +The eventdev API does not have a directed scheduling type. To support
> directed
> > +traffic, the dlb PMD detects when an event is being sent to a directed queue
> > +and overrides its scheduling type. Note that the originally selected scheduling
> > +type (atomic, ordered, or parallel) is not preserved, and an event's
> sched_type
> > +will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a
> directed
> > +port.
> > +
> > +Flow ID
> > +~~~~~~~
> > +
> > +The flow ID field is preserved in the event when it is scheduled in the
> > +DLB2.
> > +
> > +Hardware Credits
> > +~~~~~~~~~~~~~~~~
> > +
> > +DLB2 uses a hardware credit scheme to prevent software from overflowing
> hardware
> > +event storage, with each unit of storage represented by a credit. A port
> spends
> > +a credit to enqueue an event, and hardware refills the ports with credits as
> the
> > +events are scheduled to ports. Refills come from credit pools, and each port
> is
> > +a member of a load-balanced credit pool and a directed credit pool. The
> > +load-balanced credits are used to enqueue to load-balanced queues, and
> directed
> > +credits are used for directed queues.
> > +
> > +A DLB2 eventdev contains one load-balanced and one directed credit pool.
> These
> > +pools' sizes are controlled by the nb_events_limit field in struct
> > +rte_event_dev_config. The load-balanced pool is sized to contain
> > +nb_events_limit credits, and the directed pool is sized to contain
> > +nb_events_limit/4 credits. The directed pool size can be overridden with the
> > +num_dir_credits vdev argument, like so:
> > +
> > +    .. code-block:: console
> > +
> > +       --vdev=dlb1_event,num_dir_credits=<value>
> > +
> > +This can be used if the default allocation is too low or too high for the
> > +specific application needs. The PMD also supports a vdev arg that limits the
> > +max_num_events reported by rte_event_dev_info_get():
> > +
> > +    .. code-block:: console
> > +
> > +       --vdev=dlb1_event,max_num_events=<value>
> > +
> > +By default, max_num_events is reported as the total available load-balanced
> > +credits. If multiple DLB2-based applications are being used, it may be
> desirable
> > +to control how many load-balanced credits each application uses, particularly
> > +when application(s) are written to configure nb_events_limit equal to the
> > +reported max_num_events.
> > +
> > +Each port is a member of both credit pools. A port's credit allocation is
> > +defined by its low watermark, high watermark, and refill quanta. These three
> > +parameters are calculated by the dlb PMD like so:
> > +
> > +- The load-balanced high watermark is set to the port's enqueue_depth.
> > +  The directed high watermark is set to the minimum of the enqueue_depth
> and
> > +  the directed pool size divided by the total number of ports.
> > +- The refill quanta is set to half the high watermark.
> > +- The low watermark is set to the minimum of 16 and the refill quanta.
> > +
> > +When the eventdev is started, each port is pre-allocated a high watermark's
> > +worth of credits. For example, if an eventdev contains four ports with
> enqueue
> > +depths of 32 and a load-balanced credit pool size of 4096, each port will start
> > +with 32 load-balanced credits, and there will be 3968 credits available to
> > +replenish the ports. Thus, a single port is not capable of enqueueing up to the
> > +nb_events_limit (without any events being dequeued), since the other ports
> are
> > +retaining their initial credit allocation; in short, all ports must enqueue in
> > +order to reach the limit.
> > +
> > +If a port attempts to enqueue and has no credits available, the enqueue
> > +operation will fail and the application must retry the enqueue. Credits are
> > +replenished asynchronously by the DLB2 hardware.
> > +
> > +Software Credits
> > +~~~~~~~~~~~~~~~~
> > +
> > +The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a
> software
> > +credit scheme on top of the hardware credit scheme in order to comply with
> > +the per-port backpressure described in the eventdev API.
> > +
> > +The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends
> a
> > +credit when it enqueues to a queue, and credits are later replenished after
> the
> > +events are dequeued and released.
> > +
> > +In the software credit scheme, a credit is consumed when a new (.op =
> > +RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
> > +replenished when the event is released from the system (either explicitly with
> > +RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
> > +
> > +In this model, an event is "in the system" from its first enqueue into eventdev
> > +until it is last dequeued. If the event goes through multiple event queues, it
> > +is still considered "in the system" while a worker thread is processing it.
> > +
> > +A port will fail to enqueue if the number of events in the system exceeds its
> > +``new_event_threshold`` (specified at port setup time). A port will also fail
> > +to enqueue if it lacks enough hardware credits to enqueue; load-balanced
> > +credits are used to enqueue to a load-balanced queue, and directed credits
> are
> > +used to enqueue to a directed queue.
> > +
> > +The out-of-credit situations are typically transient, and an eventdev
> > +application using the DLB2 ought to retry its enqueues if they fail.
> > +If enqueue fails, DLB2 PMD sets rte_errno as follows:
> > +
> > +- -ENOSPC: Credit exhaustion (either hardware or software)
> > +- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
> > +
> > +Depending on the pipeline the application has constructed, it's possible to
> > +enter a credit deadlock scenario wherein the worker thread lacks the credit
> > +to enqueue an event, and it must dequeue an event before it can recover the
> > +credit. If the worker thread retries its enqueue indefinitely, it will not
> > +make forward progress. Such deadlock is possible if the application has event
> > +"loops", in which an event in dequeued from queue A and later enqueued
> back to
> > +queue A.
> > +
> > +Due to this, workers should stop retrying after a time, release the events it
> > +is attempting to enqueue, and dequeue more events. It is important that the
> > +worker release the events and don't simply set them aside to retry the
> enqueue
> > +again later, because the port has limited history list size (by default, twice
> > +the port's dequeue_depth).
> > +
> > +Priority
> > +~~~~~~~~
> > +
> > +The DLB2 supports event priority and per-port queue service priority, as
> > +described in the eventdev header file. The DLB2 does not support 'global'
> event
> > +queue priority established at queue creation time.
> > +
> > +DLB2 supports 8 event and queue service priority levels. For both priority
> > +types, the PMD uses the upper three bits of the priority field to determine the
> > +DLB2 priority, discarding the 5 least significant bits. The 5 least significant
> > +event priority bits are not preserved when an event is enqueued.
> > +
> > +Load-Balanced Queues
> > +~~~~~~~~~~~~~~~~~~~~
> > +
> > +A load-balanced queue can support atomic and ordered scheduling, or atomic
> and
> > +unordered scheduling, but not atomic and unordered and ordered scheduling.
> A
> > +queue's scheduling types are controlled by the event queue configuration.
> > +
> > +If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
> > +``nb_atomic_order_sequences`` determines the supported scheduling types.
> > +With non-zero ``nb_atomic_order_sequences``, the queue is configured for
> atomic
> > +and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL``
> scheduling is
> > +supported by scheduling those events as ordered events.  Note that when the
> > +event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``.
> Else if
> > +``nb_atomic_order_sequences`` is zero, the queue is configured for atomic
> and
> > +unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is
> unsupported.
> > +
> > +If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
> > +dictates the queue's scheduling type.
> > +
> > +The ``nb_atomic_order_sequences`` queue configuration field sets the
> ordered
> > +queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where
> each
> > +group is configured to contain either 1 queue with 1024 reorder entries, 2
> > +queues with 512 reorder entries, and so on down to 32 queues with 32
> entries.
> > +
> > +When a load-balanced queue is created, the PMD will configure a new
> sequence
> > +number group on-demand if num_sequence_numbers does not match a pre-
> existing
> > +group with available reorder buffer entries. If all sequence number groups are
> > +in use, no new group will be created and queue configuration will fail. (Note
> > +that when the PMD is used with a virtual DLB2 device, it cannot change the
> > +sequence number configuration.)
> > +
> > +The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD,
> because
> > +the DLB2 does not limit the number of flows a queue can track. In the DLB2,
> all
> > +load-balanced queues can use the full 16-bit flow ID range.
> > +
> > +Reconfiguration
> > +~~~~~~~~~~~~~~~
> > +
> > +The Eventdev API allows one to reconfigure a device, its ports, and its queues
> > +by first stopping the device, calling the configuration function(s), then
> > +restarting the device. The DLB2 does not support configuring an individual
> queue
> > +or port without first reconfiguring the entire device, however, so there are
> > +certain reconfiguration sequences that are valid in the eventdev API but not
> > +supported by the PMD.
> > +
> > +Specifically, the PMD supports the following configuration sequence:
> > +1. Configure and start the device
> > +2. Stop the device
> > +3. (Optional) Reconfigure the device
> > +4. (Optional) If step 3 is run:
> > +
> > +   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
> > +   b. The reconfigured port(s) lose their previous queue links.
> > +
> > +5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
> > +6. Restart the device. If the device is reconfigured in step 3 but one or more
> > +   of its ports or queues are not, the PMD will apply their previous
> > +   configuration (including port->queue links) at this time.
> > +
> > +The PMD does not support the following configuration sequences:
> > +1. Configure and start the device
> > +2. Stop the device
> > +3. Setup queue or setup port
> > +4. Start the device
> > +
> > +This sequence is not supported because the event device must be
> reconfigured
> > +before its ports or queues can be.
> > +
> > +Deferred Scheduling
> > +~~~~~~~~~~~~~~~~~~~
> > +
> > +The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once
> per
> > +dequeued event before returning from rte_event_dequeue_burst(). This frees
> the
> > +corresponding entries in the CQ, which enables the DLB2 to schedule more
> events
> > +to it.
> > +
> > +To support applications seeking finer-grained scheduling control -- for
> example
> > +deferring scheduling to get the best possible priority scheduling and
> > +load-balancing -- the PMD supports a deferred scheduling mode. In this
> mode,
> > +the CQ entry is not popped until the *subsequent*
> rte_event_dequeue_burst()
> > +call. This mode only applies to load-balanced event ports with dequeue depth
> of
> > +1.
> > +
> > +To enable deferred scheduling, use the defer_sched vdev argument like so:
> > +
> > +    .. code-block:: console
> > +
> > +       --vdev=dlb1_event,defer_sched=on
> > +
> > +Atomic Inflights Allocation
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
> > +inflight event in a temporary buffer that is divided among load-balanced
> > +queues. If a queue's atomic buffer storage fills up, this can result in
> > +head-of-line-blocking. For example:
> > +
> > +- An LDB queue allocated N atomic buffer entries
> > +- All N entries are filled with events from flow X, which is pinned to CQ 0.
> > +
> > +Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can
> be
> > +scheduled. The likelihood of this case depends on the eventdev configuration,
> > +traffic behavior, event processing latency, potential for a worker to be
> > +interrupted or otherwise delayed, etc.
> > +
> > +By default, the PMD allocates 16 buffer entries for each load-balanced
> queue,
> > +which provides an even division across all 128 queues but potentially wastes
> > +buffer space (e.g. if not all queues are used, or aren't used for atomic
> > +scheduling).
> > +
> > +The PMD provides a dev arg to override the default per-queue allocation. To
> > +increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
> > +
> > +    .. code-block:: console
> > +
> > +       --vdev=dlb1_event,atm_inflights=64
> > +
> 
> 
> Please move the doc changes to whichever patch that introduces this change,
> 
> 

Will do.


> 
> Please add the message in "Content Skipped" section,
> Reference: grep "reason" in drivers/vdpa/mlx5/meson.build
> 
> 
>

Will do

Re: [dpdk-dev] [PATCH v2 01/22] event/dlb2: add documentation and meson build infrastructure

[Kinsella, Ray]

On 17/10/2020 19:20, Timothy McDaniel wrote:
> Adds the meson build infrastructure, which includes
> compile-time constants in rte_config.h. DLB2 is
> only supported on Linux X86 platforms at this time.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> Reviewed-by: Gage Eads <gage.eads@intel.com>
> ---
>  config/rte_config.h                               |   7 +
>  doc/guides/eventdevs/dlb2.rst                     | 330 ++++++++++++++++++++++
>  doc/guides/eventdevs/index.rst                    |   1 +
>  drivers/event/dlb2/meson.build                    |   7 +
>  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |   3 +
>  drivers/event/meson.build                         |   3 +
>  6 files changed, 351 insertions(+)
>  create mode 100644 doc/guides/eventdevs/dlb2.rst
>  create mode 100644 drivers/event/dlb2/meson.build
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> 
> diff --git a/config/rte_config.h b/config/rte_config.h
> index 0bae630..fd1b3c3 100644
> --- a/config/rte_config.h
> +++ b/config/rte_config.h
> @@ -131,4 +131,11 @@
>  /* QEDE PMD defines */
>  #define RTE_LIBRTE_QEDE_FW ""
>  
> +/* DLB2 defines */
> +#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
> +#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
> +#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
> +#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
> +#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
> +
>  #endif /* _RTE_CONFIG_H_ */
> diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
> new file mode 100644
> index 0000000..fa8d6dd
> --- /dev/null
> +++ b/doc/guides/eventdevs/dlb2.rst
> @@ -0,0 +1,330 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2020 Intel Corporation.
> +
> +Driver for the Intel® Dynamic Load Balancer (DLB2)
> +==================================================
> +
> +The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
> +
> +Prerequisites
> +-------------
> +
> +Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
> +the basic DPDK environment.
> +
> +Configuration
> +-------------
> +
> +The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
> +device access. To use this operation mode, the PCIe PF device must be bound
> +to a DPDK-compatible VFIO driver, such as vfio-pci.
> +
> +Eventdev API Notes
> +------------------
> +
> +The DLB2 provides the functions of a DPDK event device; specifically, it
> +supports atomic, ordered, and parallel scheduling events from queues to ports.
> +However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
> +features are abstracted by the PMD such as directed ports.
> +
> +In general the dlb PMD is designed for ease-of-use and does not require a
> +detailed understanding of the hardware, but these details are important when
> +writing high-performance code. This section describes the places where the
> +eventdev API and DLB2 misalign.
> +
> +Scheduling Domain Configuration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +There are 32 scheduling domainis the DLB2.
> +When one is configured, it allocates load-balanced and
> +directed queues, ports, credits, and other hardware resources. Some
> +resource allocations are user-controlled -- the number of queues, for example
> +-- and others, like credit pools (one directed and one load-balanced pool per
> +scheduling domain), are not.
> +
> +The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
> +setup argument and the per-port ``new_event_threshold`` argument apply as
> +defined in the eventdev header file. The limit is applied to all enqueues,
> +regardless of whether it will consume a directed or load-balanced credit.
> +
> +Load-balanced and Directed Ports
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
> +does not have the same concept, but it has a similar one: ports and queues that
> +are singly-linked (i.e. linked to a single queue or port, respectively).
> +
> +The ``rte_event_dev_info_get()`` function reports the number of available
> +event ports and queues (among other things). For the DLB2 PMD, max_event_ports
> +and max_event_queues report the number of available load-balanced ports and
> +queues, and max_single_link_event_port_queue_pairs reports the number of
> +available directed ports and queues.
> +
> +When a scheduling domain is created in ``rte_event_dev_configure()``, the user
> +specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
> +control the total number of ports (load-balanced and directed) and the number
> +of directed ports. Hence, the number of requested load-balanced ports is
> +``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
> +specifies the total number of queues (load-balanced and directed). The number
> +of directed queues comes from ``nb_single_link_event_port_queues``, since
> +directed ports and queues come in pairs.
> +
> +When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
> +whether it should be configured as a directed (the flag is set) or a
> +load-balanced (the flag is unset) port. Similarly, the
> +``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
> +whether it is a directed or load-balanced queue.
> +
> +Load-balanced ports can only be linked to load-balanced queues, and directed
> +ports can only be linked to directed queues. Furthermore, directed ports can
> +only be linked to a single directed queue (and vice versa), and that link
> +cannot change after the eventdev is started.
> +
> +The eventdev API does not have a directed scheduling type. To support directed
> +traffic, the dlb PMD detects when an event is being sent to a directed queue
> +and overrides its scheduling type. Note that the originally selected scheduling
> +type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
> +will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
> +port.
> +
> +Flow ID
> +~~~~~~~
> +
> +The flow ID field is preserved in the event when it is scheduled in the
> +DLB2.
> +
> +Hardware Credits
> +~~~~~~~~~~~~~~~~
> +
> +DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
> +event storage, with each unit of storage represented by a credit. A port spends
> +a credit to enqueue an event, and hardware refills the ports with credits as the
> +events are scheduled to ports. Refills come from credit pools, and each port is
> +a member of a load-balanced credit pool and a directed credit pool. The
> +load-balanced credits are used to enqueue to load-balanced queues, and directed
> +credits are used for directed queues.
> +
> +A DLB2 eventdev contains one load-balanced and one directed credit pool. These
> +pools' sizes are controlled by the nb_events_limit field in struct
> +rte_event_dev_config. The load-balanced pool is sized to contain
> +nb_events_limit credits, and the directed pool is sized to contain
> +nb_events_limit/4 credits. The directed pool size can be overridden with the
> +num_dir_credits vdev argument, like so:
> +
> +    .. code-block:: console
> +
> +       --vdev=dlb1_event,num_dir_credits=<value>
> +
> +This can be used if the default allocation is too low or too high for the
> +specific application needs. The PMD also supports a vdev arg that limits the
> +max_num_events reported by rte_event_dev_info_get():
> +
> +    .. code-block:: console
> +
> +       --vdev=dlb1_event,max_num_events=<value>
> +
> +By default, max_num_events is reported as the total available load-balanced
> +credits. If multiple DLB2-based applications are being used, it may be desirable
> +to control how many load-balanced credits each application uses, particularly
> +when application(s) are written to configure nb_events_limit equal to the
> +reported max_num_events.
> +
> +Each port is a member of both credit pools. A port's credit allocation is
> +defined by its low watermark, high watermark, and refill quanta. These three
> +parameters are calculated by the dlb PMD like so:
> +
> +- The load-balanced high watermark is set to the port's enqueue_depth.
> +  The directed high watermark is set to the minimum of the enqueue_depth and
> +  the directed pool size divided by the total number of ports.
> +- The refill quanta is set to half the high watermark.
> +- The low watermark is set to the minimum of 16 and the refill quanta.
> +
> +When the eventdev is started, each port is pre-allocated a high watermark's
> +worth of credits. For example, if an eventdev contains four ports with enqueue
> +depths of 32 and a load-balanced credit pool size of 4096, each port will start
> +with 32 load-balanced credits, and there will be 3968 credits available to
> +replenish the ports. Thus, a single port is not capable of enqueueing up to the
> +nb_events_limit (without any events being dequeued), since the other ports are
> +retaining their initial credit allocation; in short, all ports must enqueue in
> +order to reach the limit.
> +
> +If a port attempts to enqueue and has no credits available, the enqueue
> +operation will fail and the application must retry the enqueue. Credits are
> +replenished asynchronously by the DLB2 hardware.
> +
> +Software Credits
> +~~~~~~~~~~~~~~~~
> +
> +The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
> +credit scheme on top of the hardware credit scheme in order to comply with
> +the per-port backpressure described in the eventdev API.
> +
> +The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
> +credit when it enqueues to a queue, and credits are later replenished after the
> +events are dequeued and released.
> +
> +In the software credit scheme, a credit is consumed when a new (.op =
> +RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
> +replenished when the event is released from the system (either explicitly with
> +RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
> +
> +In this model, an event is "in the system" from its first enqueue into eventdev
> +until it is last dequeued. If the event goes through multiple event queues, it
> +is still considered "in the system" while a worker thread is processing it.
> +
> +A port will fail to enqueue if the number of events in the system exceeds its
> +``new_event_threshold`` (specified at port setup time). A port will also fail
> +to enqueue if it lacks enough hardware credits to enqueue; load-balanced
> +credits are used to enqueue to a load-balanced queue, and directed credits are
> +used to enqueue to a directed queue.
> +
> +The out-of-credit situations are typically transient, and an eventdev
> +application using the DLB2 ought to retry its enqueues if they fail.
> +If enqueue fails, DLB2 PMD sets rte_errno as follows:
> +
> +- -ENOSPC: Credit exhaustion (either hardware or software)
> +- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
> +
> +Depending on the pipeline the application has constructed, it's possible to
> +enter a credit deadlock scenario wherein the worker thread lacks the credit
> +to enqueue an event, and it must dequeue an event before it can recover the
> +credit. If the worker thread retries its enqueue indefinitely, it will not
> +make forward progress. Such deadlock is possible if the application has event
> +"loops", in which an event in dequeued from queue A and later enqueued back to
> +queue A.
> +
> +Due to this, workers should stop retrying after a time, release the events it
> +is attempting to enqueue, and dequeue more events. It is important that the
> +worker release the events and don't simply set them aside to retry the enqueue
> +again later, because the port has limited history list size (by default, twice
> +the port's dequeue_depth).
> +
> +Priority
> +~~~~~~~~
> +
> +The DLB2 supports event priority and per-port queue service priority, as
> +described in the eventdev header file. The DLB2 does not support 'global' event
> +queue priority established at queue creation time.
> +
> +DLB2 supports 8 event and queue service priority levels. For both priority
> +types, the PMD uses the upper three bits of the priority field to determine the
> +DLB2 priority, discarding the 5 least significant bits. The 5 least significant
> +event priority bits are not preserved when an event is enqueued.
> +
> +Load-Balanced Queues
> +~~~~~~~~~~~~~~~~~~~~
> +
> +A load-balanced queue can support atomic and ordered scheduling, or atomic and
> +unordered scheduling, but not atomic and unordered and ordered scheduling. A
> +queue's scheduling types are controlled by the event queue configuration.
> +
> +If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
> +``nb_atomic_order_sequences`` determines the supported scheduling types.
> +With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
> +and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
> +supported by scheduling those events as ordered events.  Note that when the
> +event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
> +``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
> +unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
> +
> +If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
> +dictates the queue's scheduling type.
> +
> +The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
> +queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
> +group is configured to contain either 1 queue with 1024 reorder entries, 2
> +queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
> +
> +When a load-balanced queue is created, the PMD will configure a new sequence
> +number group on-demand if num_sequence_numbers does not match a pre-existing
> +group with available reorder buffer entries. If all sequence number groups are
> +in use, no new group will be created and queue configuration will fail. (Note
> +that when the PMD is used with a virtual DLB2 device, it cannot change the
> +sequence number configuration.)
> +
> +The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
> +the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
> +load-balanced queues can use the full 16-bit flow ID range.
> +
> +Reconfiguration
> +~~~~~~~~~~~~~~~
> +
> +The Eventdev API allows one to reconfigure a device, its ports, and its queues
> +by first stopping the device, calling the configuration function(s), then
> +restarting the device. The DLB2 does not support configuring an individual queue
> +or port without first reconfiguring the entire device, however, so there are
> +certain reconfiguration sequences that are valid in the eventdev API but not
> +supported by the PMD.
> +
> +Specifically, the PMD supports the following configuration sequence:
> +1. Configure and start the device
> +2. Stop the device
> +3. (Optional) Reconfigure the device
> +4. (Optional) If step 3 is run:
> +
> +   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
> +   b. The reconfigured port(s) lose their previous queue links.
> +
> +5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
> +6. Restart the device. If the device is reconfigured in step 3 but one or more
> +   of its ports or queues are not, the PMD will apply their previous
> +   configuration (including port->queue links) at this time.
> +
> +The PMD does not support the following configuration sequences:
> +1. Configure and start the device
> +2. Stop the device
> +3. Setup queue or setup port
> +4. Start the device
> +
> +This sequence is not supported because the event device must be reconfigured
> +before its ports or queues can be.
> +
> +Deferred Scheduling
> +~~~~~~~~~~~~~~~~~~~
> +
> +The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
> +dequeued event before returning from rte_event_dequeue_burst(). This frees the
> +corresponding entries in the CQ, which enables the DLB2 to schedule more events
> +to it.
> +
> +To support applications seeking finer-grained scheduling control -- for example
> +deferring scheduling to get the best possible priority scheduling and
> +load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
> +the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
> +call. This mode only applies to load-balanced event ports with dequeue depth of
> +1.
> +
> +To enable deferred scheduling, use the defer_sched vdev argument like so:
> +
> +    .. code-block:: console
> +
> +       --vdev=dlb1_event,defer_sched=on
> +
> +Atomic Inflights Allocation
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
> +inflight event in a temporary buffer that is divided among load-balanced
> +queues. If a queue's atomic buffer storage fills up, this can result in
> +head-of-line-blocking. For example:
> +
> +- An LDB queue allocated N atomic buffer entries
> +- All N entries are filled with events from flow X, which is pinned to CQ 0.
> +
> +Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
> +scheduled. The likelihood of this case depends on the eventdev configuration,
> +traffic behavior, event processing latency, potential for a worker to be
> +interrupted or otherwise delayed, etc.
> +
> +By default, the PMD allocates 16 buffer entries for each load-balanced queue,
> +which provides an even division across all 128 queues but potentially wastes
> +buffer space (e.g. if not all queues are used, or aren't used for atomic
> +scheduling).
> +
> +The PMD provides a dev arg to override the default per-queue allocation. To
> +increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
> +
> +    .. code-block:: console
> +
> +       --vdev=dlb1_event,atm_inflights=64
> +
> diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
> index bb66a5e..55d5ba8 100644
> --- a/doc/guides/eventdevs/index.rst
> +++ b/doc/guides/eventdevs/index.rst
> @@ -18,3 +18,4 @@ application through the eventdev API.
>      octeontx
>      octeontx2
>      opdl
> +    dlb2
> diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
> new file mode 100644
> index 0000000..54ba2c8
> --- /dev/null
> +++ b/drivers/event/dlb2/meson.build
> @@ -0,0 +1,7 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2019-2020 Intel Corporation
> +
> +sources = files(
> +)
> +
> +deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
> diff --git a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> new file mode 100644
> index 0000000..299ae63
> --- /dev/null
> +++ b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> @@ -0,0 +1,3 @@
> +DPDK_21.0 {

Should be DPDK_21

> +	local: *;
> +};
> diff --git a/drivers/event/meson.build b/drivers/event/meson.build
> index ebe76a7..01d3b59 100644
> --- a/drivers/event/meson.build
> +++ b/drivers/event/meson.build
> @@ -10,6 +10,9 @@ if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
>  	dpdk_conf.has('RTE_ARCH_ARM64'))
>  	drivers += 'octeontx'
>  endif
> +if (dpdk_conf.has('RTE_ARCH_X86_64') and is_linux)
> +        drivers += 'dlb2'
> +endif
>  std_deps = ['eventdev', 'kvargs']
>  config_flag_fmt = 'RTE_LIBRTE_@0@_EVENTDEV_PMD'
>  driver_name_fmt = 'rte_pmd_@0@_event'
> 

[dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[Timothy McDaniel]

The following patch series adds support for a new eventdev PMD. The DLB2
PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
hardware.
The DLB2 is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB2 hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB2 ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a
directed queue. Additionally, even though LDB ports may link multiple
queues,
the number of queues that may be linked is limited by hardware.

While reviewing the code, please be aware that this PMD has full
control over the DLB2 hardware. Intel will be extending the DLB2 PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.
Note that the DLB2 hardware is a successor of the DLB hardware, and
as such is structured similarly, both in terms of code layout and
implementation.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Major changes since V4:
================
- moved introduction of dlb in relnotes_20_11 to first patch in series
- fixed underlines in dlb.rst that were too short
- note that the code still uses its private byte-encoded versions of
  umonitor/umwait, rather than the new functions in the power
  patch that are built on top of those intrinsics. This is intentional.

Changes since V3:
================
- updated MAINTAINERS file to alphabetically insert DLB
- don't create RTE_ symbols in pmd
- converted to use version.map scheme
- converted to use .._master_lcore instead of .._main_lcore
- this patch set is based on dpdk-next-eventdev

Changes since V2:
================
- fixed meson conditional build. Moved test into driver’s meson.build
  file instead of event/meson.build
- documentation is populated as associated code is introduced
- add log_register in add dynamic logging patch
- rename RTE_xxx symbol(s) as DLB2_xxx
- replaced function ptr enqueue_four with direct call to movdir64b
- remove unused port_pages
- broke up probe patch into 3 smaller patches for easier review
- changed param order of movdir64b/movntdq to match intrinsics
- added self to MAINTAINERS files
- squashed announcement of availability into last patch in series
- correct spelling errors and delete repeated words
- DPDK_21.0 -> DPDK 21 in map file
- add experimental banner to public structs and APIs

Changes since V1:
=================
- implement changes requested in code reviews by Gage Eads and Mike
  Chen
- fix a memzone leak
- convert to use eal rte-cpuflags patch from Liang Ma

Known Issues:
- the documentation contained in dlb2.rst is not complete, and
  should be updated to include command line parameters (class of service,
        etc ...).

Depends-on: patch-82202 ("eventdev: increase MAX QUEUES PER DEV to 255")
Depends-on: patch-79539 ("eal: add new x86 cpuid support for WAITPKG")

Timothy McDaniel (23):
  event/dlb2: add documentation and meson build infrastructure
  event/dlb2: add dynamic logging
  event/dlb2: add private data structures and constants
  event/dlb2: add definitions shared with LKM or shared code
  event/dlb2: add inline functions
  event/dlb2: add eventdev probe
  event/dlb2: add flexible interface
  event/dlb2: add probe-time hardware init
  event/dlb2: add xstats
  event/dlb2: add infos get and configure
  event/dlb2: add queue and port default conf
  event/dlb2: add queue setup
  event/dlb2: add port setup
  event/dlb2: add port link
  event/dlb2: add port unlink and port unlinks in progress
  event/dlb2: add eventdev start
  event/dlb2: add enqueue and its burst variants
  event/dlb2: add dequeue and its burst variants
  event/dlb2: add eventdev stop and close
  event/dlb2: add PMD's token pop public interface
  event/dlb2: add PMD self-tests
  event/dlb2: add queue and port release
  event/dlb2: add timeout ticks entry point

 MAINTAINERS                                    |    6 +-
 app/test/test_eventdev.c                       |    7 +
 config/rte_config.h                            |    7 +
 doc/api/doxy-api-index.md                      |    1 +
 doc/guides/eventdevs/dlb2.rst                  |  365 ++
 doc/guides/eventdevs/index.rst                 |    1 +
 doc/guides/rel_notes/release_20_11.rst         |    5 +
 drivers/event/dlb2/dlb2.c                      | 3951 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c                |   74 +
 drivers/event/dlb2/dlb2_iface.h                |   74 +
 drivers/event/dlb2/dlb2_inline_fns.h           |   61 +
 drivers/event/dlb2/dlb2_log.h                  |   25 +
 drivers/event/dlb2/dlb2_priv.h                 |  578 +++
 drivers/event/dlb2/dlb2_selftest.c             | 1570 ++++++
 drivers/event/dlb2/dlb2_user.h                 |  679 +++
 drivers/event/dlb2/dlb2_xstats.c               | 1240 +++++
 drivers/event/dlb2/meson.build                 |   22 +
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 +++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 +
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c     | 6027 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  673 +++
 drivers/event/dlb2/pf/dlb2_main.h              |   97 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  728 +++
 drivers/event/dlb2/rte_pmd_dlb2.c              |   39 +
 drivers/event/dlb2/rte_pmd_dlb2.h              |   72 +
 drivers/event/dlb2/version.map                 |    9 +
 drivers/event/meson.build                      |    3 +-
 33 files changed, 22547 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
 create mode 100644 drivers/event/dlb2/dlb2_log.h
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c
 create mode 100644 drivers/event/dlb2/dlb2_user.h
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
 create mode 100644 drivers/event/dlb2/version.map

-- 
2.6.4

[dpdk-dev] [PATCH v5 01/23] event/dlb2: add documentation and meson build infrastructure

[Timothy McDaniel]

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux 64 bit X86 platforms at this time.

Adds announcement of availabililty for the new driver
for Intel Dynamic Load Balancer 2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 MAINTAINERS                            |  6 ++++-
 config/rte_config.h                    |  7 ++++++
 doc/guides/eventdevs/dlb2.rst          | 40 ++++++++++++++++++++++++++++++++++
 doc/guides/eventdevs/index.rst         |  1 +
 doc/guides/rel_notes/release_20_11.rst |  5 +++++
 drivers/event/dlb2/meson.build         | 13 +++++++++++
 drivers/event/dlb2/version.map         |  3 +++
 drivers/event/meson.build              |  3 ++-
 8 files changed, 76 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index a3d1927..727451b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1162,6 +1162,11 @@ Cavium OCTEON TX timvf
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 F: drivers/event/octeontx/timvf_*
 
+Intel DLB2
+M: Timothy McDaniel <timothy.mcdaniel@intel.com>
+F: drivers/event/dlb2/
+F: doc/guides/eventdevs/dlb2.rst
+
 Marvell OCTEON TX2
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 M: Jerin Jacob <jerinj@marvell.com>
@@ -1198,7 +1203,6 @@ M: Peter Mccarthy <peter.mccarthy@intel.com>
 F: drivers/event/opdl/
 F: doc/guides/eventdevs/opdl.rst
 
-
 Rawdev Drivers
 --------------
 
diff --git a/config/rte_config.h b/config/rte_config.h
index b78c6aa..64aa333 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -135,4 +135,11 @@
 /* QEDE PMD defines */
 #define RTE_LIBRTE_QEDE_FW ""
 
+/* DLB2 defines */
+#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
+#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
+#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
+#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
+
 #endif /* _RTE_CONFIG_H_ */
diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
new file mode 100644
index 0000000..e3091c9
--- /dev/null
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -0,0 +1,40 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Driver for the Intel® Dynamic Load Balancer (DLB2)
+==================================================
+
+The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
+
+Prerequisites
+-------------
+
+Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
+the basic DPDK environment.
+
+Configuration
+-------------
+
+The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
+device access. To use this operation mode, the PCIe PF device must be bound
+to a DPDK-compatible VFIO driver, such as vfio-pci.
+
+Eventdev API Notes
+------------------
+
+The DLB2 provides the functions of a DPDK event device; specifically, it
+supports atomic, ordered, and parallel scheduling events from queues to ports.
+However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
+features are abstracted by the PMD such as directed ports.
+
+In general the dlb PMD is designed for ease-of-use and does not require a
+detailed understanding of the hardware, but these details are important when
+writing high-performance code. This section describes the places where the
+eventdev API and DLB2 misalign.
+
+Flow ID
+~~~~~~~
+
+The flow ID field is preserved in the event when it is scheduled in the
+DLB2.
+
diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
index bb66a5e..738788d 100644
--- a/doc/guides/eventdevs/index.rst
+++ b/doc/guides/eventdevs/index.rst
@@ -11,6 +11,7 @@ application through the eventdev API.
     :maxdepth: 2
     :numbered:
 
+    dlb2
     dpaa
     dpaa2
     dsw
diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
index d8ac359..dbce3b1 100644
--- a/doc/guides/rel_notes/release_20_11.rst
+++ b/doc/guides/rel_notes/release_20_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
+
+  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
+  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
+
 * **Added write combining store APIs.**
 
   Added ``rte_write32_wc`` and ``rte_write32_wc_relaxed`` APIs
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
new file mode 100644
index 0000000..5324043
--- /dev/null
+++ b/drivers/event/dlb2/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2020 Intel Corporation
+
+if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
+        build = false
+        reason = 'only supported on ARCH_X86_64 Linux'
+        subdir_done()
+endif
+
+sources = files(
+)
+
+deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
new file mode 100644
index 0000000..4a76d1d
--- /dev/null
+++ b/drivers/event/dlb2/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index a7dac99..c433c10 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -5,7 +5,8 @@ if is_windows
 	subdir_done()
 endif
 
-drivers = ['dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw', 'dsw']
+drivers = ['dlb2', 'dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw',
+	   'dsw']
 if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
-- 
2.6.4

[dpdk-dev] [PATCH v5 02/23] event/dlb2: add dynamic logging

[Timothy McDaniel]

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c      |  7 +++++++
 drivers/event/dlb2/dlb2_log.h  | 25 +++++++++++++++++++++++++
 drivers/event/dlb2/meson.build |  3 +--
 3 files changed, 33 insertions(+), 2 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_log.h

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
new file mode 100644
index 0000000..2082dd6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2.c
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <rte_log.h>
+
+RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
new file mode 100644
index 0000000..dc1481e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_log.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_EVDEV_LOG_H_
+#define _DLB2_EVDEV_LOG_H_
+
+extern int eventdev_dlb2_log_level;
+
+/* Dynamic logging */
+#define DLB2_LOG_IMPL(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
+		__func__, ##args)
+
+#define DLB2_LOG_INFO(fmt, args...) \
+	DLB2_LOG_IMPL(INFO, fmt, ## args)
+
+#define DLB2_LOG_ERR(fmt, args...) \
+	DLB2_LOG_IMPL(ERR, fmt, ## args)
+
+/* remove debug logs at compile time unless actually debugging */
+#define DLB2_LOG_DBG(fmt, args...) \
+	RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
+
+#endif /* _DLB2_EVDEV_LOG_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 5324043..435c1ee 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,7 +7,6 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files(
-)
+sources = files('dlb2.c')
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v5 03/23] event/dlb2: add private data structures and constants

[Timothy McDaniel]

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_priv.h | 575 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 575 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_priv.h

diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
new file mode 100644
index 0000000..61567a6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -0,0 +1,575 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_PRIV_H_
+#define _DLB2_PRIV_H_
+
+#include <emmintrin.h>
+#include <stdbool.h>
+
+#include <rte_eventdev.h>
+#include <rte_config.h>
+#include "dlb2_user.h"
+#include "dlb2_log.h"
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
+#else
+#define DLB2_INC_STAT(_stat, _incr_val)
+#endif
+
+#define EVDEV_DLB2_NAME_PMD dlb2_event
+
+/*  command line arg strings */
+#define NUMA_NODE_ARG "numa_node"
+#define DLB2_MAX_NUM_EVENTS "max_num_events"
+#define DLB2_NUM_DIR_CREDITS "num_dir_credits"
+#define DEV_ID_ARG "dev_id"
+#define DLB2_DEFER_SCHED_ARG "defer_sched"
+#define DLB2_QID_DEPTH_THRESH_ARG "qid_depth_thresh"
+#define DLB2_COS_ARG "cos"
+
+/* Begin HW related defines and structs */
+
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_VFS 16
+#define DLB2_MAX_NUM_LDB_QUEUES 32
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS 64
+#define DLB2_MAX_NUM_DIR_QUEUES 64
+#define DLB2_MAX_NUM_FLOWS (64 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDITS (8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS (2 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_DIR_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQOS_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_MAX_DEVICE_PATH 32
+#define DLB2_MIN_DEQUEUE_TIMEOUT_NS 1
+/* Note: "- 1" here to support the timeout range check in eventdev_autotest */
+#define DLB2_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1)
+#define DLB2_SW_CREDIT_BATCH_SZ 32
+#define DLB2_NUM_SN_GROUPS 2
+#define DLB2_MAX_LDB_SN_ALLOC 1024
+#define DLB2_MAX_QUEUE_DEPTH_THRESHOLD 8191
+
+/* 2048 total hist list entries and 64 total ldb ports, which
+ * makes for 2048/64 == 32 hist list entries per port. However, CQ
+ * depth must be a power of 2 and must also be >= HIST LIST entries.
+ * As a result we just limit the maximum dequeue depth to 32.
+ */
+#define DLB2_MIN_CQ_DEPTH 1
+#define DLB2_MAX_CQ_DEPTH 32
+#define DLB2_MIN_HARDWARE_CQ_DEPTH 8
+#define DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \
+	DLB2_MAX_CQ_DEPTH
+
+/*
+ * Static per queue/port provisioning values
+ */
+#define DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 64
+
+#define CQ_BASE(is_dir) ((is_dir) ? DLB2_DIR_CQ_BASE : DLB2_LDB_CQ_BASE)
+#define CQ_SIZE(is_dir) ((is_dir) ? DLB2_DIR_CQ_MAX_SIZE : \
+				    DLB2_LDB_CQ_MAX_SIZE)
+#define PP_BASE(is_dir) ((is_dir) ? DLB2_DIR_PP_BASE : DLB2_LDB_PP_BASE)
+
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+
+#define DLB2_NUM_QES_PER_CACHE_LINE 4
+
+#define DLB2_MAX_ENQUEUE_DEPTH 64
+#define DLB2_MIN_ENQUEUE_DEPTH 4
+
+#define DLB2_NAME_SIZE 64
+
+#define DLB2_1K 1024
+#define DLB2_2K (2 * DLB2_1K)
+#define DLB2_4K (4 * DLB2_1K)
+#define DLB2_16K (16 * DLB2_1K)
+#define DLB2_32K (32 * DLB2_1K)
+#define DLB2_1MB (DLB2_1K * DLB2_1K)
+#define DLB2_16MB (16 * DLB2_1MB)
+
+/* Use the upper 3 bits of the event priority to select the DLB2 priority */
+#define EV_TO_DLB2_PRIO(x) ((x) >> 5)
+#define DLB2_TO_EV_PRIO(x) ((x) << 5)
+
+enum dlb2_hw_port_types {
+	DLB2_LDB_PORT,
+	DLB2_DIR_PORT,
+	DLB2_NUM_PORT_TYPES /* Must be last */
+};
+
+enum dlb2_hw_queue_types {
+	DLB2_LDB_QUEUE,
+	DLB2_DIR_QUEUE,
+	DLB2_NUM_QUEUE_TYPES /* Must be last */
+};
+
+#define PORT_TYPE(p) ((p)->is_directed ? DLB2_DIR_PORT : DLB2_LDB_PORT)
+
+/* Do not change - must match hardware! */
+enum dlb2_hw_sched_type {
+	DLB2_SCHED_ATOMIC = 0,
+	DLB2_SCHED_UNORDERED,
+	DLB2_SCHED_ORDERED,
+	DLB2_SCHED_DIRECTED,
+	/* DLB2_NUM_HW_SCHED_TYPES must be last */
+	DLB2_NUM_HW_SCHED_TYPES
+};
+
+struct dlb2_hw_rsrcs {
+	int32_t nb_events_limit;
+	uint32_t num_queues;		/* Total queues (lb + dir) */
+	uint32_t num_ldb_queues;	/* Number of available ldb queues */
+	uint32_t num_ldb_ports;         /* Number of load balanced ports */
+	uint32_t num_dir_ports;         /* Number of directed ports */
+	uint32_t num_ldb_credits;       /* Number of load balanced credits */
+	uint32_t num_dir_credits;       /* Number of directed credits */
+	uint32_t reorder_window_size;   /* Size of reorder window */
+};
+
+struct dlb2_hw_resource_info {
+	/**> Max resources that can be provided */
+	struct dlb2_hw_rsrcs hw_rsrc_max;
+	int num_sched_domains;
+	uint32_t socket_id;
+};
+
+enum dlb2_enqueue_type {
+	/**>
+	 * New : Used to inject a new packet into the QM.
+	 */
+	DLB2_ENQ_NEW,
+	/**>
+	 * Forward : Enqueues a packet, and
+	 *  - if atomic: release any lock it holds in the QM
+	 *  - if ordered: release the packet for egress re-ordering
+	 */
+	DLB2_ENQ_FWD,
+	/**>
+	 * Enqueue Drop : Release an inflight packet. Must be called with
+	 * event == NULL. Used to drop a packet.
+	 *
+	 * Note that all packets dequeued from a load-balanced port must be
+	 * released, either with DLB2_ENQ_DROP or DLB2_ENQ_FWD.
+	 */
+	DLB2_ENQ_DROP,
+
+	/* marker for array sizing etc. */
+	_DLB2_NB_ENQ_TYPES
+};
+
+/* hw-specific format - do not change */
+
+struct dlb2_event_type {
+	uint8_t major:4;
+	uint8_t unused:4;
+	uint8_t sub;
+};
+
+union dlb2_opaque_data {
+	uint16_t opaque_data;
+	struct dlb2_event_type event_type;
+};
+
+struct dlb2_msg_info {
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+};
+
+#define DLB2_NEW_CMD_BYTE 0x08
+#define DLB2_FWD_CMD_BYTE 0x0A
+#define DLB2_COMP_CMD_BYTE 0x02
+#define DLB2_POP_CMD_BYTE 0x01
+#define DLB2_NOOP_CMD_BYTE 0x00
+
+/* hw-specific format - do not change */
+struct dlb2_enqueue_qe {
+	uint64_t data;
+	/* Word 3 */
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	/* Word 4 */
+	uint16_t lock_id;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_cq_pop_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t tokens:10;
+	uint16_t rsvd2:6;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_dequeue_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t flow_id:16; /* was pp_id in v1 */
+	uint8_t debug;
+	uint8_t cq_gen:1;
+	uint8_t qid_depth:2; /* 2 bits in v2 */
+	uint8_t rsvd1:2;
+	uint8_t error:1;
+	uint8_t rsvd2:2;
+};
+
+union dlb2_port_config {
+	struct dlb2_create_ldb_port_args ldb;
+	struct dlb2_create_dir_port_args dir;
+};
+
+enum dlb2_port_state {
+	PORT_CLOSED,
+	PORT_STARTED,
+	PORT_STOPPED
+};
+
+enum dlb2_configuration_state {
+	/* The resource has not been configured */
+	DLB2_NOT_CONFIGURED,
+	/* The resource was configured, but the device was stopped */
+	DLB2_PREV_CONFIGURED,
+	/* The resource is currently configured */
+	DLB2_CONFIGURED
+};
+
+struct dlb2_port {
+	uint32_t id;
+	bool is_directed;
+	bool gen_bit;
+	uint16_t dir_credits;
+	uint32_t dequeue_depth;
+	union dlb2_port_config cfg;
+	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
+	uint16_t cached_ldb_credits;
+	uint16_t ldb_credits;
+	uint16_t cached_dir_credits;
+	bool int_armed;
+	uint16_t owed_tokens;
+	int16_t issued_releases;
+	int cq_depth;
+	uint16_t cq_idx;
+	uint16_t cq_idx_unmasked;
+	uint16_t cq_depth_mask;
+	uint16_t gen_bit_shift;
+	enum dlb2_port_state state;
+	enum dlb2_configuration_state config_state;
+	int num_mapped_qids;
+	uint8_t *qid_mappings;
+	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
+	struct dlb2_enqueue_qe *int_arm_qe;
+	struct dlb2_cq_pop_qe *consume_qe;
+	struct dlb2_eventdev *dlb2; /* back ptr */
+	struct dlb2_eventdev_port *ev_port; /* back ptr */
+};
+
+/* Per-process per-port mmio and memory pointers */
+struct process_local_port_data {
+	uint64_t *pp_addr;
+	struct dlb2_dequeue_qe *cq_base;
+	const struct rte_memzone *mz;
+	bool mmaped;
+};
+
+struct dlb2_eventdev;
+
+struct dlb2_config {
+	int configured;
+	int reserved;
+	uint32_t num_ldb_credits;
+	uint32_t num_dir_credits;
+	struct dlb2_create_sched_domain_args resources;
+};
+
+enum dlb2_cos {
+	DLB2_COS_DEFAULT = -1,
+	DLB2_COS_0 = 0,
+	DLB2_COS_1,
+	DLB2_COS_2,
+	DLB2_COS_3
+};
+
+struct dlb2_hw_dev {
+	struct dlb2_config cfg;
+	struct dlb2_hw_resource_info info;
+	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
+	uint32_t domain_id;
+	enum dlb2_cos cos_id;
+	rte_spinlock_t resource_lock; /* for MP support */
+} __rte_cache_aligned;
+
+/* End HW related defines and structs */
+
+/* Begin DLB2 PMD Eventdev related defines and structs */
+
+#define DLB2_MAX_NUM_QUEUES \
+	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
+
+#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS + DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
+
+/** Structure to hold the queue to port link establishment attributes */
+
+struct dlb2_event_queue_link {
+	uint8_t queue_id;
+	uint8_t priority;
+	bool mapped;
+	bool valid;
+};
+
+struct dlb2_traffic_stats {
+	uint64_t rx_ok;
+	uint64_t rx_drop;
+	uint64_t rx_interrupt_wait;
+	uint64_t rx_umonitor_umwait;
+	uint64_t tx_ok;
+	uint64_t total_polls;
+	uint64_t zero_polls;
+	uint64_t tx_nospc_ldb_hw_credits;
+	uint64_t tx_nospc_dir_hw_credits;
+	uint64_t tx_nospc_inflight_max;
+	uint64_t tx_nospc_new_event_limit;
+	uint64_t tx_nospc_inflight_credits;
+};
+
+/* DLB2 HW sets the 2bit qid_depth in rx QEs based on the programmable depth
+ * threshold. The global default value in config/common_base (or rte_config.h)
+ * can be overridden on a per-qid basis using a vdev command line parameter.
+ * 3: depth > threshold
+ * 2: threshold >= depth > 3/4 threshold
+ * 1: 3/4 threshold >= depth > 1/2 threshold
+ * 0: depth <= 1/2 threshold.
+ */
+#define DLB2_QID_DEPTH_LE50 0
+#define DLB2_QID_DEPTH_GT50_LE75 1
+#define DLB2_QID_DEPTH_GT75_LE100 2
+#define DLB2_QID_DEPTH_GT100 3
+#define DLB2_NUM_QID_DEPTH_STAT_VALS 4 /* 2 bits */
+
+struct dlb2_queue_stats {
+	uint64_t enq_ok;
+	uint64_t qid_depth[DLB2_NUM_QID_DEPTH_STAT_VALS];
+};
+
+struct dlb2_port_stats {
+	struct dlb2_traffic_stats traffic;
+	uint64_t tx_op_cnt[4]; /* indexed by rte_event.op */
+	uint64_t tx_implicit_rel;
+	uint64_t tx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t tx_invalid;
+	uint64_t rx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t rx_sched_invalid;
+	struct dlb2_queue_stats queue[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_eventdev_port {
+	struct dlb2_port qm_port; /* hw specific data structure */
+	struct rte_event_port_conf conf; /* user-supplied configuration */
+	uint16_t inflight_credits; /* num credits this port has right now */
+	uint16_t credit_update_quanta;
+	struct dlb2_eventdev *dlb2; /* backlink optimization */
+	struct dlb2_port_stats stats __rte_cache_aligned;
+	struct dlb2_event_queue_link link[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	int num_links;
+	uint32_t id; /* port id */
+	/* num releases yet to be completed on this port.
+	 * Only applies to load-balanced ports.
+	 */
+	uint16_t outstanding_releases;
+	uint16_t inflight_max; /* app requested max inflights for this port */
+	/* setup_done is set when the event port is setup */
+	bool setup_done;
+	/* enq_configured is set when the qm port is created */
+	bool enq_configured;
+	uint8_t implicit_release; /* release events before dequeueing */
+}  __rte_cache_aligned;
+
+struct dlb2_queue {
+	uint32_t num_qid_inflights; /* User config */
+	uint32_t num_atm_inflights; /* User config */
+	enum dlb2_configuration_state config_state;
+	int sched_type; /* LB queue only */
+	uint32_t id;
+	bool is_directed;
+};
+
+struct dlb2_eventdev_queue {
+	struct dlb2_queue qm_queue;
+	struct rte_event_queue_conf conf; /* User config */
+	int depth_threshold; /* use default if 0 */
+	uint32_t id;
+	bool setup_done;
+	uint8_t num_links;
+};
+
+enum dlb2_run_state {
+	DLB2_RUN_STATE_STOPPED = 0,
+	DLB2_RUN_STATE_STOPPING,
+	DLB2_RUN_STATE_STARTING,
+	DLB2_RUN_STATE_STARTED
+};
+
+struct dlb2_eventdev {
+	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
+	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each queue */
+	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
+	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each port */
+	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
+	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
+	struct dlb2_get_num_resources_args hw_rsrc_query_results;
+	uint32_t xstats_count_mode_queue;
+	struct dlb2_hw_dev qm_instance; /* strictly hw related */
+	uint64_t global_dequeue_wait_ticks;
+	struct dlb2_xstats_entry *xstats;
+	struct rte_eventdev *event_dev; /* backlink to dev */
+	uint32_t xstats_count_mode_dev;
+	uint32_t xstats_count_mode_port;
+	uint32_t xstats_count;
+	uint32_t inflights; /* use __atomic builtins */
+	uint32_t new_event_limit;
+	int max_num_events_override;
+	int num_dir_credits_override;
+	volatile enum dlb2_run_state run_state;
+	uint16_t num_dir_queues; /* total num of evdev dir queues requested */
+	uint16_t num_dir_credits;
+	uint16_t num_ldb_credits;
+	uint16_t num_queues; /* total queues */
+	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested */
+	uint16_t num_ports; /* total num of evdev ports requested */
+	uint16_t num_ldb_ports; /* total num of ldb ports requested */
+	uint16_t num_dir_ports; /* total num of dir ports requested */
+	bool umwait_allowed;
+	bool global_dequeue_wait; /* Not using per dequeue wait if true */
+	bool defer_sched;
+	enum dlb2_cq_poll_modes poll_mode;
+	uint8_t revision;
+	bool configured;
+	uint16_t max_ldb_credits;
+	uint16_t max_dir_credits;
+
+	/* force hw credit pool counters into exclusive cache lines */
+
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t ldb_credit_pool __rte_cache_aligned;
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t dir_credit_pool __rte_cache_aligned;
+};
+
+/* used for collecting and passing around the dev args */
+struct dlb2_qid_depth_thresholds {
+	int val[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_devargs {
+	int socket_id;
+	int max_num_events;
+	int num_dir_credits_override;
+	int dev_id;
+	int defer_sched;
+	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
+	enum dlb2_cos cos_id;
+};
+
+/* End Eventdev related defines and structs */
+
+/* Forwards for non-inlined functions */
+
+void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
+
+int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
+
+void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
+
+int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n);
+
+int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstat_names,
+		unsigned int *ids, unsigned int size);
+
+uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+					  const char *name, unsigned int *id);
+
+int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode,
+		int16_t queue_port_id,
+		const uint32_t ids[],
+		uint32_t nb_ids);
+
+int test_dlb2_eventdev(void);
+
+int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+				const char *name,
+				struct dlb2_devargs *dlb2_args);
+
+int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+				  const char *name);
+
+uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *queue);
+
+int dlb2_parse_params(const char *params,
+		      const char *name,
+		      struct dlb2_devargs *dlb2_args);
+
+/* Extern globals */
+extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
+
+#endif	/* _DLB2_PRIV_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v5 04/23] event/dlb2: add definitions shared with LKM or shared code

[Timothy McDaniel]

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_user.h | 679 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 679 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_user.h

diff --git a/drivers/event/dlb2/dlb2_user.h b/drivers/event/dlb2/dlb2_user.h
new file mode 100644
index 0000000..f4bda78
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_user.h
@@ -0,0 +1,679 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#define DLB2_MAX_NAME_LEN 64
+
+#include <linux/types.h>
+
+enum dlb2_error {
+	DLB2_ST_SUCCESS = 0,
+	DLB2_ST_NAME_EXISTS,
+	DLB2_ST_DOMAIN_UNAVAILABLE,
+	DLB2_ST_LDB_PORTS_UNAVAILABLE,
+	DLB2_ST_DIR_PORTS_UNAVAILABLE,
+	DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB2_ST_INVALID_DOMAIN_ID,
+	DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB2_ST_INVALID_LDB_QUEUE_ID,
+	DLB2_ST_INVALID_CQ_DEPTH,
+	DLB2_ST_INVALID_CQ_VIRT_ADDR,
+	DLB2_ST_INVALID_PORT_ID,
+	DLB2_ST_INVALID_QID,
+	DLB2_ST_INVALID_PRIORITY,
+	DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+	DLB2_ST_INVALID_DIR_QUEUE_ID,
+	DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB2_ST_DOMAIN_NOT_CONFIGURED,
+	DLB2_ST_INTERNAL_ERROR,
+	DLB2_ST_DOMAIN_IN_USE,
+	DLB2_ST_DOMAIN_NOT_FOUND,
+	DLB2_ST_QUEUE_NOT_FOUND,
+	DLB2_ST_DOMAIN_STARTED,
+	DLB2_ST_DOMAIN_NOT_STARTED,
+	DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB2_ST_DOMAIN_RESET_FAILED,
+	DLB2_ST_MBOX_ERROR,
+	DLB2_ST_INVALID_HIST_LIST_DEPTH,
+	DLB2_ST_NO_MEMORY,
+	DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB2_ST_INVALID_COS_ID,
+};
+
+static const char dlb2_error_strings[][128] = {
+	"DLB2_ST_SUCCESS",
+	"DLB2_ST_NAME_EXISTS",
+	"DLB2_ST_DOMAIN_UNAVAILABLE",
+	"DLB2_ST_LDB_PORTS_UNAVAILABLE",
+	"DLB2_ST_DIR_PORTS_UNAVAILABLE",
+	"DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+	"DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+	"DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+	"DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+	"DLB2_ST_INVALID_DOMAIN_ID",
+	"DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+	"DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+	"DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+	"DLB2_ST_INVALID_LDB_QUEUE_ID",
+	"DLB2_ST_INVALID_CQ_DEPTH",
+	"DLB2_ST_INVALID_CQ_VIRT_ADDR",
+	"DLB2_ST_INVALID_PORT_ID",
+	"DLB2_ST_INVALID_QID",
+	"DLB2_ST_INVALID_PRIORITY",
+	"DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+	"DLB2_ST_INVALID_DIR_QUEUE_ID",
+	"DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+	"DLB2_ST_DOMAIN_NOT_CONFIGURED",
+	"DLB2_ST_INTERNAL_ERROR",
+	"DLB2_ST_DOMAIN_IN_USE",
+	"DLB2_ST_DOMAIN_NOT_FOUND",
+	"DLB2_ST_QUEUE_NOT_FOUND",
+	"DLB2_ST_DOMAIN_STARTED",
+	"DLB2_ST_DOMAIN_NOT_STARTED",
+	"DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+	"DLB2_ST_DOMAIN_RESET_FAILED",
+	"DLB2_ST_MBOX_ERROR",
+	"DLB2_ST_INVALID_HIST_LIST_DEPTH",
+	"DLB2_ST_NO_MEMORY",
+	"DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+	"DLB2_ST_INVALID_COS_ID",
+};
+
+struct dlb2_cmd_response {
+	__u32 status; /* Interpret using enum dlb2_error */
+	__u32 id;
+};
+
+/*******************/
+/* 'dlb2' commands */
+/*******************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+	DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	This ioctl interface is the same in all driver versions and is always
+ *	the first ioctl.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+	/* Output parameters */
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ *	load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ *		percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ *	load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ *   ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+enum dlb2_cq_poll_modes {
+	DLB2_CQ_POLL_MODE_STD,
+	DLB2_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB2_CQ_POLL_MODE must be last */
+	NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/********************************/
+/* 'scheduling domain' commands */
+/********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_start_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
+#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
+
+
+#endif /* __DLB2_USER_H */
-- 
2.6.4

[dpdk-dev] [PATCH v5 05/23] event/dlb2: add inline functions

[Timothy McDaniel]

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_inline_fns.h | 61 ++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h

diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
new file mode 100644
index 0000000..ccbc4f4
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_inline_fns.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_INLINE_FNS_H_
+#define _DLB2_INLINE_FNS_H_
+
+/* Inline functions required in more than one source file. */
+
+static inline struct dlb2_eventdev *
+dlb2_pmd_priv(const struct rte_eventdev *eventdev)
+{
+	return eventdev->data->dev_private;
+}
+
+static inline void
+dlb2_umonitor(volatile void *addr)
+{
+	asm volatile(".byte 0xf3, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (addr));
+}
+
+static inline void
+dlb2_umwait(int state, uint64_t timeout)
+{
+	uint32_t eax = timeout & UINT32_MAX;
+	uint32_t edx = timeout >> 32;
+
+	asm volatile(".byte 0xf2, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (state),  "a" (eax), "d" (edx));
+}
+
+static inline void
+dlb2_movntdq_single(void *pp_addr, void *qe4)
+{
+	long long *_qe  = (long long *)qe4;
+	__v2di src_data0 = (__v2di){_qe[0], _qe[1]};
+
+	__builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
+}
+
+static inline void
+dlb2_cldemote(void *addr)
+{
+	/* Load addr into RSI, then demote the cache line of the address
+	 * contained in that register.
+	 */
+	asm volatile(".byte 0x0f, 0x1c, 0x06" :: "S" (addr));
+}
+
+static inline void
+dlb2_movdir64b(void *pp_addr, void *qe4)
+{
+	asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
+		     :
+		     : "a" (pp_addr), "d" (qe4));
+}
+
+#endif /* _DLB2_INLINE_FNS_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v5 06/23] event/dlb2: add eventdev probe

[Timothy McDaniel]

Add the eventdev portion of probe, and parse command line
options, but do not initialize hardware.

Changes since v2 patch-set probe:

Primary and secondary probe-time init has been removed, and
will be introduced in subsequent patches contained in
the v3 patch-set.

Hardware init has been moved to a subsequent patch in order to
minimize the patch size. The files dlb2/pf/dlb2_pf.c and
dlb2/pf/dlb_main.c include stubs of hardware init functions,
and these will be replaced with the real versions when the
hardware layer is committed.

Initialization of the flexible interface layer has been moved to
a subsequent patch in order to minimize patch size.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                      |  343 ++++
 drivers/event/dlb2/meson.build                 |    5 +-
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +++
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  621 ++++++
 drivers/event/dlb2/pf/dlb2_main.h              |   98 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  196 ++
 13 files changed, 7497 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2082dd6..1b11979 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2,6 +2,349 @@
  * Copyright(c) 2016-2020 Intel Corporation
  */
 
+#include <assert.h>
+#include <errno.h>
+#include <nmmintrin.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_cycles.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_io.h>
+#include <rte_kvargs.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_ring.h>
+#include <rte_string_fns.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+/*
+ * Resources exposed to eventdev. Some values overridden at runtime using
+ * values returned by the DLB kernel driver.
+ */
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
+#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
+#endif
+
+struct process_local_port_data
+dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
+
+#define DLB2_BASE_10 10
+
+static int
+dlb2_string_to_int(int *result, const char *str)
+{
+	long ret;
+	char *endptr;
+
+	if (str == NULL || result == NULL)
+		return -EINVAL;
+
+	errno = 0;
+	ret = strtol(str, &endptr, DLB2_BASE_10);
+	if (errno)
+		return -errno;
+
+	/* long int and int may be different width for some architectures */
+	if (ret < INT_MIN || ret > INT_MAX || endptr == str)
+		return -EINVAL;
+
+	*result = ret;
+	return 0;
+}
+
+static int
+set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
+{
+	int *socket_id = opaque;
+	int ret;
+
+	ret = dlb2_string_to_int(socket_id, value);
+	if (ret < 0)
+		return ret;
+
+	if (*socket_id > RTE_MAX_NUMA_NODES)
+		return -EINVAL;
+	return 0;
+}
+
+static int
+set_max_num_events(const char *key __rte_unused,
+		   const char *value,
+		   void *opaque)
+{
+	int *max_num_events = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(max_num_events, value);
+	if (ret < 0)
+		return ret;
+
+	if (*max_num_events < 0 || *max_num_events >
+			DLB2_MAX_NUM_LDB_CREDITS) {
+		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
+			     DLB2_MAX_NUM_LDB_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_num_dir_credits(const char *key __rte_unused,
+		    const char *value,
+		    void *opaque)
+{
+	int *num_dir_credits = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(num_dir_credits, value);
+	if (ret < 0)
+		return ret;
+
+	if (*num_dir_credits < 0 ||
+	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
+		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
+			     DLB2_MAX_NUM_DIR_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_dev_id(const char *key __rte_unused,
+	   const char *value,
+	   void *opaque)
+{
+	int *dev_id = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(dev_id, value);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+set_cos(const char *key __rte_unused,
+	const char *value,
+	void *opaque)
+{
+	enum dlb2_cos *cos_id = opaque;
+	int x = 0;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(&x, value);
+	if (ret < 0)
+		return ret;
+
+	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
+		DLB2_LOG_ERR(
+			"COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
+			x);
+		return -EINVAL;
+	}
+
+	*cos_id = x;
+
+	return 0;
+}
+
+
+static int
+set_qid_depth_thresh(const char *key __rte_unused,
+		     const char *value,
+		     void *opaque)
+{
+	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
+	int first, last, thresh, i;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	/* command line override may take one of the following 3 forms:
+	 * qid_depth_thresh=all:<threshold_value> ... all queues
+	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
+	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
+	 */
+	if (sscanf(value, "all:%d", &thresh) == 1) {
+		first = 0;
+		last = DLB2_MAX_NUM_QUEUES - 1;
+	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
+		/* we have everything we need */
+	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
+		last = first;
+	} else {
+		DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val, qid-qid:val, or qid:val\n");
+		return -EINVAL;
+	}
+
+	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
+		return -EINVAL;
+	}
+
+	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
+			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
+		return -EINVAL;
+	}
+
+	for (i = first; i <= last; i++)
+		qid_thresh->val[i] = thresh; /* indexed by qid */
+
+	return 0;
+}
+
+int
+dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+			    const char *name,
+			    struct dlb2_devargs *dlb2_args)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+	RTE_SET_USED(dlb2_args);
+
+	return 0;
+}
+
+int
+dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+			      const char *name)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+int
+dlb2_parse_params(const char *params,
+		  const char *name,
+		  struct dlb2_devargs *dlb2_args)
+{
+	int ret = 0;
+	static const char * const args[] = { NUMA_NODE_ARG,
+					     DLB2_MAX_NUM_EVENTS,
+					     DLB2_NUM_DIR_CREDITS,
+					     DEV_ID_ARG,
+					     DLB2_QID_DEPTH_THRESH_ARG,
+					     DLB2_COS_ARG,
+					     NULL };
+
+	if (params != NULL && params[0] != '\0') {
+		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
+
+		if (kvlist == NULL) {
+			RTE_LOG(INFO, PMD,
+				"Ignoring unsupported parameters when creating device '%s'\n",
+				name);
+		} else {
+			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
+						     set_numa_node,
+						     &dlb2_args->socket_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing numa node parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
+						 set_max_num_events,
+						 &dlb2_args->max_num_events);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist,
+					DLB2_NUM_DIR_CREDITS,
+					set_num_dir_credits,
+					&dlb2_args->num_dir_credits_override);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
+						 set_dev_id,
+						 &dlb2_args->dev_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(
+					kvlist,
+					DLB2_QID_DEPTH_THRESH_ARG,
+					set_qid_depth_thresh,
+					&dlb2_args->qid_depth_thresholds);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
+						 set_cos,
+						 &dlb2_args->cos_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing cos parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
 
+			rte_kvargs_free(kvlist);
+		}
+	}
+	return ret;
+}
 RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 435c1ee..1eab9b9 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,6 +7,9 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files('dlb2.c')
+sources = files('dlb2.c',
+		'pf/dlb2_main.c',
+		'pf/dlb2_pf.c'
+)
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
new file mode 100644
index 0000000..1d99f1e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_osdep_list.h"
+#include "dlb2_osdep_types.h"
+
+#define DLB2_MAX_NUM_VDEVS			16
+#define DLB2_MAX_NUM_DOMAINS			32
+#define DLB2_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS			64
+#define DLB2_MAX_NUM_DIR_PORTS			64
+#define DLB2_MAX_NUM_LDB_CREDITS		(8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS		(2 * 1024)
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB2_MAX_NUM_AQED_ENTRIES		2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB2_QID_PRIORITIES			8
+#define DLB2_NUM_ARB_WEIGHTS			8
+#define DLB2_MAX_WEIGHT				255
+#define DLB2_NUM_COS_DOMAINS			4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#ifdef FPGA
+#define DLB2_HZ					2000000
+#else
+#define DLB2_HZ					800000000
+#endif
+
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
+
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+	DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+	DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID		3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID		4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW		5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ	1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
+#define DLB2_VF_BASE_CQ_VECTOR_ID	     0
+#define DLB2_VF_LAST_CQ_VECTOR_ID	     30
+#define DLB2_VF_MBOX_VECTOR_ID		     31
+#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
+
+#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
+					     DLB2_MAX_NUM_DIR_PORTS + 1)
+
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE   0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
+				DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE   0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
+				DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE       0x2100000
+#define DLB2_LDB_PP_STRIDE     0x1000
+#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
+				DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE       0x2000000
+#define DLB2_DIR_PP_STRIDE     0x1000
+#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
+				DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
+struct dlb2_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb2_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb2_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
+struct dlb2_ldb_queue {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB2_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB2_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB2_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB2_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+	enum dlb2_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb2_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+	u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+	u32 bound[6] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & (1 << i))) {
+			group->slot_use_bitmap |= 1 << i;
+			return i;
+		}
+	}
+
+	return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+	int i, cnt = 0;
+
+	for (i = 0; i < 32; i++)
+		cnt += !!(group->slot_use_bitmap & (1 << i));
+
+	return cnt;
+}
+
+struct dlb2_hw_domain {
+	struct dlb2_function_resources *parent_func;
+	struct dlb2_list_entry func_list;
+	struct dlb2_list_head used_ldb_queues;
+	struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head used_dir_pq_pairs;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb2_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb2_bitmap;
+
+struct dlb2_function_resources {
+	struct dlb2_list_head avail_domains;
+	struct dlb2_list_head used_domains;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	struct dlb2_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+	struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+	struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+	struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
+struct dlb2_mbox {
+	u32 *mbox;
+	u32 *isr_in_progress;
+};
+
+struct dlb2_sw_mbox {
+	struct dlb2_mbox vdev_to_pf;
+	struct dlb2_mbox pf_to_vdev;
+	void (*pf_to_vdev_inject)(void *arg);
+	void *pf_to_vdev_inject_arg;
+};
+
+struct dlb2_hw {
+	/* BAR 0 address */
+	void  *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void  *func_kva;
+	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb2_hw_resources rsrcs;
+	struct dlb2_function_resources pf;
+	struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+	struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+	/* Virtualization */
+	int virt_mode;
+	struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
+	unsigned int pasid[DLB2_MAX_NUM_VDEVS];
+};
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
new file mode 100644
index 0000000..ce462c0
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
@@ -0,0 +1,596 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_BASE_DLB2_MBOX_H
+#define __DLB2_BASE_DLB2_MBOX_H
+
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+
+#define DLB2_MBOX_INTERFACE_VERSION 1
+
+/*
+ * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
+ * to send requests of its own (e.g. notifying a VF of an impending FLR).
+ * To avoid communication race conditions, e.g. the PF sends a response and then
+ * sends a request before the VF reads the response, the PF->VF mailbox is
+ * divided into two sections:
+ * - Bytes 0-47: PF responses
+ * - Bytes 48-63: PF requests
+ *
+ * Partitioning the PF->VF mailbox allows responses and requests to occupy the
+ * mailbox simultaneously.
+ */
+#define DLB2_PF2VF_RESP_BYTES	  48
+#define DLB2_PF2VF_RESP_BASE	  0
+#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
+
+#define DLB2_PF2VF_REQ_BYTES	  16
+#define DLB2_PF2VF_REQ_BASE	  (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
+#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
+
+/*
+ * Similarly, the VF->PF mailbox is divided into two sections:
+ * - Bytes 0-239: VF requests
+ * -- (Bytes 0-3 are unused due to a hardware errata)
+ * - Bytes 240-255: VF responses
+ */
+#define DLB2_VF2PF_REQ_BYTES	 236
+#define DLB2_VF2PF_REQ_BASE	 4
+#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
+
+#define DLB2_VF2PF_RESP_BYTES	  16
+#define DLB2_VF2PF_RESP_BASE	  (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
+#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
+
+/* VF-initiated commands */
+enum dlb2_mbox_cmd_type {
+	DLB2_MBOX_CMD_REGISTER,
+	DLB2_MBOX_CMD_UNREGISTER,
+	DLB2_MBOX_CMD_GET_NUM_RESOURCES,
+	DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
+	DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
+	DLB2_MBOX_CMD_CREATE_LDB_PORT,
+	DLB2_MBOX_CMD_CREATE_DIR_PORT,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT,
+	DLB2_MBOX_CMD_DISABLE_LDB_PORT,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT,
+	DLB2_MBOX_CMD_DISABLE_DIR_PORT,
+	DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_MAP_QID,
+	DLB2_MBOX_CMD_UNMAP_QID,
+	DLB2_MBOX_CMD_START_DOMAIN,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
+	DLB2_MBOX_CMD_ARM_CQ_INTR,
+	DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
+	DLB2_MBOX_CMD_GET_SN_ALLOCATION,
+	DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
+	DLB2_MBOX_CMD_GET_COS_BW,
+	DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
+	DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_QE_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_CMD_TYPES,
+};
+
+static const char dlb2_mbox_cmd_type_strings[][128] = {
+	"DLB2_MBOX_CMD_REGISTER",
+	"DLB2_MBOX_CMD_UNREGISTER",
+	"DLB2_MBOX_CMD_GET_NUM_RESOURCES",
+	"DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_LDB_PORT",
+	"DLB2_MBOX_CMD_CREATE_DIR_PORT",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_DISABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_DISABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_MAP_QID",
+	"DLB2_MBOX_CMD_UNMAP_QID",
+	"DLB2_MBOX_CMD_START_DOMAIN",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
+	"DLB2_MBOX_CMD_ARM_CQ_INTR",
+	"DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
+	"DLB2_MBOX_CMD_GET_SN_ALLOCATION",
+	"DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
+	"DLB2_MBOX_CMD_GET_COS_BW",
+	"DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
+	"DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
+};
+
+/* PF-initiated commands */
+enum dlb2_mbox_vf_cmd_type {
+	DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
+	DLB2_MBOX_VF_CMD_NOTIFICATION,
+	DLB2_MBOX_VF_CMD_IN_USE,
+
+	/* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_VF_CMD_TYPES,
+};
+
+static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
+	"DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
+	"DLB2_MBOX_VF_CMD_NOTIFICATION",
+	"DLB2_MBOX_VF_CMD_IN_USE",
+};
+
+#define DLB2_MBOX_CMD_TYPE(hdr) \
+	(((struct dlb2_mbox_req_hdr *)hdr)->type)
+#define DLB2_MBOX_CMD_STRING(hdr) \
+	dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
+
+enum dlb2_mbox_status_type {
+	DLB2_MBOX_ST_SUCCESS,
+	DLB2_MBOX_ST_INVALID_CMD_TYPE,
+	DLB2_MBOX_ST_VERSION_MISMATCH,
+	DLB2_MBOX_ST_INVALID_OWNER_VF,
+};
+
+static const char dlb2_mbox_status_type_strings[][128] = {
+	"DLB2_MBOX_ST_SUCCESS",
+	"DLB2_MBOX_ST_INVALID_CMD_TYPE",
+	"DLB2_MBOX_ST_VERSION_MISMATCH",
+	"DLB2_MBOX_ST_INVALID_OWNER_VF",
+};
+
+#define DLB2_MBOX_ST_TYPE(hdr) \
+	(((struct dlb2_mbox_resp_hdr *)hdr)->status)
+#define DLB2_MBOX_ST_STRING(hdr) \
+	dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
+
+/* This structure is always the first field in a request structure */
+struct dlb2_mbox_req_hdr {
+	u32 type;
+};
+
+/* This structure is always the first field in a response structure */
+struct dlb2_mbox_resp_hdr {
+	u32 status;
+};
+
+struct dlb2_mbox_register_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 min_interface_version;
+	u16 max_interface_version;
+};
+
+struct dlb2_mbox_register_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 interface_version;
+	u8 pf_id;
+	u8 vf_id;
+	u8 is_auxiliary_vf;
+	u8 primary_vf_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u16 num_sched_domains;
+	u16 num_ldb_queues;
+	u16 num_ldb_ports;
+	u16 num_cos_ldb_ports[4];
+	u16 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 max_contiguous_hist_list_entries;
+	u16 num_ldb_credits;
+	u16 num_dir_credits;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 num_ldb_queues;
+	u32 num_ldb_ports;
+	u32 num_cos_ldb_ports[4];
+	u32 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u8 cos_strict;
+	u8 padding0[3];
+	u32 padding1;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 num_sequence_numbers;
+	u32 num_qid_inflights;
+	u32 num_atomic_inflights;
+	u32 lock_id_comp_level;
+	u32 depth_threshold;
+	u32 padding;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 depth_threshold;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u16 cq_depth;
+	u16 cq_history_list_size;
+	u8 cos_id;
+	u8 cos_strict;
+	u16 padding1;
+	u64 cq_base_address;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u64 cq_base_address;
+	u16 cq_depth;
+	u16 padding0;
+	s32 queue_id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_map_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+	u32 priority;
+	u32 padding0;
+};
+
+struct dlb2_mbox_map_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_start_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+};
+
+struct dlb2_mbox_start_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 is_ldb;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding0;
+};
+
+/*
+ * The alert_id and aux_alert_data follows the format of the alerts defined in
+ * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
+ * the aux_alert_data value varies depending on the alert.
+ */
+struct dlb2_mbox_vf_alert_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 alert_id;
+	u32 aux_alert_data;
+};
+
+enum dlb2_mbox_vf_notification_type {
+	DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
+	DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
+
+	/* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
+	NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
+};
+
+struct dlb2_mbox_vf_notification_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 notification;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 in_use;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 num;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 cos_id;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 mode;
+};
+
+#endif /* __DLB2_BASE_DLB2_MBOX_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
new file mode 100644
index 0000000..c4c34eb
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
@@ -0,0 +1,230 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_H
+#define __DLB2_OSDEP_H
+
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <rte_string_fns.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include "../dlb2_main.h"
+#include "dlb2_resource.h"
+#include "../../dlb2_log.h"
+#include "../../dlb2_user.h"
+
+
+#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
+#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
+#define DLB2_CSR_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
+#define DLB2_FUNC_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
+/* Map to PMDs logging interface */
+#define DLB2_ERR(dev, fmt, args...) \
+	DLB2_LOG_ERR(fmt, ## args)
+
+#define DLB2_INFO(dev, fmt, args...) \
+	DLB2_LOG_INFO(fmt, ## args)
+
+#define DLB2_DEBUG(dev, fmt, args...) \
+	DLB2_LOG_DBG(fmt, ## args)
+
+/**
+ * os_udelay() - busy-wait for a number of microseconds
+ * @usecs: delay duration.
+ */
+static inline void os_udelay(int usecs)
+{
+	rte_delay_us(usecs);
+}
+
+/**
+ * os_msleep() - sleep for a number of milliseconds
+ * @usecs: delay duration.
+ */
+static inline void os_msleep(int msecs)
+{
+	rte_delay_ms(msecs);
+}
+
+#define DLB2_PP_BASE(__is_ldb) \
+	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
+
+/**
+ * os_map_producer_port() - map a producer port into the caller's address space
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ *
+ * This function maps the requested producer port memory into the caller's
+ * address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void *os_map_producer_port(struct dlb2_hw *hw,
+					 u8 port_id,
+					 bool is_ldb)
+{
+	uint64_t addr;
+	uint64_t pp_dma_base;
+
+	pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
+	addr = (pp_dma_base + (PAGE_SIZE * port_id));
+
+	return (void *)(uintptr_t)addr;
+}
+
+/**
+ * os_unmap_producer_port() - unmap a producer port
+ * @addr: mapped producer port address
+ *
+ * This function undoes os_map_producer_port() by unmapping the producer port
+ * memory from the caller's address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(addr);
+}
+
+/**
+ * os_fence_hcw() - fence an HCW to ensure it arrives at the device
+ * @hw: dlb2_hw handle for a particular device.
+ * @pp_addr: producer port address
+ */
+static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
+{
+	RTE_SET_USED(hw);
+
+	/* To ensure outstanding HCWs reach the device, read the PP address. IA
+	 * memory ordering prevents reads from passing older writes, and the
+	 * mfence also ensures this.
+	 */
+	rte_mb();
+
+	*(volatile u64 *)pp_addr;
+}
+
+/**
+ * DLB2_HW_ERR() - log an error message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_ERR(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_ERR(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/**
+ * DLB2_HW_DBG() - log an info message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_DBG(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_DEBUG(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/* The callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void *dlb2_complete_queue_map_unmap(void *__args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
+	int ret;
+
+	while (1) {
+		rte_spinlock_lock(&dlb2_dev->resource_mutex);
+
+		ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+		ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+		if (ret != 0) {
+			rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+			/* Relinquish the CPU so the application can process
+			 * its CQs, so this function doesn't deadlock.
+			 */
+			sched_yield();
+		} else {
+			break;
+		}
+	}
+
+	dlb2_dev->worker_launched = false;
+
+	rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+
+	return NULL;
+}
+
+
+/**
+ * os_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static inline void os_schedule_work(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+	pthread_t complete_queue_map_unmap_thread;
+	int ret;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
+				     "dlb_queue_unmap_waiter",
+				     NULL,
+				     dlb2_complete_queue_map_unmap,
+				     dlb2_dev);
+	if (ret)
+		DLB2_ERR(dlb2_dev,
+			 "Could not create queue complete map/unmap thread, err=%d\n",
+			 ret);
+	else
+		dlb2_dev->worker_launched = true;
+}
+
+/**
+ * os_worker_active() - query whether the map/unmap worker thread is active
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether a thread (launched by
+ * os_schedule_work()) is active. This function is used to determine
+ * whether or not to launch a worker thread.
+ */
+static inline bool os_worker_active(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	return dlb2_dev->worker_launched;
+}
+
+#endif /*  __DLB2_OSDEP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
new file mode 100644
index 0000000..423233b
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <rte_bitmap.h>
+#include <rte_string_fns.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include "../dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+	struct rte_bitmap *map;
+	unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+				    unsigned int len)
+{
+	struct dlb2_bitmap *bm;
+	void *mem;
+	uint32_t alloc_size;
+	uint32_t nbits = (uint32_t)len;
+
+	if (bitmap == NULL || nbits == 0)
+		return -EINVAL;
+
+	/* Allocate DLB2 bitmap control struct */
+	bm = rte_malloc("DLB2_PF",
+			sizeof(struct dlb2_bitmap),
+			RTE_CACHE_LINE_SIZE);
+
+	if (bm == NULL)
+		return -ENOMEM;
+
+	/* Allocate bitmap memory */
+	alloc_size = rte_bitmap_get_memory_footprint(nbits);
+	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
+	if (mem == NULL) {
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->map = rte_bitmap_init(len, mem, alloc_size);
+	if (bm->map == NULL) {
+		rte_free(mem);
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap == NULL)
+		return;
+
+	rte_free(bitmap->map);
+	rte_free(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++)
+		rte_bitmap_set(bitmap->map, i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	rte_bitmap_reset(bitmap->map);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set() - set a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
+				  unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_set(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+					unsigned int bit,
+					unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_set(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear() - clear a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
+				    unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_clear(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+					  unsigned int bit,
+					  unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_clear(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+						 unsigned int len)
+{
+	unsigned int i, j = 0;
+
+	if (bitmap  == NULL || bitmap->map == NULL || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			if (++j == len)
+				return i - j + 1;
+		} else {
+			j = 0;
+		}
+	}
+
+	/* No set bit range of length len? */
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			return i;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+	int weight = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			weight++;
+	}
+	return weight;
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				      bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+	int max_len = 0, len = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			len++;
+		} else {
+			if (len > max_len)
+				max_len = len;
+			len = 0;
+		}
+	}
+
+	if (len > max_len)
+		max_len = len;
+
+	return max_len;
+}
+
+/**
+ * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
+ * @dest: pointer to dlb2_bitmap structure, which will contain the results of
+ *	  the 'or' of src1 and src2.
+ * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
+ * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
+ *
+ * This function 'or's two bitmaps together and stores the result in a third
+ * bitmap. The source and destination bitmaps can be the same.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - One of the bitmaps is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
+				 struct dlb2_bitmap *src1,
+				 struct dlb2_bitmap *src2)
+{
+	unsigned int i, min;
+	int numset = 0;
+
+	if (dest  == NULL || dest->map == NULL ||
+	    src1  == NULL || src1->map == NULL ||
+	    src2  == NULL || src2->map == NULL)
+		return -EINVAL;
+
+	min = dest->len;
+	min = (min > src1->len) ? src1->len : min;
+	min = (min > src2->len) ? src2->len : min;
+
+	for (i = 0; i != min; i++) {
+		if  (rte_bitmap_get(src1->map, i) ||
+		     rte_bitmap_get(src2->map, i)) {
+			rte_bitmap_set(dest->map, i);
+			numset++;
+		} else {
+			rte_bitmap_clear(dest->map, i);
+		}
+	}
+
+	return numset;
+}
+
+#endif /*  __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
new file mode 100644
index 0000000..5531739
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_LIST_H
+#define __DLB2_OSDEP_LIST_H
+
+#include <rte_tailq.h>
+
+struct dlb2_list_entry {
+	TAILQ_ENTRY(dlb2_list_entry) node;
+};
+
+/* Dummy - just a struct definition */
+TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
+
+/* =================
+ * TAILQ Supplements
+ * =================
+ */
+
+#ifndef TAILQ_FOREACH_ENTRY
+#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)		\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter)						\
+		&& (ptr = container_of(iter, typeof(*(ptr)), name)); \
+	    (iter) = TAILQ_NEXT((iter), node))
+#endif
+
+#ifndef TAILQ_FOREACH_ENTRY_SAFE
+#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)	\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter) &&						\
+		(ptr = container_of(iter, typeof(*(ptr)), name)) &&\
+		((tvar) = TAILQ_NEXT((iter), node), 1);	\
+	    (iter) = (tvar))
+#endif
+
+/***********************/
+/*** List operations ***/
+/***********************/
+
+/**
+ * dlb2_list_init_head() - initialize the head of a list
+ * @head: list head
+ */
+static inline void dlb2_list_init_head(struct dlb2_list_head *head)
+{
+	TAILQ_INIT(head);
+}
+
+/**
+ * dlb2_list_add() - add an entry to a list
+ * @head: list head
+ * @entry: new list entry
+ */
+static inline void
+dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
+{
+	TAILQ_INSERT_TAIL(head, entry, node);
+}
+
+/**
+ * dlb2_list_del() - delete an entry from a list
+ * @entry: list entry
+ * @head: list head
+ */
+static inline void dlb2_list_del(struct dlb2_list_head *head,
+				 struct dlb2_list_entry *entry)
+{
+	TAILQ_REMOVE(head, entry, node);
+}
+
+/**
+ * dlb2_list_empty() - check if a list is empty
+ * @head: list head
+ *
+ * Return:
+ * Returns 1 if empty, 0 if not.
+ */
+static inline int dlb2_list_empty(struct dlb2_list_head *head)
+{
+	return TAILQ_EMPTY(head);
+}
+
+/**
+ * dlb2_list_splice() - splice a list
+ * @src_head: list to be added
+ * @ head: where src_head will be inserted
+ */
+static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
+				    struct dlb2_list_head *head)
+{
+	TAILQ_CONCAT(head, src_head, node);
+}
+
+/**
+ * DLB2_LIST_HEAD() - retrieve the head of the list
+ * @head: list head
+ * @type: type of the list variable
+ * @name: name of the list field within the containing struct
+ */
+#define DLB2_LIST_HEAD(head, type, name)                       \
+	(TAILQ_FIRST(&head) ?					\
+		container_of(TAILQ_FIRST(&head), type, name) :	\
+		NULL)
+
+/**
+ * DLB2_LIST_FOR_EACH() - iterate over a list
+ * @head: list head
+ * @ptr: pointer to struct containing a struct list
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ */
+#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
+	TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
+
+/**
+ * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
+ * an element is removed from the list while processing it.
+ * @ptr: pointer to struct containing a struct list
+ * @ptr_tmp: pointer to struct containing a struct list (temporary)
+ * @head: list head
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ * @iter_tmp: iterator variable (temporary)
+ */
+#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
+	TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
+
+#endif /*  __DLB2_OSDEP_LIST_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
new file mode 100644
index 0000000..0a48f7e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_TYPES_H
+#define __DLB2_OSDEP_TYPES_H
+
+#include <linux/types.h>
+
+#include <inttypes.h>
+#include <ctype.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Types for user mode PF PMD */
+typedef uint8_t         u8;
+typedef int8_t          s8;
+typedef uint16_t        u16;
+typedef int16_t         s16;
+typedef uint32_t        u32;
+typedef int32_t         s32;
+typedef uint64_t        u64;
+
+#define __iomem
+
+/* END types for user mode PF PMD */
+
+#endif /* __DLB2_OSDEP_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
new file mode 100644
index 0000000..43ecad4
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
@@ -0,0 +1,2527 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "dlb2_osdep_types.h"
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+	struct {
+		u32 isr_pend : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+	struct {
+		u32 vf0_reset_in_progress : 1;
+		u32 vf1_reset_in_progress : 1;
+		u32 vf2_reset_in_progress : 1;
+		u32 vf3_reset_in_progress : 1;
+		u32 vf4_reset_in_progress : 1;
+		u32 vf5_reset_in_progress : 1;
+		u32 vf6_reset_in_progress : 1;
+		u32 vf7_reset_in_progress : 1;
+		u32 vf8_reset_in_progress : 1;
+		u32 vf9_reset_in_progress : 1;
+		u32 vf10_reset_in_progress : 1;
+		u32 vf11_reset_in_progress : 1;
+		u32 vf12_reset_in_progress : 1;
+		u32 vf13_reset_in_progress : 1;
+		u32 vf14_reset_in_progress : 1;
+		u32 vf15_reset_in_progress : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+	struct {
+		u32 vec_mask : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+	struct {
+		u32 func_vf_bar_dis : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+	struct {
+		u32 total_vas : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+	struct {
+		u32 total_dir_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+	struct {
+		u32 total_ldb_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+	struct {
+		u32 total_dir_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+	struct {
+		u32 total_ldb_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+	struct {
+		u32 total_dir_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+	struct {
+		u32 total_ldb_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 meas : 1;
+		u32 debug : 7;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 cq_int_rearm : 1;
+		u32 dsi_error : 1;
+		u32 rsvd0 : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 rsvd0 : 3;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+	struct {
+		u32 qid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+	struct {
+		u32 vqid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+	struct {
+		u32 qid : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 debug : 8;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 isz : 1;
+		u32 dsi_error : 1;
+		u32 dlbrsvd : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 vf_synd0_parity : 1;
+		u32 vf_synd1_parity : 1;
+		u32 vf_synd2_parity : 1;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+	struct {
+		u32 sn_cfg_v : 1;
+		u32 fid_cfg_v : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS  0
+#define DLB2_CQ_ISR_MODE_MSI  1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI  3
+union dlb2_sys_ldb_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+	struct {
+		u32 keep_pf_ppid : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+	struct {
+		u32 illegal_hcw : 1;
+		u32 illegal_pp : 1;
+		u32 illegal_pasid : 1;
+		u32 illegal_qid : 1;
+		u32 disabled_qid : 1;
+		u32 illegal_ldb_qid_cfg : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+	struct {
+		u32 msix_0_ack : 1;
+		u32 msix_1_ack : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+	struct {
+		u32 msix_0_passthru : 1;
+		u32 msix_1_passthru : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED     0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+	struct {
+		u32 mode : 1;
+		u32 poll_mode : 1;
+		u32 poll_mask : 1;
+		u32 poll_lock : 1;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+	struct {
+		u32 cq : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 alarm : 1;
+		u32 cwd : 1;
+		u32 vf_pf_mb : 1;
+		u32 rsvd0 : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+	struct {
+		u32 qid_fid_limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+	struct {
+		u32 compress_code : 3;
+		u32 rsvd0 : 29;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+	(DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+	struct {
+		u32 count : 15;
+		u32 rsvd0 : 17;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+	struct {
+		u32 sn : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+	struct {
+		u32 mode : 3;
+		u32 slot : 4;
+		u32 rsvz0 : 1;
+		u32 grp : 1;
+		u32 rsvz1 : 1;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+	struct {
+		u32 en : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+	struct {
+		u32 write_pointer : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+	struct {
+		u32 base : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+	struct {
+		u32 limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+	struct {
+		u32 pop_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+	struct {
+		u32 push_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+	struct {
+		u32 depth : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+	struct {
+		u32 write_pointer : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 dlb_cor_alarm_enable : 1;
+		u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+		u32 cfg_64bytes_qe_dir_cq_mode : 1;
+		u32 pad_write_ldb : 1;
+		u32 pad_write_dir : 1;
+		u32 pad_first_write_ldb : 1;
+		u32 pad_first_write_dir : 1;
+		u32 rsvz0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+	struct {
+		u32 egress_credit_status_empty : 1;
+		u32 egress_credit_status_afull : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+		u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+		u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+		u32 chp_rop_pipe_credit_status_empty : 1;
+		u32 chp_rop_pipe_credit_status_afull : 1;
+		u32 qed_to_cq_pipe_credit_status_empty : 1;
+		u32 qed_to_cq_pipe_credit_status_afull : 1;
+		u32 egress_lsp_token_credit_status_empty : 1;
+		u32 egress_lsp_token_credit_status_afull : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 rsvz0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+	struct {
+		u32 sn_mode_0 : 3;
+		u32 rszv0 : 5;
+		u32 sn_mode_1 : 3;
+		u32 rszv1 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+	struct {
+		u32 unit_single_step_mode : 1;
+		u32 rr_en : 1;
+		u32 rszv0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+	struct {
+		u32 prio : 24;
+		u32 v : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+	struct {
+		u32 qid_p0 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p1 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p2 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p3 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+	struct {
+		u32 qid_p4 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p5 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p6 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p7 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+	struct {
+		u32 token_depth_select : 4;
+		u32 disable_wb_opt : 1;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+	struct {
+		u32 token_count : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+	struct {
+		u32 thresh : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+	(DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+	(DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+	struct {
+		u32 depth : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+	struct {
+		u32 cq : 8;
+		u32 qidix : 3;
+		u32 value : 1;
+		u32 nalb_haswork_v : 1;
+		u32 rlist_haswork_v : 1;
+		u32 slist_haswork_v : 1;
+		u32 inflight_ok_v : 1;
+		u32 aqed_nfull_v : 1;
+		u32 rsvz0 : 15;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+	struct {
+		u32 transfer : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+	struct {
+		u32 bw_range : 9;
+		u32 rsvz0 : 22;
+		u32 no_extra_credit : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+	struct {
+		u32 disab_atq_empty_arb : 1;
+		u32 inc_tok_unit_idle : 1;
+		u32 disab_rlist_pri : 1;
+		u32 inc_cmp_unit_idle : 1;
+		u32 rsvz0 : 2;
+		u32 dir_single_op : 1;
+		u32 dir_half_bw : 1;
+		u32 dir_single_out : 1;
+		u32 dir_disab_multi : 1;
+		u32 atq_single_op : 1;
+		u32 atq_half_bw : 1;
+		u32 atq_single_out : 1;
+		u32 atq_disab_multi : 1;
+		u32 dirrpl_single_op : 1;
+		u32 dirrpl_half_bw : 1;
+		u32 dirrpl_single_out : 1;
+		u32 lbrpl_single_op : 1;
+		u32 lbrpl_half_bw : 1;
+		u32 lbrpl_single_out : 1;
+		u32 ldb_single_op : 1;
+		u32 ldb_half_bw : 1;
+		u32 ldb_disab_multi : 1;
+		u32 atm_single_sch : 1;
+		u32 atm_single_cmp : 1;
+		u32 ldb_ce_tog_arb : 1;
+		u32 rsvz1 : 1;
+		u32 smon0_valid_sel : 2;
+		u32 smon0_value_sel : 1;
+		u32 smon0_compare_sel : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+	struct {
+		u32 chp_pf_reset_done : 1;
+		u32 rop_pf_reset_done : 1;
+		u32 lsp_pf_reset_done : 1;
+		u32 nalb_pf_reset_done : 1;
+		u32 ap_pf_reset_done : 1;
+		u32 dp_pf_reset_done : 1;
+		u32 qed_pf_reset_done : 1;
+		u32 dqed_pf_reset_done : 1;
+		u32 aqed_pf_reset_done : 1;
+		u32 sys_pf_reset_done : 1;
+		u32 pf_reset_active : 1;
+		u32 flrsm_state : 7;
+		u32 rsvd0 : 13;
+		u32 dlb_proc_reset_done : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+	struct {
+		u32 chp_pipeidle : 1;
+		u32 rop_pipeidle : 1;
+		u32 lsp_pipeidle : 1;
+		u32 nalb_pipeidle : 1;
+		u32 ap_pipeidle : 1;
+		u32 dp_pipeidle : 1;
+		u32 qed_pipeidle : 1;
+		u32 dqed_pipeidle : 1;
+		u32 aqed_pipeidle : 1;
+		u32 sys_pipeidle : 1;
+		u32 chp_unit_idle : 1;
+		u32 rop_unit_idle : 1;
+		u32 lsp_unit_idle : 1;
+		u32 nalb_unit_idle : 1;
+		u32 ap_unit_idle : 1;
+		u32 dp_unit_idle : 1;
+		u32 qed_unit_idle : 1;
+		u32 dqed_unit_idle : 1;
+		u32 aqed_unit_idle : 1;
+		u32 sys_unit_idle : 1;
+		u32 rsvd1 : 4;
+		u32 mstr_cfg_ring_idle : 1;
+		u32 mstr_cfg_mstr_idle : 1;
+		u32 mstr_flr_clkreq_b : 1;
+		u32 mstr_proc_idle : 1;
+		u32 mstr_proc_idle_masked : 1;
+		u32 rsvd0 : 2;
+		u32 dlb_func_idle : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+	struct {
+		u32 prochot : 1;
+		u32 pgcb_dlb_idle : 1;
+		u32 pgcb_dlb_pg_rdy_ack_b : 1;
+		u32 pmsm_pgcb_req_b : 1;
+		u32 pgbc_pmc_pg_req_b : 1;
+		u32 pmc_pgcb_pg_ack_b : 1;
+		u32 pmc_pgcb_fet_en_b : 1;
+		u32 pgcb_fet_en_b : 1;
+		u32 rsvz0 : 1;
+		u32 rsvz1 : 1;
+		u32 fuse_force_on : 1;
+		u32 fuse_proc_disable : 1;
+		u32 rsvz2 : 1;
+		u32 rsvz3 : 1;
+		u32 pm_fsm_d0tod3_ok : 1;
+		u32 pm_fsm_d3tod0_ok : 1;
+		u32 dlb_in_d3 : 1;
+		u32 rsvz4 : 7;
+		u32 pmsm : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+	struct {
+		u32 disable : 1;
+		u32 rsvz0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+	struct {
+		u32 isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+	struct {
+		u32 pf_isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+	struct {
+		u32 isr_pend : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+	struct {
+		u32 reset_in_progress : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+	struct {
+		u32 vf_msi_isr : 32;
+	} field;
+	u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
new file mode 100644
index 0000000..503fdf3
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
@@ -0,0 +1,1913 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_osdep_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_start_domain() - start a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: start domain arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int dlb2_hw_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp,
+			 bool vdev_request,
+			 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_request,
+		    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding unmap procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_finish_map_qid_procedures() - finish any pending map procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding map procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ *					interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+			  int port_id,
+			  bool is_ldb,
+			  bool vdev_request,
+			  unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+					 u32 *ldb_interrupts,
+					 u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+				 u32 *ldb_interrupts,
+				 u32 *dir_interrupts);
+
+/**
+ * dlb2_read_vf_intr_status() - read the VF interrupt status register
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function can be called from a VF's interrupt handler to determine
+ * which interrupts have fired. The first 31 bits correspond to CQ interrupt
+ * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
+ *
+ * Return:
+ * Returns a bit vector indicating which interrupt vectors are active.
+ */
+u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_intr_status() - ack VF interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function ACKs a VF's interrupts. Its interrupts argument should be the
+ * value returned by dlb2_read_vf_intr_status().
+ */
+void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function clears the VF's MSI interrupt pending register. Its interrupts
+ * argument should be contain the MSI vectors to ACK. For example, if MSI MME
+ * is in mode 0, then one bit 0 should ever be set.
+ */
+void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When done processing the PF mailbox request, this function unsets
+ * the PF's mailbox ISR register.
+ */
+void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
+
+/**
+ * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
+ *					vdevs
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the vdev->PF ISR fires, this function can be called to determine which
+ * vdev(s) are requesting service. This bitvector must be passed to
+ * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
+ * vdevs.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested service.
+ */
+u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ *
+ * When done processing all VF mailbox requests, this function unsets the VF's
+ * mailbox ISR register.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
+ *				    FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the VF FLR ISR fires, this function can be called to determine which
+ * VF(s) are requesting FLRs. This bitvector must passed to
+ * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
+ */
+u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF FLR requests, this function unsets the VF's FLR
+ * ISR register.
+ */
+void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF requests, this function communicates to the
+ * hardware that processing is complete.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
+			     u32 mbox_bitvec,
+			     u32 flr_bitvec);
+
+/**
+ * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the watchdog timer interrupt cause registers to
+ * determine which port(s) had a watchdog timeout, and notifies the
+ * application(s) that own the port(s).
+ */
+void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads and logs the alarm syndrome, then acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_request,
+			      unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of resources in use by the PF or a VF. It
+ * fills in the fields that args points to, except the following:
+ * - max_contiguous_atomic_inflights
+ * - max_contiguous_hist_list_entries
+ * - max_contiguous_ldb_credits
+ * - max_contiguous_dir_credits
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
+				   struct dlb2_get_num_resources_args *arg,
+				   bool vdev_request,
+				   unsigned int vdev_id);
+
+/**
+ * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
+ *				       the PF
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function sends a VF->PF mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the PF has processed
+ * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
+ * when the PF has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
+ *				 request
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * processing a VF->PF mailbox request. It should only be called after sending
+ * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vf_flr_complete() - check the status of a VF FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * executing the VF FLR. It should only be called after setting the VF's FLR
+ * bit.
+ */
+bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
+ *					vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function sends a PF->vdev mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the vdev has
+ * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
+ * determine when the vdev has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
+ *			       request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function returns a boolean indicating whether the vdev has finished
+ * processing a PF->vdev mailbox request. It should only be called after
+ * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
+			     unsigned int vf_id,
+			     void *data,
+			     int len);
+
+/**
+ * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
+			       unsigned int vf_id,
+			       void *data,
+			       int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_reset_vdev() - reset the hardware owned by a virtual device
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function resets the hardware owned by a vdev, by resetting the vdev's
+ * domains one by one.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function returns whether or not the vdev's resource assignments are
+ * locked. If locked, no resources can be added to or subtracted from the
+ * group.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_lock_vdev() - lock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sets a flag indicating that the vdev is using its resources.
+ * When the vdev is locked, its resource assignment cannot be changed.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_unlock_vdev() - unlock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function unlocks the vdev's resource assignment, allowing it to be
+ * modified.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of scheduling domains to assign to this vdev
+ *
+ * This function assigns num scheduling domains to the specified vdev. If the
+ * vdev already has domains assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB queues to assign to this vdev
+ *
+ * This function assigns num LDB queues to the specified vdev. If the vdev
+ * already has LDB queues assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports to the specified vdev. If the vdev
+ * already has LDB ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @cos: class-of-service ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports from class-of-service cos to the
+ * specified vdev. If the vdev already has LDB ports from this class-of-service
+ * assigned, this existing assignment is adjusted accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
+				   u32 id,
+				   u32 cos,
+				   u32 num);
+
+/**
+ * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR ports to assign to this vdev
+ *
+ * This function assigns num DIR ports to the specified vdev. If the vdev
+ * already has DIR ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB credit credits to assign to this vdev
+ *
+ * This function assigns num LDB credit to the specified vdev. If the vdev
+ * already has LDB credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR credits to assign to this vdev
+ *
+ * This function assigns num DIR credit to the specified vdev. If the vdev
+ * already has DIR credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of history list entries to assign to this vdev
+ *
+ * This function assigns num history list entries to the specified vdev. If the
+ * vdev already has history list entries assigned, this existing assignment is
+ * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
+ * this function may fail if a sufficiently large contiguous chunk is not
+ * available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of atomic inflights to assign to this vdev
+ *
+ * This function assigns num atomic inflights to the specified vdev. If the vdev
+ * already has atomic inflights assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of entries, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function takes any resources currently assigned to the vdev and
+ * reassigns them to the PF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_notify_vf() - send an alarm to a VF
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID
+ * @notification: notification
+ *
+ * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
+ *
+ * Return:
+ * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
+ */
+int dlb2_notify_vf(struct dlb2_hw *hw,
+		   unsigned int vf_id,
+		   u32 notification);
+
+/**
+ * dlb2_vdev_in_use() - query whether a virtual device is in use
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sends a mailbox request to the vdev to query whether the vdev
+ * is in use.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
+ * an internal error occurs.
+ */
+int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+enum dlb2_virt_mode {
+	DLB2_VIRT_NONE,
+	DLB2_VIRT_SRIOV,
+	DLB2_VIRT_SIOV,
+
+	/* NUM_DLB2_VIRT_MODES must be last */
+	NUM_DLB2_VIRT_MODES,
+};
+
+/**
+ * dlb2_hw_set_virt_mode() - set the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: either none, SR-IOV, or Scalable IOV.
+ *
+ * This function sets the virtualization mode of the device. This controls
+ * whether the device uses a software or hardware mailbox.
+ *
+ * This should be called by the PF driver when either SR-IOV or Scalable IOV is
+ * selected as the virtualization mechanism, and by the VF/VDEV driver during
+ * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
+ *
+ * Errors:
+ * EINVAL - Invalid mode.
+ */
+int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
+
+/**
+ * dlb2_hw_get_virt_mode() - get the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function gets the virtualization mode of the device.
+ */
+enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_register_sw_mbox() - register a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
+ * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
+ * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
+ * @inject_arg: user argument for pf2vdev_inject callback.
+ *
+ * When Scalable IOV is enabled, the VDCM must register a software mailbox for
+ * every virtual device during vdev creation.
+ *
+ * This function notifies the driver to use a software mailbox using the
+ * provided pointers, instead of the device's hardware mailbox. When the driver
+ * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
+ * go to the software mailbox instead of the hardware one. This is used in
+ * Scalable IOV virtualization.
+ */
+void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
+			      unsigned int vdev_id,
+			      u32 *vdev2pf_mbox,
+			      u32 *pf2vdev_mbox,
+			      void (*pf2vdev_inject)(void *),
+			      void *inject_arg);
+
+/**
+ * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function notifies the driver to stop using a previously registered
+ * software mailbox.
+ */
+void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ * @addr_lo: least-significant 32 bits of address.
+ * @data: 32 data bits.
+ *
+ * This sets up the CQ's IMS entry with the provided address and data values.
+ * This function should only be called if the device is configured for Scalable
+ * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
+ * not needed.
+ */
+void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb,
+				u32 addr_lo,
+				u32 data);
+
+/**
+ * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ *
+ * This clears the CQ's IMS entry, reverting it to its reset state.
+ */
+void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb);
+
+/**
+ * dlb2_hw_register_pasid() - register a vdev's PASID
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @pasid: the vdev's PASID.
+ *
+ * This function stores the user-supplied PASID, and uses it when configuring
+ * the vdev's CQs.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+int dlb2_hw_register_pasid(struct dlb2_hw *hw,
+			   unsigned int vdev_id,
+			   unsigned int pasid);
+
+/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vf_request,
+				unsigned int vf_id);
+
+/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ *	to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ *	port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ *	    cause the total bandwidth across all classes of service to exceed
+ *	    100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+enum dlb2_wd_tmo {
+	/* 40s watchdog timeout */
+	DLB2_WD_TMO_40S,
+	/* 10s watchdog timeout */
+	DLB2_WD_TMO_10S,
+	/* 1s watchdog timeout */
+	DLB2_WD_TMO_1S,
+
+	/* Must be last */
+	NUM_DLB2_WD_TMOS,
+};
+
+/**
+ * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
+ *	caller-specified timeout.
+ * @hw: dlb2_hw handle for a particular device.
+ * @tmo: watchdog timeout.
+ *
+ * This function should be called during device initialization and after reset.
+ * The watchdog timer interrupt must also be enabled per-CQ, using either
+ * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid timeout.
+ */
+int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
+
+/**
+ * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid directed port ID.
+ */
+int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid load-balanced port ID.
+ */
+int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *	ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * load-balanced CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_ldb_cq_interrupt().
+ */
+void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical directed port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * directed CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_dir_cq_interrupt().
+ */
+void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+#endif /* __DLB2_RESOURCE_H */
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
new file mode 100644
index 0000000..3c51cab
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -0,0 +1,621 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "base/dlb2_resource.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_regs.h"
+#include "dlb2_main.h"
+#include "../dlb2_user.h"
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+
+#define PF_ID_ZERO 0	/* PF ONLY! */
+#define NO_OWNER_VF 0	/* PF ONLY! */
+#define NOT_VF_REQ false /* PF ONLY! */
+
+#define DLB2_PCI_CFG_SPACE_SIZE 256
+#define DLB2_PCI_CAP_POINTER 0x34
+#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
+#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
+#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
+#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
+#define DLB2_PCI_EXT_CAP_ID_ERR 1
+#define DLB2_PCI_ERR_UNCOR_MASK 8
+#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
+
+#define DLB2_PCI_EXP_DEVCTL 8
+#define DLB2_PCI_LNKCTL 16
+#define DLB2_PCI_SLTCTL 24
+#define DLB2_PCI_RTCTL 28
+#define DLB2_PCI_EXP_DEVCTL2 40
+#define DLB2_PCI_LNKCTL2 48
+#define DLB2_PCI_SLTCTL2 56
+#define DLB2_PCI_CMD 4
+#define DLB2_PCI_X_CMD 2
+#define DLB2_PCI_EXP_DEVSTA 10
+#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
+#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
+
+#define DLB2_PCI_CAP_ID_EXP       0x10
+#define DLB2_PCI_CAP_ID_MSIX      0x11
+#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
+#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
+#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
+
+#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
+#define DLB2_PCI_PRI_ALLOC_REQ           0xC
+#define DLB2_PCI_PRI_CTRL                0x4
+#define DLB2_PCI_MSIX_FLAGS              0x2
+#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
+#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
+#define DLB2_PCI_ERR_ROOT_STATUS         0x30
+#define DLB2_PCI_ERR_COR_STATUS          0x10
+#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
+#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
+#define DLB2_PCI_ACS_CAP                 0x4
+#define DLB2_PCI_ACS_CTRL                0x6
+#define DLB2_PCI_ACS_SV                  0x1
+#define DLB2_PCI_ACS_RR                  0x4
+#define DLB2_PCI_ACS_CR                  0x8
+#define DLB2_PCI_ACS_UF                  0x10
+#define DLB2_PCI_ACS_EC                  0x20
+
+/* Stubs: Allow building partial probe patch */
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	int ret = 0;
+	RTE_SET_USED(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+/* End stubs */
+
+static int
+dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint32_t hdr;
+	size_t sz;
+	int pos;
+
+	pos = DLB2_PCI_CFG_SPACE_SIZE;
+	sz = sizeof(hdr);
+
+	while (pos > 0xFF) {
+		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
+			return -1;
+
+		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
+			return pos;
+
+		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint8_t pos;
+	int ret;
+	uint16_t hdr;
+
+	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
+	pos &= 0xFC;
+
+	if (ret != 1)
+		return -1;
+
+	while (pos > 0x3F) {
+		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
+		if (ret != 2)
+			return -1;
+
+		if (DLB2_PCI_CAP_ID(hdr) == id)
+			return pos;
+
+		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
+			return -1;
+
+		pos = DLB2_PCI_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+	rte_spinlock_init(&dlb2_dev->resource_mutex);
+	rte_spinlock_init(&dlb2_dev->measurement_lock);
+
+	return 0;
+}
+
+static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
+{
+	u32 retries = 0;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
+		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+		union dlb2_cfg_mstr_cfg_pm_status pm_st;
+		u32 addr;
+
+		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+			break;
+
+		rte_delay_ms(1);
+	};
+
+	if (retries == DLB2_READY_RETRY_LIMIT) {
+		printf("[%s()] wait for device ready timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+struct dlb2_dev *
+dlb2_probe(struct rte_pci_device *pdev)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret = 0;
+
+	DLB2_INFO(dlb2_dev, "probe\n");
+
+	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
+			      RTE_CACHE_LINE_SIZE);
+
+	if (dlb2_dev == NULL) {
+		ret = -ENOMEM;
+		goto dlb2_dev_malloc_fail;
+	}
+
+	/* PCI Bus driver has already mapped bar space into process.
+	 * Save off our IO register and FUNC addresses.
+	 */
+
+	/* BAR 0 */
+	if (pdev->mem_resource[0].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
+	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.func_kva,
+		  (void *)dlb2_dev->hw.func_phys_addr,
+		  (void *)(pdev->mem_resource[0].len));
+
+	/* BAR 2 */
+	if (pdev->mem_resource[2].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
+	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.csr_kva,
+		  (void *)dlb2_dev->hw.csr_phys_addr,
+		  (void *)(pdev->mem_resource[2].len));
+
+	dlb2_dev->pdev = pdev;
+
+	/* PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb2_pf_enable_pm(dlb2_dev);
+
+	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb2_pf_reset(dlb2_dev);
+	if (ret)
+		goto dlb2_reset_fail;
+
+	ret = dlb2_pf_init_driver_state(dlb2_dev);
+	if (ret)
+		goto init_driver_state_fail;
+
+	ret = dlb2_resource_init(&dlb2_dev->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	return dlb2_dev;
+
+resource_init_fail:
+	dlb2_resource_free(&dlb2_dev->hw);
+init_driver_state_fail:
+dlb2_reset_fail:
+pci_mmap_bad_addr:
+wait_for_device_ready_fail:
+	rte_free(dlb2_dev);
+dlb2_dev_malloc_fail:
+	rte_errno = ret;
+	return NULL;
+}
+
+int
+dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
+{
+	int ret = 0;
+	int i = 0;
+	uint32_t dword[16];
+	uint16_t cmd;
+	off_t off;
+
+	uint16_t dev_ctl_word;
+	uint16_t dev_ctl2_word;
+	uint16_t lnk_word;
+	uint16_t lnk_word2;
+	uint16_t slt_word;
+	uint16_t slt_word2;
+	uint16_t rt_ctl_word;
+	uint32_t pri_reqs_dword;
+	uint16_t pri_ctrl_word;
+
+	int pcie_cap_offset;
+	int pri_cap_offset;
+	int msix_cap_offset;
+	int err_cap_offset;
+	int acs_cap_offset;
+	int wait_count;
+
+	uint16_t devsta_busy_word;
+	uint16_t devctl_word;
+
+	struct rte_pci_device *pdev = dlb2_dev->pdev;
+
+	/* Save PCI config state */
+
+	for (i = 0; i < 16; i++) {
+		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
+			return ret;
+	}
+
+	pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
+
+	if (pcie_cap_offset < 0) {
+		printf("[%s()] failed to find the pcie capability\n",
+		       __func__);
+		return pcie_cap_offset;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
+		dev_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
+		lnk_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
+		slt_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_RTCTL;
+	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
+		rt_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
+		dev_ctl2_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
+		lnk_word2 = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
+		slt_word2 = 0;
+
+	off = DLB2_PCI_EXT_CAP_ID_PRI;
+	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (pri_cap_offset >= 0) {
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
+			pri_reqs_dword = 0;
+	}
+
+	/* clear the PCI command register before issuing the FLR */
+
+	off = DLB2_PCI_CMD;
+	cmd = 0;
+	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+		printf("[%s()] failed to write the pci command\n",
+		       __func__);
+		return ret;
+	}
+
+	/* issue the FLR */
+	for (wait_count = 0; wait_count < 4; wait_count++) {
+		int sleep_time;
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
+		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to read the pci device status\n",
+			       __func__);
+			return ret;
+		}
+
+		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
+			break;
+
+		sleep_time = (1 << (wait_count)) * 100;
+		rte_delay_ms(sleep_time);
+	}
+
+	if (wait_count == 4) {
+		printf("[%s()] wait for pci pending transactions timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to read the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
+
+	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to write the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	rte_delay_ms(100);
+
+	/* Restore PCI config state */
+
+	if (pcie_cap_offset >= 0) {
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie device control at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_RTCTL;
+		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	if (pri_cap_offset >= 0) {
+		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
+
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
+		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ERR;
+	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (err_cap_offset >= 0) {
+		uint32_t tmp;
+
+		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	for (i = 16; i > 0; i--) {
+		off = (i - 1) * 4;
+		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_CMD;
+	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
+		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+			printf("[%s()] failed to write the pci command\n",
+			       __func__);
+			return ret;
+		}
+	}
+
+	msix_cap_offset = dlb2_pci_find_capability(pdev,
+						   DLB2_PCI_CAP_ID_MSIX);
+	if (msix_cap_offset >= 0) {
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
+			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ACS;
+	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (acs_cap_offset >= 0) {
+		uint16_t acs_cap, acs_ctrl, acs_mask;
+		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
+		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
+			acs_cap = 0;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
+		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
+		acs_ctrl |= (acs_cap & acs_mask);
+
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
+		acs_mask |= DLB2_PCI_ACS_EC;
+		acs_ctrl &= ~acs_mask;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
new file mode 100644
index 0000000..f082a94
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -0,0 +1,98 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+#endif
+
+#include "base/dlb2_hw_types.h"
+#include "../dlb2_user.h"
+
+#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
+
+struct dlb2_dev;
+
+struct dlb2_port_memory {
+	struct dlb2_list_head list;
+	void *cq_base;
+	bool valid;
+};
+
+struct dlb2_dev {
+	struct rte_pci_device *pdev;
+	struct dlb2_hw hw;
+	/* struct list_head list; */
+	struct device *dlb2_device;
+	bool domain_reset_failed;
+	/* The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	rte_spinlock_t resource_mutex;
+	rte_spinlock_t measurement_lock;
+	bool worker_launched;
+	u8 revision;
+};
+
+struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
+int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
+int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+
+#endif /* __DLB2_MAIN_H */
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
new file mode 100644
index 0000000..ae4d785
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -0,0 +1,196 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_devargs.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_pci.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+#include "dlb2_main.h"
+#include "base/dlb2_hw_types.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_resource.h"
+
+static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
+
+/* Stubs: Allow building partial probe patch */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(arg);
+	RTE_SET_USED(vdev_req);
+	RTE_SET_USED(vdev_id);
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+/* End stubs */
+
+static void
+dlb2_pf_iface_fn_ptrs_init(void)
+{
+/* flexible iface fn ptr assignments will go here */
+}
+
+/* PCI DEV HOOKS */
+static int
+dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
+{
+	int ret = 0;
+	struct rte_pci_device *pci_dev;
+	struct dlb2_devargs dlb2_args = {
+		.socket_id = rte_socket_id(),
+		.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+		.num_dir_credits_override = -1,
+		.qid_depth_thresholds = { {0} },
+		.cos_id = DLB2_COS_DEFAULT
+	};
+	struct dlb2_eventdev *dlb2;
+
+	DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
+		     eventdev->data->dev_id, eventdev->data->socket_id);
+
+	dlb2_pf_iface_fn_ptrs_init();
+
+	pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
+
+		/* Probe the DLB2 PF layer */
+		dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
+
+		if (dlb2->qm_instance.pf_dev == NULL) {
+			DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
+				     rte_errno);
+			ret = -rte_errno;
+			goto dlb2_probe_failed;
+		}
+
+		/* Were we invoked with runtime parameters? */
+		if (pci_dev->device.devargs) {
+			ret = dlb2_parse_params(pci_dev->device.devargs->args,
+						pci_dev->device.devargs->name,
+						&dlb2_args);
+			if (ret) {
+				DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
+					     ret, rte_errno);
+				goto dlb2_probe_failed;
+			}
+		}
+
+		ret = dlb2_primary_eventdev_probe(eventdev,
+						  event_dlb2_pf_name,
+						  &dlb2_args);
+	} else {
+		ret = dlb2_secondary_eventdev_probe(eventdev,
+						    event_dlb2_pf_name);
+	}
+	if (ret)
+		goto dlb2_probe_failed;
+
+	DLB2_LOG_INFO("DLB2 PF Probe success\n");
+
+	return 0;
+
+dlb2_probe_failed:
+
+	DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
+
+	return ret;
+}
+
+#define EVENTDEV_INTEL_VENDOR_ID 0x8086
+
+static const struct rte_pci_id pci_id_dlb2_map[] = {
+	{
+		RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
+			       PCI_DEVICE_ID_INTEL_DLB2_PF)
+	},
+	{
+		.vendor_id = 0,
+	},
+};
+
+static int
+event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
+		     struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
+					     sizeof(struct dlb2_eventdev),
+					     dlb2_eventdev_pci_init,
+					     event_dlb2_pf_name);
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+}
+
+static int
+event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_remove(pci_dev, NULL);
+
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+
+}
+
+static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
+	.id_table = pci_id_dlb2_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe = event_dlb2_pci_probe,
+	.remove = event_dlb2_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
-- 
2.6.4

[dpdk-dev] [PATCH v5 07/23] event/dlb2: add flexible interface

[Timothy McDaniel]

This commit introduces the flexible interface. This
interface allows the core code to operate in PF mode (direct
hardware access) or bifurcated mode (hardware configured via
kernel driver). This driver currently only supports PF mode
but bifurcated mode will be added in a future DPDK patch-set.
Note that the flexible interface is not used for data path
operations, and thus there are no performance concerns
related to the use of function pointers.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_iface.c | 28 ++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.h | 29 +++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build  |  1 +
 3 files changed, 58 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h

diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
new file mode 100644
index 0000000..0fc9991
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+
+#include "dlb2_priv.h"
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+void (*dlb2_iface_low_level_io_init)(void);
+
+int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+				     uint8_t *revision);
+
+void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+				   enum dlb2_cq_poll_modes *mode);
+
+int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
new file mode 100644
index 0000000..4fb416e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_IFACE_H_
+#define _DLB2_IFACE_H_
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+extern void (*dlb2_iface_low_level_io_init)(void);
+
+extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+					    uint8_t *revision);
+
+extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+					  enum dlb2_cq_poll_modes *mode);
+
+extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
+
+#endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 1eab9b9..491e76d 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -8,6 +8,7 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
 endif
 
 sources = files('dlb2.c',
+		'dlb2_iface.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c'
 )
-- 
2.6.4

[dpdk-dev] [PATCH v5 08/23] event/dlb2: add probe-time hardware init

[Timothy McDaniel]

This commit adds probe-time low level hardware
initialization.  It also adds probe-time init for both
primary and secondary DPDK processes.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 188 +++++++++++++++++++-
 drivers/event/dlb2/meson.build             |   3 +-
 drivers/event/dlb2/pf/base/dlb2_resource.c | 274 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  23 +--
 drivers/event/dlb2/pf/dlb2_main.h          |   1 -
 drivers/event/dlb2/pf/dlb2_pf.c            |  78 ++++++--
 6 files changed, 523 insertions(+), 44 deletions(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 1b11979..16653db 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -31,6 +31,7 @@
 #include <rte_string_fns.h>
 
 #include "dlb2_priv.h"
+#include "dlb2_iface.h"
 #include "dlb2_inline_fns.h"
 
 /*
@@ -40,10 +41,108 @@
 #if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
 #error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
 #endif
+static struct rte_event_dev_info evdev_dlb2_default_info = {
+	.driver_name = "", /* probe will set */
+	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
+	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
+	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
+#else
+	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
+#endif
+	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
+	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
+	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
+	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
+	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
+	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+	.max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
+	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
+			  RTE_EVENT_DEV_CAP_EVENT_QOS |
+			  RTE_EVENT_DEV_CAP_BURST_MODE |
+			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
+			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
+			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
+};
 
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/* override defaults with value(s) provided on command line */
+static void
+dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
+				 int *qid_depth_thresholds)
+{
+	int q;
+
+	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
+		if (qid_depth_thresholds[q] != 0)
+			dlb2->ev_queues[q].depth_threshold =
+				qid_depth_thresholds[q];
+	}
+}
+
+static int
+dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
+	int ret;
+
+	/* Query driver resources provisioned for this device */
+
+	ret = dlb2_iface_get_num_resources(handle,
+					   &dlb2->hw_rsrc_query_results);
+	if (ret) {
+		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
+		return ret;
+	}
+
+	/* Complete filling in device resource info returned to evdev app,
+	 * overriding any default values.
+	 * The capabilities (CAPs) were set at compile time.
+	 */
+
+	evdev_dlb2_default_info.max_event_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	evdev_dlb2_default_info.max_event_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	evdev_dlb2_default_info.max_num_events =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	/* Save off values used when creating the scheduling domain. */
+
+	handle->info.num_sched_domains =
+		dlb2->hw_rsrc_query_results.num_sched_domains;
+
+	handle->info.hw_rsrc_max.nb_events_limit =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	handle->info.hw_rsrc_max.num_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues +
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.num_ldb_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	handle->info.hw_rsrc_max.num_ldb_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	handle->info.hw_rsrc_max.num_dir_ports =
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.reorder_window_size =
+		dlb2->hw_rsrc_query_results.num_hist_list_entries;
+
+	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
+
+	return 0;
+}
+
 #define DLB2_BASE_10 10
 
 static int
@@ -235,14 +334,71 @@ set_qid_depth_thresh(const char *key __rte_unused,
 	return 0;
 }
 
+static void
+dlb2_entry_points_init(struct rte_eventdev *dev)
+{
+	RTE_SET_USED(dev);
+
+	/* Eventdev PMD entry points */
+}
+
 int
 dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    const char *name,
 			    struct dlb2_devargs *dlb2_args)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
-	RTE_SET_USED(dlb2_args);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	dlb2->event_dev = dev; /* backlink */
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	dlb2->max_num_events_override = dlb2_args->max_num_events;
+	dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
+	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
+					    &dlb2->revision);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_hardware_init(&dlb2->qm_instance);
+
+	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
+			     err);
+		return err;
+	}
+
+	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	dlb2_init_queue_depth_thresholds(dlb2,
+					 dlb2_args->qid_depth_thresholds.val);
 
 	return 0;
 }
@@ -251,8 +407,30 @@ int
 dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
 			      const char *name)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 491e76d..28fef3f 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -10,7 +10,8 @@ endif
 sources = files('dlb2.c',
 		'dlb2_iface.c',
 		'pf/dlb2_main.c',
-		'pf/dlb2_pf.c'
+		'pf/dlb2_pf.c',
+		'pf/base/dlb2_resource.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
new file mode 100644
index 0000000..6de8b95
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_mbox.h"
+#include "dlb2_osdep.h"
+#include "dlb2_osdep_bitmap.h"
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+	int i;
+
+	dlb2_list_init_head(&domain->used_ldb_queues);
+	dlb2_list_init_head(&domain->used_dir_pq_pairs);
+	dlb2_list_init_head(&domain->avail_ldb_queues);
+	dlb2_list_init_head(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+	int i;
+
+	dlb2_list_init_head(&rsrc->avail_domains);
+	dlb2_list_init_head(&rsrc->used_domains);
+	dlb2_list_init_head(&rsrc->avail_ldb_queues);
+	dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
+	arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
+	arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
+	arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb2_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	struct dlb2_list_entry *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. This is application
+	 * dependent, but the driver interleaves port IDs as much as possible
+	 * to reduce the likelihood of this. This initial allocation maximizes
+	 * the average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	/* Zero-out resource tracking data structures */
+	memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+	memset(&hw->pf, 0, sizeof(hw->pf));
+
+	dlb2_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+		dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+		dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_domains, list);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_ldb_queues, list);
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+		struct dlb2_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
+			      &port->func_list);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+				DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+	if (ret)
+		goto unwind;
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+					DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+		if (ret)
+			goto unwind;
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb2_resource_free(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+	r0.field.disable = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 3c51cab..210b3186 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -19,6 +19,7 @@
 #include "dlb2_main.h"
 #include "../dlb2_user.h"
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 
 #define PF_ID_ZERO 0	/* PF ONLY! */
@@ -72,27 +73,6 @@
 #define DLB2_PCI_ACS_UF                  0x10
 #define DLB2_PCI_ACS_EC                  0x20
 
-/* Stubs: Allow building partial probe patch */
-void dlb2_resource_free(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-int dlb2_resource_init(struct dlb2_hw *hw)
-{
-	int ret = 0;
-	RTE_SET_USED(hw);
-
-	return ret;
-}
-
-void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-/* End stubs */
-
 static int
 dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
 {
@@ -149,7 +129,6 @@ static int
 dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
 {
 	rte_spinlock_init(&dlb2_dev->resource_mutex);
-	rte_spinlock_init(&dlb2_dev->measurement_lock);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index f082a94..f3bee71 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -38,7 +38,6 @@ struct dlb2_dev {
 	 * hardware registers.
 	 */
 	rte_spinlock_t resource_mutex;
-	rte_spinlock_t measurement_lock;
 	bool worker_launched;
 	u8 revision;
 };
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index ae4d785..7d64309 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -32,6 +32,7 @@
 #include <rte_string_fns.h>
 
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 #include "dlb2_main.h"
 #include "base/dlb2_hw_types.h"
@@ -40,35 +41,82 @@
 
 static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
 
-/* Stubs: Allow building partial probe patch */
-int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
-			      struct dlb2_get_num_resources_args *arg,
-			      bool vdev_req,
-			      unsigned int vdev_id)
+static void
+dlb2_pf_low_level_io_init(void)
+{
+	int i;
+	/* Addresses will be initialized at port create */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		/* First directed ports */
+		dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
+
+		/* Now load balanced ports */
+		dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
+	}
+}
+
+static int
+dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
+{
+	RTE_SET_USED(handle);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+static int
+dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
+			   uint8_t *revision)
 {
-	RTE_SET_USED(hw);
-	RTE_SET_USED(arg);
-	RTE_SET_USED(vdev_req);
-	RTE_SET_USED(vdev_id);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	*revision = dlb2_dev->revision;
 
 	return 0;
 }
 
-void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+static void
+dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+	dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_num_resources_args *rsrcs)
 {
-	RTE_SET_USED(hw);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
 }
 
-void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+static int
+dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
+			 enum dlb2_cq_poll_modes *mode)
 {
-	RTE_SET_USED(hw);
+	RTE_SET_USED(handle);
+
+	*mode = DLB2_CQ_POLL_MODE_SPARSE;
+
+	return 0;
 }
-/* End stubs */
 
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
-/* flexible iface fn ptr assignments will go here */
+	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
+	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
+	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
+	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
+	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v5 09/23] event/dlb2: add xstats

[Timothy McDaniel]

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
Reviewed-by: Chen, Mike Ximing <mike.ximing.chen@intel.com>
---
 drivers/event/dlb2/dlb2.c        |   35 +-
 drivers/event/dlb2/dlb2_xstats.c | 1240 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build   |    1 +
 3 files changed, 1273 insertions(+), 3 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 16653db..925824e 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,6 +70,21 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/*
+ * DUMMY - added so that xstats path will compile/link.
+ * Will be replaced by real version in a subsequent
+ * patch.
+ */
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	RTE_SET_USED(dlb2);
+	RTE_SET_USED(queue);
+
+	return 0;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -337,9 +352,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
-	RTE_SET_USED(dev);
-
-	/* Eventdev PMD entry points */
+	/* Expose PMD's eventdev interface */
+	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dump             = dlb2_eventdev_dump,
+		.xstats_get       = dlb2_eventdev_xstats_get,
+		.xstats_get_names = dlb2_eventdev_xstats_get_names,
+		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
+		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+	};
+
+	dev->dev_ops = &dlb2_eventdev_entry_ops;
 }
 
 int
@@ -391,6 +413,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Complete xtstats runtime initialization */
+	err = dlb2_xstats_init(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
+		return err;
+	}
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
new file mode 100644
index 0000000..21391f7
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_xstats.c
@@ -0,0 +1,1240 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eventdev.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+enum dlb2_xstats_type {
+	/* common to device and port */
+	rx_ok,				/**< Receive an event */
+	rx_drop,                        /**< Error bit set in received QE */
+	rx_interrupt_wait,		/**< Wait on an interrupt */
+	rx_umonitor_umwait,		/**< Block using umwait */
+	tx_ok,				/**< Transmit an event */
+	total_polls,			/**< Call dequeue_burst */
+	zero_polls,			/**< Call dequeue burst and return 0 */
+	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
+	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
+	tx_nospc_inflight_max,		/**< Reach the new_event_threshold */
+	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
+	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
+	/* device specific */
+	nb_events_limit,
+	inflight_events,
+	ldb_pool_size,
+	dir_pool_size,
+	/* port specific */
+	tx_new,				/**< Send an OP_NEW event */
+	tx_fwd,				/**< Send an OP_FORWARD event */
+	tx_rel,				/**< Send an OP_RELEASE event */
+	tx_implicit_rel,		/**< Issue an implicit event release */
+	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED event */
+	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL event */
+	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event */
+	tx_sched_directed,		/**< Send a directed event */
+	tx_invalid,                     /**< Send an event with an invalid op */
+	outstanding_releases,		/**< # of releases a port owes */
+	max_outstanding_releases,	/**< max # of releases a port can owe */
+	rx_sched_ordered,		/**< Dequeue an ordered event */
+	rx_sched_unordered,		/**< Dequeue an unordered event */
+	rx_sched_atomic,		/**< Dequeue an atomic event */
+	rx_sched_directed,		/**< Dequeue an directed event */
+	rx_sched_invalid,               /**< Dequeue event sched type invalid */
+	/* common to port and queue */
+	is_configured,			/**< Port is configured */
+	is_load_balanced,		/**< Port is LDB */
+	hw_id,				/**< Hardware ID */
+	/* queue specific */
+	num_links,			/**< Number of ports linked */
+	sched_type,			/**< Queue sched type */
+	enq_ok,				/**< # events enqueued to the queue */
+	current_depth,			/**< Current queue depth */
+	depth_threshold,		/**< Programmed depth threshold */
+	depth_le50_threshold,
+	/**< Depth LE to 50% of the configured hardware threshold */
+	depth_gt50_le75_threshold,
+	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold */
+	depth_gt75_le100_threshold,
+	/**< Depth GT 75%. but LE to the configured hardware threshold */
+	depth_gt100_threshold
+	/**< Depth GT 100% of the configured hw threshold */
+};
+
+typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
+		uint16_t obj_idx, /* port or queue id */
+		enum dlb2_xstats_type stat, int extra_arg);
+
+enum dlb2_xstats_fn_type {
+	DLB2_XSTATS_FN_DEV,
+	DLB2_XSTATS_FN_PORT,
+	DLB2_XSTATS_FN_QUEUE
+};
+
+struct dlb2_xstats_entry {
+	struct rte_event_dev_xstats_name name;
+	uint64_t reset_value; /* an offset to be taken away to emulate resets */
+	enum dlb2_xstats_fn_type fn_id;
+	enum dlb2_xstats_type stat;
+	enum rte_event_dev_xstats_mode mode;
+	int extra_arg;
+	uint16_t obj_idx;
+	uint8_t reset_allowed; /* when set, this value can be reset */
+};
+
+/* Some device stats are simply a summation of the corresponding port values */
+static uint64_t
+dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
+			     int which_stat)
+{
+	int i;
+	uint64_t val = 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
+
+		if (!port->setup_done)
+			continue;
+
+		switch (which_stat) {
+		case rx_ok:
+			val += port->stats.traffic.rx_ok;
+			break;
+		case rx_drop:
+			val += port->stats.traffic.rx_drop;
+			break;
+		case rx_interrupt_wait:
+			val += port->stats.traffic.rx_interrupt_wait;
+			break;
+		case rx_umonitor_umwait:
+			val += port->stats.traffic.rx_umonitor_umwait;
+			break;
+		case tx_ok:
+			val += port->stats.traffic.tx_ok;
+			break;
+		case total_polls:
+			val += port->stats.traffic.total_polls;
+			break;
+		case zero_polls:
+			val += port->stats.traffic.zero_polls;
+			break;
+		case tx_nospc_ldb_hw_credits:
+			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
+			break;
+		case tx_nospc_dir_hw_credits:
+			val += port->stats.traffic.tx_nospc_dir_hw_credits;
+			break;
+		case tx_nospc_inflight_max:
+			val += port->stats.traffic.tx_nospc_inflight_max;
+			break;
+		case tx_nospc_new_event_limit:
+			val += port->stats.traffic.tx_nospc_new_event_limit;
+			break;
+		case tx_nospc_inflight_credits:
+			val += port->stats.traffic.tx_nospc_inflight_credits;
+			break;
+		default:
+			return -1;
+		}
+	}
+	return val;
+}
+
+static uint64_t
+get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
+	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	switch (type) {
+	case rx_ok:
+	case rx_drop:
+	case rx_interrupt_wait:
+	case rx_umonitor_umwait:
+	case tx_ok:
+	case total_polls:
+	case zero_polls:
+	case tx_nospc_ldb_hw_credits:
+	case tx_nospc_dir_hw_credits:
+	case tx_nospc_inflight_max:
+	case tx_nospc_new_event_limit:
+	case tx_nospc_inflight_credits:
+		return dlb2_device_traffic_stat_get(dlb2, type);
+	case nb_events_limit:
+		return dlb2->new_event_limit;
+	case inflight_events:
+		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
+	case ldb_pool_size:
+		return dlb2->num_ldb_credits;
+	case dir_pool_size:
+		return dlb2->num_dir_credits;
+	default: return -1;
+	}
+}
+
+static uint64_t
+get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
+
+	switch (type) {
+	case rx_ok: return ev_port->stats.traffic.rx_ok;
+
+	case rx_drop: return ev_port->stats.traffic.rx_drop;
+
+	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
+
+	case rx_umonitor_umwait:
+		return ev_port->stats.traffic.rx_umonitor_umwait;
+
+	case tx_ok: return ev_port->stats.traffic.tx_ok;
+
+	case total_polls: return ev_port->stats.traffic.total_polls;
+
+	case zero_polls: return ev_port->stats.traffic.zero_polls;
+
+	case tx_nospc_ldb_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
+
+	case tx_nospc_dir_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
+
+	case tx_nospc_inflight_max:
+		return ev_port->stats.traffic.tx_nospc_inflight_max;
+
+	case tx_nospc_new_event_limit:
+		return ev_port->stats.traffic.tx_nospc_new_event_limit;
+
+	case tx_nospc_inflight_credits:
+		return ev_port->stats.traffic.tx_nospc_inflight_credits;
+
+	case is_configured: return ev_port->setup_done;
+
+	case is_load_balanced: return !ev_port->qm_port.is_directed;
+
+	case hw_id: return ev_port->qm_port.id;
+
+	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
+
+	case tx_fwd: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
+
+	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
+
+	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
+
+	case tx_sched_ordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case tx_sched_unordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case tx_sched_atomic:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case tx_sched_directed:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case tx_invalid: return ev_port->stats.tx_invalid;
+
+	case outstanding_releases: return ev_port->outstanding_releases;
+
+	case max_outstanding_releases:
+		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	case rx_sched_ordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case rx_sched_unordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case rx_sched_atomic:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case rx_sched_directed:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
+
+	default: return -1;
+	}
+}
+
+static uint64_t
+dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
+{
+	int port = 0;
+	uint64_t tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
+
+	return tally;
+}
+
+static uint64_t
+dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
+{
+	int port = 0;
+	uint64_t enq_ok_tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
+
+	return enq_ok_tally;
+}
+
+static uint64_t
+get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_queue *ev_queue =
+		&dlb2->ev_queues[obj_idx];
+
+	switch (type) {
+	case is_configured: return ev_queue->setup_done;
+
+	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
+
+	case hw_id: return ev_queue->qm_queue.id;
+
+	case num_links: return ev_queue->num_links;
+
+	case sched_type: return ev_queue->qm_queue.sched_type;
+
+	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
+
+	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
+
+	case depth_threshold: return ev_queue->depth_threshold;
+
+	case depth_le50_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_LE50);
+
+	case depth_gt50_le75_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT50_LE75);
+
+	case depth_gt75_le100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT75_LE100);
+
+	case depth_gt100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT100);
+
+	default: return -1;
+	}
+}
+
+int
+dlb2_xstats_init(struct dlb2_eventdev *dlb2)
+{
+	/*
+	 * define the stats names and types. Used to build up the device
+	 * xstats array
+	 * There are multiple set of stats:
+	 *   - device-level,
+	 *   - per-port,
+	 *   - per-qid,
+	 *
+	 * For each of these sets, we have three parallel arrays, one for the
+	 * names, the other for the stat type parameter to be passed in the fn
+	 * call to get that stat. The third array allows resetting or not.
+	 * All these arrays must be kept in sync
+	 */
+	static const char * const dev_stats[] = {
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"nb_events_limit",
+		"inflight_events",
+		"ldb_pool_size",
+		"dir_pool_size",
+	};
+	static const enum dlb2_xstats_type dev_types[] = {
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		nb_events_limit,
+		inflight_events,
+		ldb_pool_size,
+		dir_pool_size,
+	};
+	/* Note: generated device stats are not allowed to be reset. */
+	static const uint8_t dev_reset_allowed[] = {
+		0, /* rx_ok */
+		0, /* rx_drop */
+		0, /* rx_interrupt_wait */
+		0, /* rx_umonitor_umwait */
+		0, /* tx_ok */
+		0, /* total_polls */
+		0, /* zero_polls */
+		0, /* tx_nospc_ldb_hw_credits */
+		0, /* tx_nospc_dir_hw_credits */
+		0, /* tx_nospc_inflight_max */
+		0, /* tx_nospc_new_event_limit */
+		0, /* tx_nospc_inflight_credits */
+		0, /* nb_events_limit */
+		0, /* inflight_events */
+		0, /* ldb_pool_size */
+		0, /* dir_pool_size */
+	};
+	static const char * const port_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"tx_new",
+		"tx_fwd",
+		"tx_rel",
+		"tx_implicit_rel",
+		"tx_sched_ordered",
+		"tx_sched_unordered",
+		"tx_sched_atomic",
+		"tx_sched_directed",
+		"tx_invalid",
+		"outstanding_releases",
+		"max_outstanding_releases",
+		"rx_sched_ordered",
+		"rx_sched_unordered",
+		"rx_sched_atomic",
+		"rx_sched_directed",
+		"rx_sched_invalid"
+	};
+	static const enum dlb2_xstats_type port_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		tx_new,
+		tx_fwd,
+		tx_rel,
+		tx_implicit_rel,
+		tx_sched_ordered,
+		tx_sched_unordered,
+		tx_sched_atomic,
+		tx_sched_directed,
+		tx_invalid,
+		outstanding_releases,
+		max_outstanding_releases,
+		rx_sched_ordered,
+		rx_sched_unordered,
+		rx_sched_atomic,
+		rx_sched_directed,
+		rx_sched_invalid
+	};
+	static const uint8_t port_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		1, /* rx_ok */
+		1, /* rx_drop */
+		1, /* rx_interrupt_wait */
+		1, /* rx_umonitor_umwait */
+		1, /* tx_ok */
+		1, /* total_polls */
+		1, /* zero_polls */
+		1, /* tx_nospc_ldb_hw_credits */
+		1, /* tx_nospc_dir_hw_credits */
+		1, /* tx_nospc_inflight_max */
+		1, /* tx_nospc_new_event_limit */
+		1, /* tx_nospc_inflight_credits */
+		1, /* tx_new */
+		1, /* tx_fwd */
+		1, /* tx_rel */
+		1, /* tx_implicit_rel */
+		1, /* tx_sched_ordered */
+		1, /* tx_sched_unordered */
+		1, /* tx_sched_atomic */
+		1, /* tx_sched_directed */
+		1, /* tx_invalid */
+		0, /* outstanding_releases */
+		0, /* max_outstanding_releases */
+		1, /* rx_sched_ordered */
+		1, /* rx_sched_unordered */
+		1, /* rx_sched_atomic */
+		1, /* rx_sched_directed */
+		1  /* rx_sched_invalid */
+	};
+
+	/* QID specific stats */
+	static const char * const qid_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"num_links",
+		"sched_type",
+		"enq_ok",
+		"current_depth",
+		"depth_threshold",
+		"depth_le50_threshold",
+		"depth_gt50_le75_threshold",
+		"depth_gt75_le100_threshold",
+		"depth_gt100_threshold",
+	};
+	static const enum dlb2_xstats_type qid_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		num_links,
+		sched_type,
+		enq_ok,
+		current_depth,
+		depth_threshold,
+		depth_le50_threshold,
+		depth_gt50_le75_threshold,
+		depth_gt75_le100_threshold,
+		depth_gt100_threshold,
+	};
+	static const uint8_t qid_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		0, /* num_links */
+		0, /* sched_type */
+		1, /* enq_ok */
+		0, /* current_depth */
+		0, /* depth_threshold */
+		1, /* depth_le50_threshold */
+		1, /* depth_gt50_le75_threshold */
+		1, /* depth_gt75_le100_threshold */
+		1, /* depth_gt100_threshold */
+	};
+
+	/* ---- end of stat definitions ---- */
+
+	/* check sizes, since a missed comma can lead to strings being
+	 * joined by the compiler.
+	 */
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
+
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_reset_allowed));
+
+	/* other vars */
+	const unsigned int count = RTE_DIM(dev_stats) +
+			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
+			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
+	unsigned int i, port, qid, stat_id = 0;
+
+	dlb2->xstats = rte_zmalloc_socket(NULL,
+			sizeof(dlb2->xstats[0]) * count, 0,
+			dlb2->qm_instance.info.socket_id);
+	if (dlb2->xstats == NULL)
+		return -ENOMEM;
+
+#define sname dlb2->xstats[stat_id].name.name
+	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
+		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
+			.fn_id = DLB2_XSTATS_FN_DEV,
+			.stat = dev_types[i],
+			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
+			.reset_allowed = dev_reset_allowed[i],
+		};
+		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
+	}
+	dlb2->xstats_count_mode_dev = stat_id;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
+		dlb2->xstats_offset_for_port[port] = stat_id;
+
+		uint32_t count_offset = stat_id;
+
+		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_PORT,
+				.obj_idx = port,
+				.stat = port_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_PORT,
+				.reset_allowed = port_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "port_%u_%s",
+				 port, port_stats[i]);
+		}
+
+		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_port = stat_id - dlb2->xstats_count_mode_dev;
+
+	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
+		uint32_t count_offset = stat_id;
+
+		dlb2->xstats_offset_for_qid[qid] = stat_id;
+
+		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_QUEUE,
+				.obj_idx = qid,
+				.stat = qid_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
+				.reset_allowed = qid_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "qid_%u_%s",
+				 qid, qid_stats[i]);
+		}
+
+		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_queue = stat_id -
+		(dlb2->xstats_count_mode_dev + dlb2->xstats_count_mode_port);
+#undef sname
+
+	dlb2->xstats_count = stat_id;
+
+	return 0;
+}
+
+void
+dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
+{
+	rte_free(dlb2->xstats);
+	dlb2->xstats_count = 0;
+}
+
+int
+dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstats_names,
+		unsigned int *ids, unsigned int size)
+{
+	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+	uint32_t start_offset = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			break;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			break;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
+		break;
+	default:
+		return -EINVAL;
+	};
+
+	if (xstats_mode_count > size || ids == NULL || xstats_names == NULL)
+		return xstats_mode_count;
+
+	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
+		if (dlb2->xstats[i].mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != dlb2->xstats[i].obj_idx)
+			continue;
+
+		xstats_names[xidx] = dlb2->xstats[i].name;
+		if (ids)
+			ids[xidx] = start_offset + xidx;
+		xidx++;
+	}
+	return xidx;
+}
+
+static int
+dlb2_xstats_update(struct dlb2_eventdev *dlb2,
+		enum rte_event_dev_xstats_mode mode,
+		uint8_t queue_port_id, const unsigned int ids[],
+		uint64_t values[], unsigned int n, const uint32_t reset)
+{
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			goto invalid_value;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			goto invalid_value;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		break;
+	default:
+		goto invalid_value;
+	};
+
+	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
+		dlb2_xstats_fn fn;
+
+		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != xs->obj_idx)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			goto invalid_value;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+
+		if (values)
+			values[xidx] = val;
+
+		if (xs->reset_allowed && reset)
+			xs->reset_value += val;
+
+		xidx++;
+	}
+
+	return xidx;
+
+invalid_value:
+	return -EINVAL;
+}
+
+int
+dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	const uint32_t reset = 0;
+
+	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
+				  reset);
+}
+
+uint64_t
+dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+				 const char *name, unsigned int *id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	dlb2_xstats_fn fn;
+
+	for (i = 0; i < dlb2->xstats_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (strncmp(xs->name.name, name,
+			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
+			if (id != NULL)
+				*id = i;
+
+			switch (xs->fn_id) {
+			case DLB2_XSTATS_FN_DEV:
+				fn = get_dev_stat;
+				break;
+			case DLB2_XSTATS_FN_PORT:
+				fn = get_port_stat;
+				break;
+			case DLB2_XSTATS_FN_QUEUE:
+				fn = get_queue_stat;
+				break;
+			default:
+				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
+					  xs->fn_id);
+				return (uint64_t)-1;
+			}
+
+			return fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+		}
+	}
+	if (id != NULL)
+		*id = (uint32_t)-1;
+	return (uint64_t)-1;
+}
+
+static void
+dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
+			uint32_t num)
+{
+	uint32_t i;
+	dlb2_xstats_fn fn;
+
+	for (i = start; i < start + num; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (!xs->reset_allowed)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			return;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
+		xs->reset_value = val;
+	}
+}
+
+static int
+dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
+			const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_QUEUE,
+					queue_id, ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	if (ids == NULL)
+		dlb2_xstats_reset_range(dlb2,
+			dlb2->xstats_offset_for_qid[queue_id],
+			dlb2->xstats_count_per_qid[queue_id]);
+
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
+		       const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+	int offset = dlb2->xstats_offset_for_port[port_id];
+	int nb_stat = dlb2->xstats_count_per_port[port_id];
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_PORT, port_id,
+					ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
+		      uint32_t nb_ids)
+{
+	uint32_t i;
+
+	if (ids) {
+		for (i = 0; i < nb_ids; i++) {
+			uint32_t id = ids[i];
+
+			if (id >= dlb2->xstats_count_mode_dev)
+				return -EINVAL;
+			dlb2_xstats_reset_range(dlb2, id, 1);
+		}
+	} else {
+		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
+			dlb2_xstats_reset_range(dlb2, i, 1);
+	}
+
+	return 0;
+}
+
+int
+dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+			   enum rte_event_dev_xstats_mode mode,
+			   int16_t queue_port_id,
+			   const uint32_t ids[],
+			   uint32_t nb_ids)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	uint32_t i;
+
+	/* handle -1 for queue_port_id here, looping over all ports/queues */
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
+			return -EINVAL;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+				if (dlb2_xstats_reset_port(dlb2, i,
+							   ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
+			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
+						   ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
+				if (dlb2_xstats_reset_queue(dlb2, i,
+							    ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
+			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
+						    ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	};
+
+	return 0;
+}
+
+void
+dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	int i;
+
+	if (f == NULL) {
+		printf("Invalid file pointer\n");
+		return;
+	}
+
+	if (dev == NULL) {
+		fprintf(f, "Invalid event device\n");
+		return;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (dlb2 == NULL) {
+		fprintf(f, "DLB2 Event device cannot be dumped!\n");
+		return;
+	}
+
+	if (!dlb2->configured)
+		fprintf(f, "DLB2 Event device is not configured\n");
+
+	handle = &dlb2->qm_instance;
+
+	fprintf(f, "================\n");
+	fprintf(f, "DLB2 Device Dump\n");
+	fprintf(f, "================\n");
+
+	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
+		dlb2->umwait_allowed ? "yes" : "no");
+
+	/* Generic top level device information */
+
+	fprintf(f, "device is configured and run state =");
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		fprintf(f, "STOPPED\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
+		fprintf(f, "STOPPING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
+		fprintf(f, "STARTING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
+		fprintf(f, "STARTED\n");
+	else
+		fprintf(f, "UNEXPECTED\n");
+
+	fprintf(f, "domain ID=%u, socket_id=%u, evdev=%p\n",
+		handle->domain_id, handle->info.socket_id, dlb2->event_dev);
+
+	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
+		dlb2->num_dir_ports, dlb2->num_dir_queues);
+
+	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
+		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
+
+	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
+		handle->cfg.resources.num_atomic_inflights,
+		handle->cfg.resources.num_hist_list_entries);
+
+	fprintf(f, "results from most recent hw resource query:\n");
+
+	fprintf(f, "\tnum_sched_domains = %u\n",
+		dlb2->hw_rsrc_query_results.num_sched_domains);
+
+	fprintf(f, "\tnum_ldb_queues = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_queues);
+
+	fprintf(f, "\tnum_ldb_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_ports);
+
+	fprintf(f, "\tnum_dir_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_ports);
+
+	fprintf(f, "\tnum_atomic_inflights = %u\n",
+		dlb2->hw_rsrc_query_results.num_atomic_inflights);
+
+	fprintf(f, "\tnum_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.num_hist_list_entries);
+
+	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
+
+	fprintf(f, "\tnum_ldb_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_credits);
+
+	fprintf(f, "\tnum_dir_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_credits);
+
+	/* Port level information */
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
+		int j;
+
+		if (!p->enq_configured)
+			fprintf(f, "Port_%d is not configured\n", i);
+
+		fprintf(f, "Port_%d\n", i);
+		fprintf(f, "=======\n");
+
+		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
+			p->id, p->setup_done);
+
+		fprintf(f, "\tconfig state=%d, port state=%d\n",
+			p->qm_port.config_state, p->qm_port.state);
+
+		fprintf(f, "\tport is %s\n",
+			p->qm_port.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\toutstanding releases=%u\n",
+			p->outstanding_releases);
+
+		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
+			p->inflight_max, p->inflight_credits);
+
+		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
+			p->credit_update_quanta, p->implicit_release);
+
+		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (p->link[j].valid)
+				fprintf(f, "id=%u prio=%u ",
+					p->link[j].queue_id,
+					p->link[j].priority);
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
+
+		fprintf(f, "\tcached_ldb_credits=%u\n",
+			p->qm_port.cached_ldb_credits);
+
+		fprintf(f, "\tldb_credits = %u\n",
+			p->qm_port.ldb_credits);
+
+		fprintf(f, "\tcached_dir_credits = %u\n",
+			p->qm_port.cached_dir_credits);
+
+		fprintf(f, "\tdir_credits = %u\n",
+			p->qm_port.dir_credits);
+
+		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
+			p->qm_port.gen_bit,
+			p->qm_port.cq_idx,
+			p->qm_port.cq_depth);
+
+		fprintf(f, "\tinterrupt armed=%d\n",
+			p->qm_port.int_armed);
+
+		fprintf(f, "\tPort statistics\n");
+
+		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
+			p->stats.traffic.rx_ok);
+
+		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
+			p->stats.traffic.rx_drop);
+
+		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
+			p->stats.traffic.rx_interrupt_wait);
+
+		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
+			p->stats.traffic.rx_umonitor_umwait);
+
+		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
+			p->stats.traffic.tx_ok);
+
+		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
+			p->stats.traffic.total_polls);
+
+		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
+			p->stats.traffic.zero_polls);
+
+		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_ldb_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_dir_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_max);
+
+		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_new_event_limit);
+
+		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_credits);
+
+		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
+
+		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
+
+		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
+
+		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
+			p->stats.tx_implicit_rel);
+
+		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
+			p->stats.tx_invalid);
+
+		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
+			p->stats.rx_sched_invalid);
+	}
+
+	/* Queue level information */
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
+		int j, k;
+
+		if (!q->setup_done)
+			fprintf(f, "Queue_%d is not configured\n", i);
+
+		fprintf(f, "Queue_%d\n", i);
+		fprintf(f, "========\n");
+
+		fprintf(f, "\tevqueue_%u is set up\n", q->id);
+
+		fprintf(f, "\tqueue is %s\n",
+			q->qm_queue.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
+
+		for (j = 0; j < dlb2->num_ports; j++) {
+			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
+
+			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++) {
+				if (p->link[k].valid &&
+				    p->link[k].queue_id == q->id)
+					fprintf(f, "id=%u prio=%u ",
+						p->id, p->link[k].priority);
+			}
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\tcurrent depth: %u events\n",
+			dlb2_get_queue_depth(dlb2, q));
+
+		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
+			q->qm_queue.num_qid_inflights, q->qm_queue.sched_type);
+	}
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 28fef3f..dc16c96 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files('dlb2.c',
 		'dlb2_iface.c',
+		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c'
-- 
2.6.4

[dpdk-dev] [PATCH v5 10/23] event/dlb2: add infos get and configure

[Timothy McDaniel]

Add support for configuring the DLB2 hardware.
In particular, this patch configures the DLB2
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  116 +
 drivers/event/dlb2/dlb2.c                  |  313 +++
 drivers/event/dlb2/dlb2_iface.c            |    5 +
 drivers/event/dlb2/dlb2_iface.h            |    4 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 3237 ++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |   15 +
 drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
 7 files changed, 3734 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index e3091c9..5f6c486 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -32,9 +32,125 @@ detailed understanding of the hardware, but these details are important when
 writing high-performance code. This section describes the places where the
 eventdev API and DLB2 misalign.
 
+Scheduling Domain Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are 32 scheduling domainis the DLB2.
+When one is configured, it allocates load-balanced and
+directed queues, ports, credits, and other hardware resources. Some
+resource allocations are user-controlled -- the number of queues, for example
+-- and others, like credit pools (one directed and one load-balanced pool per
+scheduling domain), are not.
+
+The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
+setup argument and the per-port ``new_event_threshold`` argument apply as
+defined in the eventdev header file. The limit is applied to all enqueues,
+regardless of whether it will consume a directed or load-balanced credit.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Reconfiguration
+~~~~~~~~~~~~~~~
+
+The Eventdev API allows one to reconfigure a device, its ports, and its queues
+by first stopping the device, calling the configuration function(s), then
+restarting the device. The DLB2 does not support configuring an individual queue
+or port without first reconfiguring the entire device, however, so there are
+certain reconfiguration sequences that are valid in the eventdev API but not
+supported by the PMD.
+
+Specifically, the PMD supports the following configuration sequence:
+1. Configure and start the device
+2. Stop the device
+3. (Optional) Reconfigure the device
+4. (Optional) If step 3 is run:
+
+   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
+   b. The reconfigured port(s) lose their previous queue links.
+
+5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
+6. Restart the device. If the device is reconfigured in step 3 but one or more
+   of its ports or queues are not, the PMD will apply their previous
+   configuration (including port->queue links) at this time.
+
+The PMD does not support the following configuration sequences:
+1. Configure and start the device
+2. Stop the device
+3. Setup queue or setup port
+4. Start the device
+
+This sequence is not supported because the event device must be reconfigured
+before its ports or queues can be.
+
+Atomic Inflights Allocation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
+inflight event in a temporary buffer that is divided among load-balanced
+queues. If a queue's atomic buffer storage fills up, this can result in
+head-of-line-blocking. For example:
+
+- An LDB queue allocated N atomic buffer entries
+- All N entries are filled with events from flow X, which is pinned to CQ 0.
+
+Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
+scheduled. The likelihood of this case depends on the eventdev configuration,
+traffic behavior, event processing latency, potential for a worker to be
+interrupted or otherwise delayed, etc.
+
+By default, the PMD allocates 16 buffer entries for each load-balanced queue,
+which provides an even division across all 128 queues but potentially wastes
+buffer space (e.g. if not all queues are used, or aren't used for atomic
+scheduling).
+
+The PMD provides a dev arg to override the default per-queue allocation. To
+increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,atm_inflights=64
+
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 925824e..6798e66 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -85,6 +85,25 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
 	return 0;
 }
 
+static void
+dlb2_free_qe_mem(struct dlb2_port *qm_port)
+{
+	if (qm_port == NULL)
+		return;
+
+	rte_free(qm_port->qe4);
+	qm_port->qe4 = NULL;
+
+	rte_free(qm_port->int_arm_qe);
+	qm_port->int_arm_qe = NULL;
+
+	rte_free(qm_port->consume_qe);
+	qm_port->consume_qe = NULL;
+
+	rte_memzone_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz);
+	dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz = NULL;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -350,10 +369,304 @@ set_qid_depth_thresh(const char *key __rte_unused,
 }
 
 static void
+dlb2_eventdev_info_get(struct rte_eventdev *dev,
+		       struct rte_event_dev_info *dev_info)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret;
+
+	ret = dlb2_hw_query_resources(dlb2);
+	if (ret) {
+		const struct rte_eventdev_data *data = dev->data;
+
+		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+			     ret, data->dev_id);
+		/* fn is void, so fall through and return values set up in
+		 * probe
+		 */
+	}
+
+	/* Add num resources currently owned by this domain.
+	 * These would become available if the scheduling domain were reset due
+	 * to the application recalling eventdev_configure to *reconfigure* the
+	 * domain.
+	 */
+	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
+	evdev_dlb2_default_info.max_event_queues += dlb2->num_ldb_queues;
+	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
+
+	evdev_dlb2_default_info.max_event_queues =
+		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
+			RTE_EVENT_MAX_QUEUES_PER_DEV);
+
+	evdev_dlb2_default_info.max_num_events =
+		RTE_MIN(evdev_dlb2_default_info.max_num_events,
+			dlb2->max_num_events_override);
+
+	*dev_info = evdev_dlb2_default_info;
+}
+
+static int
+dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
+			    const struct dlb2_hw_rsrcs *resources_asked)
+{
+	int ret = 0;
+	struct dlb2_create_sched_domain_args *cfg;
+
+	if (resources_asked == NULL) {
+		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
+		ret = EINVAL;
+		goto error_exit;
+	}
+
+	/* Map generic qm resources to dlb2 resources */
+	cfg = &handle->cfg.resources;
+
+	/* DIR ports and queues */
+
+	cfg->num_dir_ports = resources_asked->num_dir_ports;
+
+	cfg->num_dir_credits = resources_asked->num_dir_credits;
+
+	/* LDB queues */
+
+	cfg->num_ldb_queues = resources_asked->num_ldb_queues;
+
+	/* LDB ports */
+
+	cfg->cos_strict = 0; /* Best effort */
+	cfg->num_cos_ldb_ports[0] = 0;
+	cfg->num_cos_ldb_ports[1] = 0;
+	cfg->num_cos_ldb_ports[2] = 0;
+	cfg->num_cos_ldb_ports[3] = 0;
+
+	switch (handle->cos_id) {
+	case DLB2_COS_0:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[0] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_1:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[1] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_2:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[2] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_3:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[3] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_DEFAULT:
+		/* all ldb ports are don't care ports from a cos perspective */
+		cfg->num_ldb_ports =
+			resources_asked->num_ldb_ports;
+		break;
+	}
+
+	cfg->num_ldb_credits =
+		resources_asked->num_ldb_credits;
+
+	cfg->num_atomic_inflights =
+		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
+		cfg->num_ldb_queues;
+
+	cfg->num_hist_list_entries = resources_asked->num_ldb_ports *
+		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d\n",
+		     cfg->num_ldb_queues,
+		     resources_asked->num_ldb_ports,
+		     cfg->num_dir_ports,
+		     cfg->num_atomic_inflights,
+		     cfg->num_hist_list_entries,
+		     cfg->num_ldb_credits,
+		     cfg->num_dir_credits);
+
+	/* Configure the QM */
+
+	ret = dlb2_iface_sched_domain_create(handle, cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: domain create failed, ret = %d, extra status: %s\n",
+			     ret,
+			     dlb2_error_strings[cfg->response.status]);
+
+		goto error_exit;
+	}
+
+	handle->domain_id = cfg->response.id;
+	handle->cfg.configured = true;
+
+error_exit:
+
+	return ret;
+}
+
+static void
+dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	enum dlb2_configuration_state config_state;
+	int i, j;
+
+	dlb2_iface_domain_reset(dlb2);
+
+	/* Free all dynamically allocated port memory */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
+
+	/* If reconfiguring, mark the device's queues and ports as "previously
+	 * configured." If the user doesn't reconfigure them, the PMD will
+	 * reapply their previous configuration when the device is started.
+	 */
+	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
+		DLB2_NOT_CONFIGURED;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		dlb2->ev_ports[i].qm_port.config_state = config_state;
+		/* Reset setup_done so ports can be reconfigured */
+		dlb2->ev_ports[i].setup_done = false;
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			dlb2->ev_ports[i].link[j].mapped = false;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++)
+		dlb2->ev_queues[i].qm_queue.config_state = config_state;
+
+	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
+		dlb2->ev_queues[i].setup_done = false;
+
+	dlb2->num_ports = 0;
+	dlb2->num_ldb_ports = 0;
+	dlb2->num_dir_ports = 0;
+	dlb2->num_queues = 0;
+	dlb2->num_ldb_queues = 0;
+	dlb2->num_dir_queues = 0;
+	dlb2->configured = false;
+}
+
+/* Note: 1 QM instance per QM device, QM instance/device == event device */
+static int
+dlb2_eventdev_configure(const struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
+	const struct rte_eventdev_data *data = dev->data;
+	const struct rte_event_dev_config *config = &data->dev_conf;
+	int ret;
+
+	/* If this eventdev is already configured, we must release the current
+	 * scheduling domain before attempting to configure a new one.
+	 */
+	if (dlb2->configured) {
+		dlb2_hw_reset_sched_domain(dev, true);
+
+		ret = dlb2_hw_query_resources(dlb2);
+		if (ret) {
+			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+				     ret, data->dev_id);
+			return ret;
+		}
+	}
+
+	if (config->nb_event_queues > rsrcs->num_queues) {
+		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_queues,
+			     rsrcs->num_queues);
+		return -EINVAL;
+	}
+	if (config->nb_event_ports > (rsrcs->num_ldb_ports
+			+ rsrcs->num_dir_ports)) {
+		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_ports,
+			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
+		return -EINVAL;
+	}
+	if (config->nb_events_limit > rsrcs->nb_events_limit) {
+		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_events_limit,
+			     rsrcs->nb_events_limit);
+		return -EINVAL;
+	}
+
+	if (config->event_dev_cfg & RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT)
+		dlb2->global_dequeue_wait = false;
+	else {
+		uint32_t timeout32;
+
+		dlb2->global_dequeue_wait = true;
+
+		/* note size mismatch of timeout vals in eventdev lib. */
+		timeout32 = config->dequeue_timeout_ns;
+
+		dlb2->global_dequeue_wait_ticks =
+			timeout32 * (rte_get_timer_hz() / 1E9);
+	}
+
+	/* Does this platform support umonitor/umwait? */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
+		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
+			DLB2_LOG_ERR("invalid value (%d) for RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
+				     RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
+			return -EINVAL;
+		}
+		dlb2->umwait_allowed = true;
+	}
+
+	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
+	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs->num_dir_ports;
+	/* 1 dir queue per dir port */
+	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports;
+
+	/* Scale down nb_events_limit by 4 for directed credits, since there
+	 * are 4x as many load-balanced credits.
+	 */
+	rsrcs->num_ldb_credits = 0;
+	rsrcs->num_dir_credits = 0;
+
+	if (rsrcs->num_ldb_queues)
+		rsrcs->num_ldb_credits = config->nb_events_limit;
+	if (rsrcs->num_dir_ports)
+		rsrcs->num_dir_credits = config->nb_events_limit / 4;
+	if (dlb2->num_dir_credits_override != -1)
+		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
+
+	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
+		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
+		return -ENODEV;
+	}
+
+	dlb2->new_event_limit = config->nb_events_limit;
+	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
+
+	/* Save number of ports/queues for this event dev */
+	dlb2->num_ports = config->nb_event_ports;
+	dlb2->num_queues = config->nb_event_queues;
+	dlb2->num_dir_ports = rsrcs->num_dir_ports;
+	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
+	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
+	dlb2->num_dir_queues = dlb2->num_dir_ports;
+	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
+	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
+	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
+	dlb2->max_dir_credits = rsrcs->num_dir_credits;
+
+	dlb2->configured = true;
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dev_infos_get    = dlb2_eventdev_info_get,
+		.dev_configure    = dlb2_eventdev_configure,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 0fc9991..a829b9b 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -26,3 +26,8 @@ int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
+
+int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				struct dlb2_create_sched_domain_args *args);
+
+void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4fb416e..6663dab 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -26,4 +26,8 @@ extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
 
+extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				 struct dlb2_create_sched_domain_args *args);
+
+extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 6de8b95..63a68a6 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -12,6 +12,27 @@
 #include "dlb2_regs.h"
 #include "dlb2_resource.h"
 
+#include "../../dlb2_priv.h"
+#include "../../dlb2_inline_fns.h"
+
+#define DLB2_DOM_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, domain_list, iter)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, func_list, iter)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, domain_list, it, it_tmp)
+
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, func_list, it, it_tmp)
+
 static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
 {
 	int i;
@@ -272,3 +293,3219 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
 
 	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
 }
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+	union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+	r0.field.count = domain->num_ldb_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+	r1.field.count = domain->num_dir_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+		       struct dlb2_function_resources *rsrcs,
+		       u32 domain_id,
+		       u32 cos_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	RTE_SET_USED(iter);
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+				   typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				   struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_ports,
+				   u32 cos_id,
+				   struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_ldb_port *port;
+
+		port = dlb2_get_next_ldb_port(hw, rsrcs,
+					      domain->id.phys_id, cos_id);
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_ports[cos_id],
+			      &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_ports[cos_id],
+			      &port->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_create_sched_domain_args *args,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      num,
+						      i,
+						      resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+					ret = __dlb2_attach_ldb_ports(hw,
+								      rsrcs,
+								      domain,
+								      1,
+								      cos_id,
+								      resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret < 0)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      1,
+						      j,
+						      resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 u32 num_ports,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+					   typeof(*port));
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_dir_pq_pairs, &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_dir_pq_pairs, &port->domain_list);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+					struct dlb2_hw_domain *domain,
+					u32 num_atomic_inflights,
+					struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+				     struct dlb2_hw_domain *domain,
+				     u32 num_hist_list_entries,
+				     struct dlb2_cmd_response *resp)
+{
+	struct dlb2_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb2_bitmap_find_set_bit_range(bitmap,
+						      num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+				  struct dlb2_function_resources *rsrcs,
+				  struct dlb2_hw_domain *domain,
+				  u32 num_queues,
+				  struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb2_ldb_queue *queue;
+
+		queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+					    typeof(*queue));
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_queues, &queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_queues, &queue->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+			     struct dlb2_function_resources *rsrcs,
+			     struct dlb2_hw_domain *domain,
+			     struct dlb2_create_sched_domain_args *args,
+			     struct dlb2_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb2_attach_ldb_queues(hw,
+				     rsrcs,
+				     domain,
+				     args->num_ldb_queues,
+				     resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_ports(hw,
+				    rsrcs,
+				    domain,
+				    args,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_ports(hw,
+				    rsrcs,
+				    domain,
+				    args->num_dir_ports,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_credits(rsrcs,
+				      domain,
+				      args->num_ldb_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_credits(rsrcs,
+				      domain,
+				      args->num_dir_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+						   domain,
+						   args->num_hist_list_entries,
+						   resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_atomic_inflights(rsrcs,
+					   domain,
+					   args->num_atomic_inflights,
+					   resp);
+	if (ret < 0)
+		return ret;
+
+	dlb2_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+				  struct dlb2_create_sched_domain_args *args,
+				  struct dlb2_cmd_response *resp)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb2_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+				  struct dlb2_create_sched_domain_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		    args->num_ldb_queues);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		    args->num_ldb_ports);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		    args->num_cos_ldb_ports[0]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		    args->cos_strict);
+	DLB2_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		    args->num_dir_ports);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		    args->num_atomic_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		    args->num_hist_list_entries);
+	DLB2_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		    args->num_ldb_credits);
+	DLB2_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		    args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ *	domain and its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+	if (ret)
+		return ret;
+
+	domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available domains\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (domain->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_domains contains configured domains.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	dlb2_init_domain_rsrc_lists(domain);
+
+	ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to verify args.\n",
+			    __func__);
+
+		return ret;
+	}
+
+	dlb2_list_del(&rsrcs->avail_domains, &domain->func_list);
+
+	dlb2_list_add(&rsrcs->used_domains, &domain->func_list);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+	DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+			     struct dlb2_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb2_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	int ret;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_disable(hw, port);
+
+		ret = dlb2_drain_dir_cq(hw, port);
+		if (ret < 0)
+			return ret;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_dir_pq_pair *queue)
+{
+	union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *queue)
+{
+	return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+				      struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		dlb2_movdir64b(pp_addr, hcw);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int ret, i;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			if (toggle_port)
+				dlb2_ldb_port_cq_disable(hw, port);
+
+			ret = dlb2_drain_ldb_cq(hw, port);
+			if (ret < 0)
+				return ret;
+
+			if (toggle_port)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+
+	return 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_atm_active r1;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_ldb_queue *queue)
+{
+	return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb2_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+					   struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to unmap domain queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = true;
+
+			dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+			   u32 id,
+			   bool vdev_req,
+			   unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iter1) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter2)
+			if (queue->id.virt_id == id)
+				return queue;
+	}
+
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue, iter1)
+		if (queue->id.virt_id == id)
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+						      u32 id,
+						      bool vdev_req,
+						      unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iteration;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	RTE_SET_USED(iteration);
+
+	if (id >= DLB2_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iteration)
+		if (domain->id.virt_id == id)
+			return domain;
+
+	return NULL;
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot,
+					   enum dlb2_qid_map_state new_state)
+{
+	enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb2_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB2_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB2_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB2_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB2_HW_DBG(hw,
+		    "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB2_HW_ERR(hw,
+		    "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+				enum dlb2_qid_map_state state,
+				int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+				      enum dlb2_qid_map_state state,
+				      struct dlb2_ldb_queue *queue,
+				      int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+					      struct dlb2_hw_domain *domain,
+					      struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain,
+					     struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+						struct dlb2_ldb_port *port,
+						int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 0;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 1;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+					struct dlb2_ldb_port *p,
+					struct dlb2_ldb_queue *q,
+					u8 priority)
+{
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_lsp_cq2qid0 r1;
+	union dlb2_atm_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix_00 r3;
+	union dlb2_lsp_qid2cqidix2_00 r4;
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+	    !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+	r0.field.v |= 1 << i;
+	r0.field.prio |= (priority & 0x7) << i * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+	else
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		r1.field.qid_p0 = q->id.phys_id;
+	if (i == 1 || i == 5)
+		r1.field.qid_p1 = q->id.phys_id;
+	if (i == 2 || i == 6)
+		r1.field.qid_p2 = q->id.phys_id;
+	if (i == 3 || i == 7)
+		r1.field.qid_p3 = q->id.phys_id;
+
+	if (i < 4)
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+	else
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r4.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+						  p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		r2.field.cq_p0 |= 1 << i;
+		r3.field.cq_p0 |= 1 << i;
+		r4.field.cq_p0 |= 1 << i;
+		break;
+
+	case 1:
+		r2.field.cq_p1 |= 1 << i;
+		r3.field.cq_p1 |= 1 << i;
+		r4.field.cq_p1 |= 1 << i;
+		break;
+
+	case 2:
+		r2.field.cq_p2 |= 1 << i;
+		r3.field.cq_p2 |= 1 << i;
+		r4.field.cq_p2 |= 1 << i;
+		break;
+
+	case 3:
+		r2.field.cq_p3 |= 1 << i;
+		r3.field.cq_p3 |= 1 << i;
+		r4.field.cq_p3 |= 1 << i;
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r3.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		    r4.val);
+
+	dlb2_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB2_QUEUE_MAPPED;
+
+	return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	/* Set the atomic scheduling haswork bit */
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.rlist_haswork_v = r0.field.count > 0;
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      u8 slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.rlist_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.nalb_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+					      struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+	r0.field.limit = queue->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+						struct dlb2_ldb_queue *queue)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+						struct dlb2_hw_domain *domain,
+						struct dlb2_ldb_port *port,
+						struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_lsp_qid_ldb_infl_cnt r0;
+	enum dlb2_qid_map_state state;
+	int slot, ret, i;
+	u8 prio;
+	RTE_SET_USED(iter);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: non-zero QID inflight count\n",
+			    __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			state = DLB2_QUEUE_MAPPED;
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+					 struct dlb2_ldb_port *port,
+					 struct dlb2_ldb_queue *queue,
+					 u8 priority)
+{
+	union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	int domain_id, slot, ret;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+		DLB2_HW_ERR(hw,
+			    "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		if (port->enabled)
+			dlb2_ldb_port_cq_enable(hw, port);
+
+		dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain,
+					struct dlb2_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		union dlb2_lsp_qid_ldb_infl_cnt r0;
+		struct dlb2_ldb_queue *queue;
+		int qid;
+
+		if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: unable to find queue %d\n",
+				    __func__, qid);
+			continue;
+		}
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count)
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb2_ldb_port_cq_disable(hw, port);
+
+		dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count) {
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+
+			dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port,
+				   struct dlb2_ldb_queue *queue)
+{
+	enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_atm_qid2cqidix_00 r1;
+	union dlb2_lsp_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix2_00 r3;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB2_QUEUE_MAPPED;
+	in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: QID %d isn't mapped\n",
+			    __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+	r0.field.v &= ~(1 << i);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		r1.field.cq_p0 &= ~(1 << i);
+		r2.field.cq_p0 &= ~(1 << i);
+		r3.field.cq_p0 &= ~(1 << i);
+		break;
+
+	case 1:
+		r1.field.cq_p1 &= ~(1 << i);
+		r2.field.cq_p1 &= ~(1 << i);
+		r3.field.cq_p1 &= ~(1 << i);
+		break;
+
+	case 2:
+		r1.field.cq_p2 &= ~(1 << i);
+		r2.field.cq_p2 &= ~(1 << i);
+		r3.field.cq_p2 &= ~(1 << i);
+		break;
+
+	case 3:
+		r1.field.cq_p3 &= ~(1 << i);
+		r2.field.cq_p3 &= ~(1 << i);
+		r3.field.cq_p3 &= ~(1 << i);
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+		    r1.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+		    r3.val);
+
+	dlb2_flush_csr(hw);
+
+	unmapped = DLB2_QUEUE_UNMAPPED;
+
+	return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_ldb_port *port,
+				 struct dlb2_ldb_queue *queue,
+				 u8 prio)
+{
+	if (domain->started)
+		return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   int slot)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb2_ldb_port_qid_map *map;
+		struct dlb2_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain,
+					  struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (r0.field.count > 0)
+		return false;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = false;
+
+			dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_dir_vpp_v r1;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		unsigned int offs;
+		u32 virt_id;
+
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+	}
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_ldb_vpp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			unsigned int offs;
+			u32 virt_id;
+
+			if (hw->virt_mode == DLB2_VIRT_SRIOV)
+				virt_id = port->id.virt_id;
+			else
+				virt_id = port->id.phys_id;
+
+			offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+			DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+	union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+				    r0.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+				    r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+	union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+			    r0.val);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+			    r1.val);
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+		union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+		union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+				    r1.val);
+
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID_V(idx),
+				    r2.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID2QID(idx),
+				    r3.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID_V(idx),
+				    r1.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID2QID(idx),
+				    r2.val);
+		}
+	}
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+					       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_sn_chk_enbl r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.en = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			int i;
+
+			for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+				if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					    __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		port->enabled = false;
+
+		dlb2_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	union dlb2_sys_dir_pp_v r1;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+			    r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_ldb_pp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_ldb_queue *queue;
+	int i;
+	RTE_SET_USED(iter);
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty ldb queue %d\n",
+				    __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port, iter) {
+			if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb2_ldb_cq_token_count(hw, ldb_port)) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to empty ldb port %d\n",
+					    __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir queue %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb2_dir_cq_token_count(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir port %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						   struct dlb2_ldb_port *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP2PP(offs),
+			    DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP_V(offs),
+			    DLB2_SYS_VF_LDB_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    DLB2_SYS_LDB_PP_V_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_DSBL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_BASE_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID0(port->id.phys_id),
+		    DLB2_LSP_CQ2QID0_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID1(port->id.phys_id),
+		    DLB2_LSP_CQ2QID1_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+		    DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			__dlb2_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+				       struct dlb2_dir_pq_pair *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_DSBL_RST);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_FMT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP2PP(offs),
+			    DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP_V(offs),
+			    DLB2_SYS_VF_DIR_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		__dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+			    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_ITS(queue_id),
+			    DLB2_SYS_LDB_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN(queue_id),
+			    DLB2_CHP_ORD_QID_SN_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+			    DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_V(queue_id),
+			    DLB2_SYS_LDB_QID_V_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_CFG_V(queue_id),
+			    DLB2_SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB2_CSR_WR(hw,
+				    offs[queue->sn_group],
+				    DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX2(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX2_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_ATM_QID2CQIDIX(queue_id, i),
+				    DLB2_ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb2_domain_reset_dir_port_registers(hw, domain);
+
+	dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_dir_pq_pair *tmp_dir_port;
+	struct dlb2_ldb_queue *tmp_ldb_queue;
+	struct dlb2_ldb_port *tmp_ldb_port;
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_list_head *list;
+	int ret, i;
+	RTE_SET_USED(tmp_dir_port);
+	RTE_SET_USED(tmp_ldb_queue);
+	RTE_SET_USED(tmp_ldb_port);
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list = &domain->used_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb2_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		dlb2_list_del(&domain->used_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list = &domain->avail_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		ldb_queue->owned = false;
+
+		dlb2_list_del(&domain->avail_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		list = &domain->used_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB2_QUEUE_UNMAPPED;
+
+			dlb2_list_del(&domain->used_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list = &domain->avail_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			ldb_port->owned = false;
+
+			dlb2_list_del(&domain->avail_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list = &domain->used_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		dlb2_list_del(&domain->used_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list = &domain->avail_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				    domain->hist_list_entry_base,
+				    domain->total_hist_list_entries);
+	if (ret) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			    __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	dlb2_list_del(&rsrcs->used_domains, &domain->func_list);
+	dlb2_list_add(&rsrcs->avail_domains, &domain->func_list);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain,
+					    struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_ldb_port *port;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (!dlb2_list_empty(&domain->used_ldb_ports[i]))
+			break;
+	}
+
+	if (i == DLB2_NUM_COS_DOMAINS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: No configured LDB ports\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb2_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+	 * prior to this in dlb2_domain_drain_mapped_queues().
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings != 0 ||
+		    dlb2_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ *	hardware resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain  == NULL || !domain->configured)
+		return -EINVAL;
+
+	/* Disable VPPs */
+	if (vdev_req) {
+		dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+		dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+	}
+
+	/* Disable CQ interrupts */
+	dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+	dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+	dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb2_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_mapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	dlb2_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb2_domain_disable_dir_cqs(hw, domain);
+
+	/* Disable PPs */
+	dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+	ret = dlb2_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Reset the QID and port state. */
+	dlb2_domain_reset_registers(hw, domain);
+
+	/* Hardware reset complete. Reset the domain's software state */
+	ret = dlb2_domain_reset_software_state(hw, domain);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 210b3186..285ad2c 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -598,3 +598,18 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
 
 	return 0;
 }
+
+int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+			    struct dlb2_create_sched_domain_args *args,
+			    struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_sched_domain(hw, args, resp, NOT_VF_REQ,
+					   PF_ID_ZERO);
+}
+
+int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 7d64309..9310f72 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -108,15 +108,59 @@ dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
 	return 0;
 }
 
+static int
+dlb2_pf_sched_domain_create(struct dlb2_hw_dev *handle,
+			    struct dlb2_create_sched_domain_args *arg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (dlb2_dev->domain_reset_failed) {
+		response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = dlb2_pf_create_sched_domain(&dlb2_dev->hw, arg, &response);
+	if (ret)
+		goto done;
+
+done:
+
+	arg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static void
+dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret;
+
+	dlb2_dev = (struct dlb2_dev *)dlb2->qm_instance.pf_dev;
+	ret = dlb2_pf_reset_domain(&dlb2_dev->hw, dlb2->qm_instance.domain_id);
+	if (ret)
+		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
 	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
 	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_domain_reset = dlb2_pf_domain_reset;
 	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
 	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v5 11/23] event/dlb2: add queue and port default conf

[Timothy McDaniel]

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 6798e66..ef1c000 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -661,12 +661,42 @@ dlb2_eventdev_configure(const struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
+				    uint8_t port_id,
+				    struct rte_event_port_conf *port_conf)
+{
+	RTE_SET_USED(port_id);
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	port_conf->new_event_threshold = dlb2->new_event_limit;
+	port_conf->dequeue_depth = 32;
+	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
+	port_conf->event_port_cfg = 0;
+}
+
+static void
+dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
+				     uint8_t queue_id,
+				     struct rte_event_queue_conf *queue_conf)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(queue_id);
+
+	queue_conf->nb_atomic_flows = 1024;
+	queue_conf->nb_atomic_order_sequences = 64;
+	queue_conf->event_queue_cfg = 0;
+	queue_conf->priority = 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v5 12/23] event/dlb2: add queue setup

[Timothy McDaniel]

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  73 +++--
 drivers/event/dlb2/dlb2.c                  | 312 +++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |  12 +
 drivers/event/dlb2/dlb2_iface.h            |  12 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 464 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  82 +++++
 7 files changed, 926 insertions(+), 39 deletions(-)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 5f6c486..b154cac 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -47,45 +47,40 @@ setup argument and the per-port ``new_event_threshold`` argument apply as
 defined in the eventdev header file. The limit is applied to all enqueues,
 regardless of whether it will consume a directed or load-balanced credit.
 
-Load-balanced and Directed Ports
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
-does not have the same concept, but it has a similar one: ports and queues that
-are singly-linked (i.e. linked to a single queue or port, respectively).
-
-The ``rte_event_dev_info_get()`` function reports the number of available
-event ports and queues (among other things). For the DLB2 PMD, max_event_ports
-and max_event_queues report the number of available load-balanced ports and
-queues, and max_single_link_event_port_queue_pairs reports the number of
-available directed ports and queues.
-
-When a scheduling domain is created in ``rte_event_dev_configure()``, the user
-specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
-control the total number of ports (load-balanced and directed) and the number
-of directed ports. Hence, the number of requested load-balanced ports is
-``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
-specifies the total number of queues (load-balanced and directed). The number
-of directed queues comes from ``nb_single_link_event_port_queues``, since
-directed ports and queues come in pairs.
-
-When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
-whether it should be configured as a directed (the flag is set) or a
-load-balanced (the flag is unset) port. Similarly, the
-``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
-whether it is a directed or load-balanced queue.
-
-Load-balanced ports can only be linked to load-balanced queues, and directed
-ports can only be linked to directed queues. Furthermore, directed ports can
-only be linked to a single directed queue (and vice versa), and that link
-cannot change after the eventdev is started.
-
-The eventdev API does not have a directed scheduling type. To support directed
-traffic, the dlb PMD detects when an event is being sent to a directed queue
-and overrides its scheduling type. Note that the originally selected scheduling
-type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
-will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
-port.
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
 
 Flow ID
 ~~~~~~~
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index ef1c000..8c1e06d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -688,6 +688,317 @@ dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
 	queue_conf->priority = 0;
 }
 
+static int32_t
+dlb2_get_sn_allocation(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_allocation_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_set_sn_allocation(struct dlb2_eventdev *dlb2, int group, int num)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_set_sn_allocation_args cfg;
+	int ret;
+
+	cfg.num = num;
+	cfg.group = group;
+
+	ret = dlb2_iface_set_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: set_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int32_t
+dlb2_get_sn_occupancy(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_occupancy_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_occupancy(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_occupancy ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+/* Query the current sequence number allocations and, if they conflict with the
+ * requested LDB queue configuration, attempt to re-allocate sequence numbers.
+ * This is best-effort; if it fails, the PMD will attempt to configure the
+ * load-balanced queue and return an error.
+ */
+static void
+dlb2_program_sn_allocation(struct dlb2_eventdev *dlb2,
+			   const struct rte_event_queue_conf *queue_conf)
+{
+	int grp_occupancy[DLB2_NUM_SN_GROUPS];
+	int grp_alloc[DLB2_NUM_SN_GROUPS];
+	int i, sequence_numbers;
+
+	sequence_numbers = (int)queue_conf->nb_atomic_order_sequences;
+
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		int total_slots;
+
+		grp_alloc[i] = dlb2_get_sn_allocation(dlb2, i);
+		if (grp_alloc[i] < 0)
+			return;
+
+		total_slots = DLB2_MAX_LDB_SN_ALLOC / grp_alloc[i];
+
+		grp_occupancy[i] = dlb2_get_sn_occupancy(dlb2, i);
+		if (grp_occupancy[i] < 0)
+			return;
+
+		/* DLB has at least one available slot for the requested
+		 * sequence numbers, so no further configuration required.
+		 */
+		if (grp_alloc[i] == sequence_numbers &&
+		    grp_occupancy[i] < total_slots)
+			return;
+	}
+
+	/* None of the sequence number groups are configured for the requested
+	 * sequence numbers, so we have to reconfigure one of them. This is
+	 * only possible if a group is not in use.
+	 */
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		if (grp_occupancy[i] == 0)
+			break;
+	}
+
+	if (i == DLB2_NUM_SN_GROUPS) {
+		printf("[%s()] No groups with %d sequence_numbers are available or have free slots\n",
+		       __func__, sequence_numbers);
+		return;
+	}
+
+	/* Attempt to configure slot i with the requested number of sequence
+	 * numbers. Ignore the return value -- if this fails, the error will be
+	 * caught during subsequent queue configuration.
+	 */
+	dlb2_set_sn_allocation(dlb2, i, sequence_numbers);
+}
+
+static int32_t
+dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 const struct rte_event_queue_conf *evq_conf)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_queue *queue = &ev_queue->qm_queue;
+	struct dlb2_create_ldb_queue_args cfg;
+	int32_t ret;
+	uint32_t qm_qid;
+	int sched_type = -1;
+
+	if (evq_conf == NULL)
+		return -EINVAL;
+
+	if (evq_conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_ALL_TYPES) {
+		if (evq_conf->nb_atomic_order_sequences != 0)
+			sched_type = RTE_SCHED_TYPE_ORDERED;
+		else
+			sched_type = RTE_SCHED_TYPE_PARALLEL;
+	} else
+		sched_type = evq_conf->schedule_type;
+
+	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
+	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
+	cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences;
+
+	if (sched_type != RTE_SCHED_TYPE_ORDERED) {
+		cfg.num_sequence_numbers = 0;
+		cfg.num_qid_inflights = 2048;
+	}
+
+	/* App should set this to the number of hardware flows they want, not
+	 * the overall number of flows they're going to use. E.g. if app is
+	 * using 64 flows and sets compression to 64, best-case they'll get
+	 * 64 unique hashed flows in hardware.
+	 */
+	switch (evq_conf->nb_atomic_flows) {
+	/* Valid DLB2 compression levels */
+	case 64:
+	case 128:
+	case 256:
+	case 512:
+	case (1 * 1024): /* 1K */
+	case (2 * 1024): /* 2K */
+	case (4 * 1024): /* 4K */
+	case (64 * 1024): /* 64K */
+		cfg.lock_id_comp_level = evq_conf->nb_atomic_flows;
+		break;
+	default:
+		/* Invalid compression level */
+		cfg.lock_id_comp_level = 0; /* no compression */
+	}
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create LB event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	qm_qid = cfg.response.id;
+
+	/* Save off queue config for debug, resource lookups, and reconfig */
+	queue->num_qid_inflights = cfg.num_qid_inflights;
+	queue->num_atm_inflights = cfg.num_atomic_inflights;
+
+	queue->sched_type = sched_type;
+	queue->config_state = DLB2_CONFIGURED;
+
+	DLB2_LOG_DBG("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n",
+		     qm_qid,
+		     cfg.num_atomic_inflights,
+		     cfg.num_sequence_numbers,
+		     cfg.num_qid_inflights);
+
+	return qm_qid;
+}
+
+static int
+dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int32_t qm_qid;
+
+	if (queue_conf->nb_atomic_order_sequences)
+		dlb2_program_sn_allocation(dlb2, queue_conf);
+
+	qm_qid = dlb2_hw_create_ldb_queue(dlb2, ev_queue, queue_conf);
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the load-balanced queue\n");
+
+		return qm_qid;
+	}
+
+	dlb2->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int dlb2_num_dir_queues_setup(struct dlb2_eventdev *dlb2)
+{
+	int i, num = 0;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].setup_done &&
+		    dlb2->ev_queues[i].qm_queue.is_directed)
+			num++;
+	}
+
+	return num;
+}
+
+static void
+dlb2_queue_link_teardown(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue)
+{
+	struct dlb2_eventdev_port *ev_port;
+	int i, j;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (!ev_port->link[j].valid ||
+			    ev_port->link[j].queue_id != ev_queue->id)
+				continue;
+
+			ev_port->link[j].valid = false;
+			ev_port->num_links--;
+		}
+	}
+
+	ev_queue->num_links = 0;
+}
+
+static int
+dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
+			  uint8_t ev_qid,
+			  const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_queue *ev_queue;
+	int ret;
+
+	if (queue_conf == NULL)
+		return -EINVAL;
+
+	if (ev_qid >= dlb2->num_queues)
+		return -EINVAL;
+
+	ev_queue = &dlb2->ev_queues[ev_qid];
+
+	ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg &
+		RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+	ev_queue->id = ev_qid;
+	ev_queue->conf = *queue_conf;
+
+	if (!ev_queue->qm_queue.is_directed) {
+		ret = dlb2_eventdev_ldb_queue_setup(dev, ev_queue, queue_conf);
+	} else {
+		/* The directed queue isn't setup until link time, at which
+		 * point we know its directed port ID. Directed queue setup
+		 * will only fail if this queue is already setup or there are
+		 * no directed queues left to configure.
+		 */
+		ret = 0;
+
+		ev_queue->qm_queue.config_state = DLB2_NOT_CONFIGURED;
+
+		if (ev_queue->setup_done ||
+		    dlb2_num_dir_queues_setup(dlb2) == dlb2->num_dir_queues)
+			ret = -EINVAL;
+	}
+
+	/* Tear down pre-existing port->queue links */
+	if (!ret && dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_queue_link_teardown(dlb2, ev_queue);
+
+	if (!ret)
+		ev_queue->setup_done = true;
+
+	return ret;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -696,6 +1007,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a829b9b..4c07574 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -31,3 +31,15 @@ int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				struct dlb2_create_sched_domain_args *args);
 
 void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_ldb_queue_args *cfg);
+
+int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_sn_allocation_args *args);
+
+int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_set_sn_allocation_args *args);
+
+int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_occupancy_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 6663dab..4c88fe0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -30,4 +30,16 @@ extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				 struct dlb2_create_sched_domain_args *args);
 
 extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+extern int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_queue_args *cfg);
+
+extern int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_allocation_args *args);
+
+extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_set_sn_allocation_args *args);
+
+extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				  struct dlb2_get_sn_occupancy_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 63a68a6..e6ea0d7 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3509,3 +3509,467 @@ unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
 
 	return num;
 }
+
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_ldb_queue *queue,
+				     struct dlb2_create_ldb_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+	union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+	union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+	union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+	union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+	union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+	union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+	union dlb2_sys_ldb_qid_its r7 = { {0} };
+	union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+	union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+	union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+	union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+	union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+	union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+	struct dlb2_sn_group *sn_group;
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r3.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	r4.field.limit = args->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+	r5.field.limit = queue->aqed_limit;
+
+	if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+		r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+		    r5.val);
+
+	switch (args->lock_id_comp_level) {
+	case 64:
+		r6.field.compress_code = 1;
+		break;
+	case 128:
+		r6.field.compress_code = 2;
+		break;
+	case 256:
+		r6.field.compress_code = 3;
+		break;
+	case 512:
+		r6.field.compress_code = 4;
+		break;
+	case 1024:
+		r6.field.compress_code = 5;
+		break;
+	case 2048:
+		r6.field.compress_code = 6;
+		break;
+	case 4096:
+		r6.field.compress_code = 7;
+		break;
+	case 0:
+	case 65536:
+		r6.field.compress_code = 0;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+		    r6.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r7.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+		    r7.val);
+
+	r8.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+		    r8.val);
+
+	r9.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+		    r9.val);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	r10.field.qid_fid_limit = 512;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+		    r10.val);
+
+	/* Configure SNs */
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	r11.field.mode = sn_group->mode;
+	r11.field.slot = queue->sn_slot;
+	r11.field.grp  = sn_group->id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+	r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+	r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		r0.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+		r1.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+		r2.field.vqid = queue->id.virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+			    r2.val);
+	}
+
+	r13.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+				  struct dlb2_ldb_queue *queue,
+				  struct dlb2_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb2_sn_group_full(group)) {
+			slot = dlb2_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no sequence number slots available\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+				struct dlb2_hw_domain *domain,
+				struct dlb2_ldb_queue *queue,
+				struct dlb2_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_ldb_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (dlb2_list_empty(&domain->avail_ldb_queues)) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb2_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_ldb_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		    args->num_sequence_numbers);
+	DLB2_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		    args->num_qid_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		    args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+
+	dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			    __func__, __LINE__);
+		return ret;
+	}
+
+	dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_queues, &queue->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_queues, &queue->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+						unsigned int group_id,
+						unsigned long val)
+{
+	DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+	DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+	DLB2_HW_DBG(hw, "\tValue:    %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val)
+{
+	u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+	union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+	struct dlb2_sn_group *group;
+	int mode;
+
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	group = &hw->rsrcs.sn_groups[group_id];
+
+	/*
+	 * Once the first load-balanced queue using an SN group is configured,
+	 * the group cannot be changed.
+	 */
+	if (group->slot_use_bitmap != 0)
+		return -EPERM;
+
+	for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+		if (val == valid_allocations[mode])
+			break;
+
+	if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+		return -EINVAL;
+
+	group->mode = mode;
+	group->sequence_numbers_per_queue = val;
+
+	r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+	r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+	DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+	dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 285ad2c..14bb04b 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -613,3 +613,13 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
 {
 	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_ldb_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 9310f72..a6cd178 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -150,6 +150,84 @@ dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
 		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
 }
 
+static int
+dlb2_pf_ldb_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_ldb_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_ldb_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw_dev *handle,
+			 struct dlb2_get_sn_occupancy_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_number_occupancy(&dlb2_dev->hw,
+						       args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_numbers(&dlb2_dev->hw, args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_set_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_set_group_sequence_numbers(&dlb2_dev->hw, args->group,
+					      args->num);
+
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -161,6 +239,10 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
+	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
+	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
+	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v5 13/23] event/dlb2: add port setup

[Timothy McDaniel]

Configure the load balanded (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  75 +++
 drivers/event/dlb2/dlb2.c                  | 498 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   8 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 922 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  28 +
 drivers/event/dlb2/pf/dlb2_pf.c            | 180 ++++++
 7 files changed, 1719 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index b154cac..bbd6ac8 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -82,6 +82,81 @@ The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
 the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
 load-balanced queues can use the full 16-bit flow ID range.
 
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 8c1e06d..9e8dcce 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -999,6 +999,503 @@ dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
 	return ret;
 }
 
+static int
+dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_cq_pop_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_cq_pop_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL)	{
+		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->consume_qe = qe;
+
+	qe->qe_valid = 0;
+	qe->qe_frag = 0;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
+	 * and so on.
+	 */
+	qe->tokens = 0;	/* set at run time */
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_enqueue_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_enqueue_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL) {
+		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->int_arm_qe = qe;
+
+	/* V2 - INT ARM is CQ_TOKEN + FRAG */
+	qe->qe_valid = 0;
+	qe->qe_frag = 1;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
+{
+	int ret, sz;
+
+	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct dlb2_enqueue_qe);
+
+	qm_port->qe4 = rte_zmalloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
+
+	if (qm_port->qe4 == NULL) {
+		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	ret = dlb2_init_consume_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	return 0;
+
+error_exit:
+
+	dlb2_free_qe_mem(qm_port);
+
+	return ret;
+}
+
+static int
+dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_ldb_port_args cfg = {0};
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_CQ_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	cfg.cq_history_list_size = DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	if (handle->cos_id == DLB2_COS_DEFAULT)
+		cfg.cos_id = 0;
+	else
+		cfg.cos_id = handle->cos_id;
+
+	cfg.cos_strict = 0;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* If there are no directed ports, the kernel driver will ignore this
+	 * port's directed credit settings. Don't use enqueue_depth if it would
+	 * require more directed credits than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2; /* back ptr */
+	/*
+	 * Allocate and init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_ldb_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(qm_port->cfg.ldb));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     qm_port->ldb_credits,
+		     qm_port->dir_credits);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
+
+	return ret;
+}
+
+static void
+dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port)
+{
+	struct dlb2_eventdev_queue *ev_queue;
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (!ev_port->link[i].valid)
+			continue;
+
+		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
+
+		ev_port->link[i].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+}
+
+static int
+dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_port_args cfg = {0};
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (dlb2 == NULL || handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* Directed queues are configured at link time. */
+	cfg.queue_id = -1;
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* Don't use enqueue_depth if it would require more directed credits
+	 * than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2;  /* back ptr */
+
+	/*
+	 * Init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE to be aligned
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_dir_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = cfg.cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(qm_port->cfg.dir));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     dir_credit_high_watermark,
+		     ldb_credit_high_watermark);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_setup(struct rte_eventdev *dev,
+			 uint8_t ev_port_id,
+			 const struct rte_event_port_conf *port_conf)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_eventdev_port *ev_port;
+	int ret;
+
+	if (dev == NULL || port_conf == NULL) {
+		DLB2_LOG_ERR("Null parameter\n");
+		return -EINVAL;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
+		return -EINVAL;
+
+	if (port_conf->dequeue_depth >
+		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
+	    port_conf->enqueue_depth >
+		evdev_dlb2_default_info.max_event_port_enqueue_depth)
+		return -EINVAL;
+
+	ev_port = &dlb2->ev_ports[ev_port_id];
+	/* configured? */
+	if (ev_port->setup_done) {
+		DLB2_LOG_ERR("evport %d is already configured\n", ev_port_id);
+		return -EINVAL;
+	}
+
+	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
+		RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (!ev_port->qm_port.is_directed) {
+		ret = dlb2_hw_create_ldb_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the lB port ve portId=%d\n",
+				     ev_port_id);
+
+			return ret;
+		}
+	} else {
+		ret = dlb2_hw_create_dir_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the DIR port\n");
+			return ret;
+		}
+	}
+
+	/* Save off port config for reconfig */
+	ev_port->conf = *port_conf;
+
+	ev_port->id = ev_port_id;
+	ev_port->enq_configured = true;
+	ev_port->setup_done = true;
+	ev_port->inflight_max = port_conf->new_event_threshold;
+	ev_port->implicit_release = !(port_conf->event_port_cfg &
+		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
+	ev_port->outstanding_releases = 0;
+	ev_port->inflight_credits = 0;
+	ev_port->credit_update_quanta = RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
+	ev_port->dlb2 = dlb2; /* reverse link */
+
+	/* Tear down pre-existing port->queue links */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
+
+	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1009,6 +1506,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
+		.port_setup       = dlb2_eventdev_port_setup,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 4c07574..01818c7 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -43,3 +43,11 @@ int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				   struct dlb2_get_sn_occupancy_args *args);
+
+int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_dir_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4c88fe0..a6b923a 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -42,4 +42,12 @@ extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				  struct dlb2_get_sn_occupancy_args *args);
+
+extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_ldb_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_dir_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index e6ea0d7..a633d3e 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3973,3 +3973,925 @@ int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_ldb_port *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_pp2vas r0 = { {0} };
+	union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+		union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_ldb_port *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_ldb_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+	union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+	union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+	union dlb2_chp_hist_list_lim r5 = { {0} };
+	union dlb2_chp_hist_list_base r6 = { {0} };
+	union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+	union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+	union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+	union dlb2_sys_ldb_cq_at r10 = { {0} };
+	union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+	union dlb2_chp_ldb_cq2vas r12 = { {0} };
+	union dlb2_lsp_cq2priov r13 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r14.field.token_count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    r14.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	r5.field.limit = port->hist_list_entry_limit - 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+	r6.field.base = port->hist_list_entry_base;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	r7.field.limit = args->cq_history_list_size;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+	r8.field.push_ptr = r6.field.base;
+	r8.field.generation = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    r8.val);
+
+	r9.field.pop_ptr = r6.field.base;
+	r9.field.generation = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+	/* Disable the port's QID mappings */
+	r13.field.v = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+	return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_ldb_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb2_ldb_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+	if (ret < 0)
+		return ret;
+
+	dlb2_ldb_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_ldb_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		    args->cq_history_list_size);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+	DLB2_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		    args->cos_strict);
+}
+
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_ldb_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+		resp->status = DLB2_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		if (dlb2_list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	} else {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			if (!dlb2_list_empty(&domain->avail_ldb_ports[i]))
+				break;
+		}
+
+		if (i == DLB2_NUM_COS_DOMAINS) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ *	its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	int ret, cos_id, i;
+
+	dlb2_log_create_ldb_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->cos_strict) {
+		cos_id = args->cos_id;
+
+		port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+					  typeof(*port));
+	} else {
+		int idx;
+
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+			port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+						  typeof(*port));
+			if (port)
+				break;
+		}
+
+		cos_id = idx;
+	}
+
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (port->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_ldb_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_ports[cos_id], &port->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_ports[cos_id], &port->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_dir_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+}
+
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+			    bool vdev_req,
+			    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+
+	return NULL;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_dir_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the queue is already configured, validate
+	 * the queue ID, its domain, and whether the queue is configured.
+	 */
+	if (args->queue_id != -1) {
+		struct dlb2_dir_pq_pair *queue;
+
+		queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+						    vdev_req,
+						    domain);
+
+		if (queue == NULL || queue->domain_id.phys_id !=
+				domain->id.phys_id ||
+				!queue->queue_configured) {
+			resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the port's queue is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->queue_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_dir_pq_pair *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_dir_pp2vas r0 = { {0} };
+	union dlb2_sys_dir_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+		union dlb2_sys_dir_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_dir_pq_pair *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_dir_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+	union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+	union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+	union dlb2_sys_dir_cq_fmt r9 = { {0} };
+	union dlb2_sys_dir_cq_at r10 = { {0} };
+	union dlb2_sys_dir_cq_pasid r11 = { {0} };
+	union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r13.field.count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    r13.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.disable_wb_opt = 0;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	r9.field.keep_pf_ppid = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+	return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_dir_pq_pair *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_dir_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb2_dir_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+
+	if (ret < 0)
+		return ret;
+
+	dlb2_dir_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ *	and queue. The port/queue pair have the same ID and name.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *port;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (args->queue_id != -1)
+		port = dlb2_get_domain_used_dir_pq(args->queue_id,
+						   vdev_req,
+						   domain);
+	else
+		port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					  typeof(*port));
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_dir_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs, &port->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs, &port->domain_list);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 14bb04b..2a089f6 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -623,3 +623,31 @@ dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_ldb_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
+
+int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_dir_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a6cd178..2a82b8f 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -228,6 +228,184 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static void *
+dlb2_alloc_coherent_aligned(const struct rte_memzone **mz, uintptr_t *phys,
+			    size_t size, int align)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t core_id = rte_lcore_id();
+	unsigned int socket_id;
+
+	snprintf(mz_name, sizeof(mz_name) - 1, "event_dlb2_pf_%lx",
+		 (unsigned long)rte_get_timer_cycles());
+	if (core_id == (unsigned int)LCORE_ID_ANY)
+		core_id = rte_get_main_lcore();
+	socket_id = rte_lcore_to_socket_id(core_id);
+	*mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
+					 RTE_MEMZONE_IOVA_CONTIG, align);
+	if (*mz == NULL) {
+		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu bytes - %s\n",
+			     size, rte_strerror(rte_errno));
+		*phys = 0;
+		return NULL;
+	}
+	*phys = (*mz)->iova;
+	return (*mz)->addr;
+}
+
+static int
+dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_ldb_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret, cq_alloc_depth;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = false;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	cq_alloc_depth = RTE_MAX(cfg->cq_depth, DLB2_MIN_HARDWARE_CQ_DEPTH);
+	alloc_sz = cq_alloc_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].cq_base = (void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+	return ret;
+}
+
+static int
+dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_dir_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = true;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	alloc_sz = cfg->cq_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
+		(void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -240,6 +418,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v5 14/23] event/dlb2: add port link

[Timothy McDaniel]

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 302 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 633 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
 6 files changed, 1007 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 9e8dcce..c6593c7 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1496,6 +1496,307 @@ dlb2_eventdev_port_setup(struct rte_eventdev *dev,
 	return 0;
 }
 
+static int16_t
+dlb2_hw_map_ldb_qid_to_port(struct dlb2_hw_dev *handle,
+			    uint32_t qm_port_id,
+			    uint16_t qm_qid,
+			    uint8_t priority)
+{
+	struct dlb2_map_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	/* Build message */
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+	cfg.priority = EV_TO_DLB2_PRIO(priority);
+
+	ret = dlb2_iface_map_qid(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: map qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		DLB2_LOG_ERR("dlb2: grp=%d, qm_port=%d, qm_qid=%d prio=%d\n",
+			     handle->domain_id, cfg.port_id,
+			     cfg.qid,
+			     cfg.priority);
+	} else {
+		DLB2_LOG_DBG("dlb2: mapped queue %d to qm_port %d\n",
+			     qm_qid, qm_port_id);
+	}
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port,
+			  struct dlb2_eventdev_queue *ev_queue,
+			  uint8_t priority)
+{
+	int first_avail = -1;
+	int ret, i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid) {
+			if (ev_port->link[i].queue_id == ev_queue->id &&
+			    ev_port->link[i].priority == priority) {
+				if (ev_port->link[i].mapped)
+					return 0; /* already mapped */
+				first_avail = i;
+			}
+		} else if (first_avail == -1)
+			first_avail = i;
+	}
+	if (first_avail == -1) {
+		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
+			     ev_port->qm_port.id);
+		return -EINVAL;
+	}
+
+	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
+					  ev_port->qm_port.id,
+					  ev_queue->qm_queue.id,
+					  priority);
+
+	if (!ret)
+		ev_port->link[first_avail].mapped = true;
+
+	return ret;
+}
+
+static int32_t
+dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 int32_t qm_port_id)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_queue_args cfg;
+	int32_t ret;
+
+	/* The directed port is always configured before its queue */
+	cfg.port_id = qm_port_id;
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_dir_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	int32_t qm_qid;
+
+	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port->qm_port.id);
+
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the DIR queue\n");
+		return qm_qid;
+	}
+
+	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int
+dlb2_do_port_link(struct rte_eventdev *dev,
+		  struct dlb2_eventdev_queue *ev_queue,
+		  struct dlb2_eventdev_port *ev_port,
+		  uint8_t prio)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int err;
+
+	/* Don't link until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	if (ev_queue->qm_queue.is_directed)
+		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue, ev_port);
+	else
+		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue, prio);
+
+	if (err) {
+		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
+			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
+			     ev_queue->id, ev_port->id);
+
+		rte_errno = err;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
+			uint8_t queue_id,
+			bool link_exists,
+			int index)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	bool port_is_dir, queue_is_dir;
+
+	if (queue_id > dlb2->num_queues) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	ev_queue = &dlb2->ev_queues[queue_id];
+
+	if (!ev_queue->setup_done &&
+	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	port_is_dir = ev_port->qm_port.is_directed;
+	queue_is_dir = ev_queue->qm_queue.is_directed;
+
+	if (port_is_dir != queue_is_dir) {
+		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
+			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
+			     port_is_dir ? "DIR" : "LDB", ev_port->id);
+
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if there is space for the requested link */
+	if (!link_exists && index == -1) {
+		DLB2_LOG_ERR("no space for new link\n");
+		rte_errno = -ENOSPC;
+		return -1;
+	}
+
+	/* Check if the directed port is already linked */
+	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if the directed queue is already linked */
+	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
+			     ev_queue->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
+			const uint8_t queues[], const uint8_t priorities[],
+			uint16_t nb_links)
+
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i, j;
+
+	RTE_SET_USED(dev);
+
+	if (ev_port == NULL) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!ev_port->setup_done &&
+	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
+	 * queues pointer.
+	 */
+	if (nb_links == 0) {
+		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
+		     nb_links,
+		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
+		     ev_port->id);
+
+	for (i = 0; i < nb_links; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		uint8_t queue_id, prio;
+		bool found = false;
+		int index = -1;
+
+		queue_id = queues[i];
+		prio = priorities[i];
+
+		/* Check if the link already exists. */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].valid) {
+				if (ev_port->link[j].queue_id == queue_id) {
+					found = true;
+					index = j;
+					break;
+				}
+			} else if (index == -1) {
+				index = j;
+			}
+
+		/* could not link */
+		if (index == -1)
+			break;
+
+		/* Check if already linked at the requested priority */
+		if (found && ev_port->link[j].priority == prio)
+			continue;
+
+		if (dlb2_validate_port_link(ev_port, queue_id, found, index))
+			break; /* return index of offending queue */
+
+		ev_queue = &dlb2->ev_queues[queue_id];
+
+		if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+			break; /* return index of offending queue */
+
+		ev_queue->num_links++;
+
+		ev_port->link[index].queue_id = queue_id;
+		ev_port->link[index].priority = prio;
+		ev_port->link[index].valid = true;
+		/* Entry already exists?  If so, then must be prio change */
+		if (!found)
+			ev_port->num_links++;
+	}
+	return i;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1507,6 +1808,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_link        = dlb2_eventdev_port_link,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 01818c7..7d1ba73 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -51,3 +51,9 @@ int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 				  struct dlb2_create_dir_port_args *cfg,
 				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_dir_queue_args *cfg);
+
+int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+			  struct dlb2_map_qid_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index a6b923a..4eddd14 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -50,4 +50,10 @@ extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 struct dlb2_create_dir_port_args *cfg,
 					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+					struct dlb2_create_dir_queue_args *cfg);
+
+extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+				 struct dlb2_map_qid_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index a633d3e..14ad262 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -4895,3 +4895,636 @@ int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_dir_pq_pair *queue,
+				     struct dlb2_create_dir_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_dir_vasqid_v r0 = { {0} };
+	union dlb2_sys_dir_qid_its r1 = { {0} };
+	union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+	union dlb2_sys_dir_qid_v r5 = { {0} };
+
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r0.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r1.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+		    r1.val);
+
+	r2.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+		    r2.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		r3.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+		r4.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+	}
+
+	r5.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+	queue->queue_configured = true;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_dir_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_dir_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = dlb2_get_domain_used_dir_pq(args->port_id,
+						   vdev_req,
+						   domain);
+
+		if (port == NULL || port->domain_id.phys_id !=
+				domain->id.phys_id || !port->port_configured) {
+			resp->status = DLB2_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the queue's port is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->port_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->port_id != -1)
+		queue = dlb2_get_domain_used_dir_pq(args->port_id,
+						    vdev_req,
+						    domain);
+	else
+		queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					   typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &queue->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs,
+			      &queue->domain_list);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot,
+					      struct dlb2_map_qid_args *args)
+{
+	union dlb2_lsp_cq2priov r0;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+	r0.field.v |= 1 << slot;
+	r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+	dlb2_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+					      struct dlb2_ldb_queue *queue,
+					      struct dlb2_cmd_response *resp)
+{
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB2_QUEUE_UNMAPPED;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+			  bool vdev_req,
+			  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter)
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+			      bool vdev_req,
+			      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+
+		DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+	}
+
+	return NULL;
+}
+
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+				    u32 domain_id,
+				    struct dlb2_map_qid_args *args,
+				    struct dlb2_cmd_response *resp,
+				    bool vdev_req,
+				    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	struct dlb2_ldb_queue *queue;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB2_QID_PRIORITIES) {
+		resp->status = DLB2_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_map_qid_args *args,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	DLB2_HW_DBG(hw, "\tPriority:  %d\n",
+		    args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	int ret, i, id;
+	u8 prio;
+
+	dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_map_qid_args(hw,
+				       domain_id,
+				       args,
+				       resp,
+				       vdev_req,
+				       vdev_id);
+	if (ret)
+		return ret;
+
+	prio = args->priority;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	if (port->num_pending_removals)
+		dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].pending_priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+		if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb2_qid_map_state st;
+
+			if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+				DLB2_HW_ERR(hw,
+					    "[%s():%d] Internal error: port slot tracking failed\n",
+					    __func__, __LINE__);
+				return -EFAULT;
+			}
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb2_port_slot_state_transition(hw, port, queue,
+							      i, st);
+			if (ret)
+				return ret;
+
+			DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 2a089f6..a010e25 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -651,3 +651,13 @@ dlb2_pf_create_dir_port(struct dlb2_hw *hw,
 				       NOT_VF_REQ,
 				       PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_dir_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 2a82b8f..37be4b1 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -406,6 +406,54 @@ dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_dir_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_dir_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_dir_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
+		struct dlb2_map_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_map_qid(&dlb2_dev->hw,
+			      handle->domain_id,
+			      cfg,
+			      &response,
+			      false,
+			      0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -419,7 +467,9 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
 	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
+	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v5 15/23] event/dlb2: add port unlink and port unlinks in progress

[Timothy McDaniel]

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 163 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 283 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  51 ++++++
 5 files changed, 509 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index c6593c7..4ef7b6d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1797,6 +1797,166 @@ dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
 	return i;
 }
 
+static int16_t
+dlb2_hw_unmap_ldb_qid_from_port(struct dlb2_hw_dev *handle,
+				uint32_t qm_port_id,
+				uint16_t qm_qid)
+{
+	struct dlb2_unmap_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+
+	ret = dlb2_iface_unmap_qid(handle, &cfg);
+	if (ret < 0)
+		DLB2_LOG_ERR("dlb2: unmap qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_detach_ldb(struct dlb2_eventdev *dlb2,
+			    struct dlb2_eventdev_port *ev_port,
+			    struct dlb2_eventdev_queue *ev_queue)
+{
+	int ret, i;
+
+	/* Don't unlink until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid &&
+		    ev_port->link[i].queue_id == ev_queue->id)
+			break; /* found */
+	}
+
+	/* This is expected with eventdev API!
+	 * It blindly attemmpts to unmap all queues.
+	 */
+	if (i == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_LOG_DBG("dlb2: ignoring LB QID %d not mapped for qm_port %d.\n",
+			     ev_queue->qm_queue.id,
+			     ev_port->qm_port.id);
+		return 0;
+	}
+
+	ret = dlb2_hw_unmap_ldb_qid_from_port(&dlb2->qm_instance,
+					      ev_port->qm_port.id,
+					      ev_queue->qm_queue.id);
+	if (!ret)
+		ev_port->link[i].mapped = false;
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
+			  uint8_t queues[], uint16_t nb_unlinks)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (queues == NULL || nb_unlinks == 0) {
+		DLB2_LOG_DBG("dlb2: queues is NULL or nb_unlinks is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	/* FIXME: How to handle unlink on a directed port? */
+	if (ev_port->qm_port.is_directed) {
+		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
+			     ev_port->id);
+		rte_errno = 0;
+		return nb_unlinks; /* as if success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	for (i = 0; i < nb_unlinks; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		int ret, j;
+
+		if (queues[i] >= dlb2->num_queues) {
+			DLB2_LOG_ERR("dlb2: invalid queue id %d\n", queues[i]);
+			rte_errno = -EINVAL;
+			return i; /* return index of offending queue */
+		}
+
+		ev_queue = &dlb2->ev_queues[queues[i]];
+
+		/* Does a link exist? */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].queue_id == queues[i] &&
+			    ev_port->link[j].valid)
+				break;
+
+		if (j == DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+			continue;
+
+		ret = dlb2_event_queue_detach_ldb(dlb2, ev_port, ev_queue);
+		if (ret) {
+			DLB2_LOG_ERR("unlink err=%d for port %d queue %d\n",
+				     ret, ev_port->id, queues[i]);
+			rte_errno = -ENOENT;
+			return i; /* return index of offending queue */
+		}
+
+		ev_port->link[j].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+
+	return nb_unlinks;
+}
+
+static int
+dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
+				       void *event_port)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	struct dlb2_pending_port_unmaps_args cfg;
+	int ret;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	cfg.port_id = ev_port->qm_port.id;
+	dlb2 = ev_port->dlb2;
+	handle = &dlb2->qm_instance;
+	ret = dlb2_iface_pending_port_unmaps(handle, &cfg);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: num_unlinks_in_progress ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1809,6 +1969,9 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
 		.port_link        = dlb2_eventdev_port_link,
+		.port_unlink      = dlb2_eventdev_port_unlink,
+		.port_unlinks_in_progress =
+				    dlb2_eventdev_port_unlinks_in_progress,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 7d1ba73..bd241f3 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -57,3 +57,9 @@ int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 			  struct dlb2_map_qid_args *cfg);
+
+int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+			    struct dlb2_unmap_qid_args *cfg);
+
+int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4eddd14..e892036 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -56,4 +56,10 @@ extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 				 struct dlb2_map_qid_args *cfg);
+
+extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+				   struct dlb2_unmap_qid_args *cfg);
+
+extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 14ad262..4a3d96d 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5528,3 +5528,286 @@ int dlb2_hw_map_qid(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_unmap_qid_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+		DLB2_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			    hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+				      u32 domain_id,
+				      struct dlb2_unmap_qid_args *args,
+				      struct dlb2_cmd_response *resp,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	struct dlb2_ldb_port *port;
+	int slot;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			    __func__, args->qid);
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		return 0;
+
+	resp->status = DLB2_ST_INVALID_QID;
+	return -EINVAL;
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	bool unmap_complete;
+	int i, ret, id;
+
+	dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_unmap_qid_args(hw,
+					 domain_id,
+					 args,
+					 resp,
+					 vdev_req,
+					 vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+		st = DLB2_QUEUE_UNMAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		st = DLB2_QUEUE_UNMAP_IN_PROG;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB2_QUEUE_MAPPED;
+	if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: no available CQ slots\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asynchronous procedure. It requires
+	 * stopping the DLB2 from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb2_ldb_port_cq_disable(hw, port);
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete && !os_worker_active(hw))
+		os_schedule_work(hw);
+
+unmap_qid_done:
+	resp->status = 0;
+
+	return 0;
+}
+
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+				  struct dlb2_pending_port_unmaps_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+
+	dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 37be4b1..33ea73e 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -454,6 +454,55 @@ dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw_dev *handle,
+		  struct dlb2_unmap_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_unmap_qid(&dlb2_dev->hw,
+				handle->domain_id,
+				cfg,
+				&response,
+				false,
+				0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
+			    struct dlb2_pending_port_unmaps_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_pending_port_unmaps(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -470,6 +519,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
+	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v5 16/23] event/dlb2: add eventdev start

[Timothy McDaniel]

Add support for the eventdev start entry point.
We delay initializing some resources until
eventdev start, since the number of linked queues can be
used to determine if we are dealing with a ldb or dir resource.
If this is a device restart, then the previous configuration
will be reapplied.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 133 +++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   3 +
 drivers/event/dlb2/dlb2_iface.h            |   3 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 123 ++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +++
 drivers/event/dlb2/pf/dlb2_pf.c            |  25 ++++++
 6 files changed, 297 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 4ef7b6d..2d233bf 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1957,6 +1957,138 @@ dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
 	return cfg.response.id;
 }
 
+static int
+dlb2_eventdev_reapply_configuration(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret, i;
+
+	/* If an event queue or port was previously configured, but hasn't been
+	 * reconfigured, reapply its original configuration.
+	 */
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+
+		ev_queue = &dlb2->ev_queues[i];
+
+		if (ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_queue_setup(dev, i, &ev_queue->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure queue %d", i);
+			return ret;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+
+		if (ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_port_setup(dev, i, &ev_port->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure ev_port %d",
+				     i);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_apply_port_links(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int i;
+
+	/* Perform requested port->queue links */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+		int j;
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			struct dlb2_eventdev_queue *ev_queue;
+			uint8_t prio, queue_id;
+
+			if (!ev_port->link[j].valid)
+				continue;
+
+			prio = ev_port->link[j].priority;
+			queue_id = ev_port->link[j].queue_id;
+
+			if (dlb2_validate_port_link(ev_port, queue_id, true, j))
+				return -EINVAL;
+
+			ev_queue = &dlb2->ev_queues[queue_id];
+
+			if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_start(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_start_domain_args cfg;
+	int ret, i;
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_ERR("bad state %d for dev_start\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return -EINVAL;
+	}
+	dlb2->run_state = DLB2_RUN_STATE_STARTING;
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	/* If the device was configured more than once, some event ports and/or
+	 * queues may need to be reconfigured.
+	 */
+	ret = dlb2_eventdev_reapply_configuration(dev);
+	if (ret)
+		return ret;
+
+	/* The DLB PMD delays port links until the device is started. */
+	ret = dlb2_eventdev_apply_port_links(dev);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		if (!dlb2->ev_ports[i].setup_done) {
+			DLB2_LOG_ERR("dlb2: port %d not setup", i);
+			return -ESTALE;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0) {
+			DLB2_LOG_ERR("dlb2: queue %d is not linked", i);
+			return -ENOLINK;
+		}
+	}
+
+	ret = dlb2_iface_sched_domain_start(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: sched_domain_start ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STARTED;
+	DLB2_LOG_DBG("dlb2: sched_domain_start completed OK\n");
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1964,6 +2096,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.dev_start        = dlb2_eventdev_start,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index bd241f3..a86191d 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -63,3 +63,6 @@ int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				     struct dlb2_start_domain_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index e892036..7ead374 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -62,4 +62,7 @@ extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				struct dlb2_start_domain_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 4a3d96d..c14a2f3 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5811,3 +5811,126 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 struct dlb2_cmd_response *resp,
+					 bool vdev_req,
+					 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_start_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 start domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_hw_start_domain() - Lock the domain configuration
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @arg: User-provided arguments (unused, here for ioctl callback template).
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int
+dlb2_hw_start_domain(struct dlb2_hw *hw,
+		     u32 domain_id,
+		     __attribute((unused)) struct dlb2_start_domain_args *arg,
+		     struct dlb2_cmd_response *resp,
+		     bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_queue;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+	RTE_SET_USED(arg);
+	RTE_SET_USED(iter);
+
+	dlb2_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb2_verify_start_domain_args(hw,
+					    domain_id,
+					    resp,
+					    vdev_req,
+					    vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB2 will drop all
+	 * incoming traffic to those queues.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, ldb_queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r0.val);
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+	}
+
+	dlb2_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index a010e25..aee8336 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -661,3 +661,13 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_start_domain(struct dlb2_hw *hw,
+		     u32 id,
+		     struct dlb2_start_domain_args *args,
+		     struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_start_domain(hw, id, args, resp, NOT_VF_REQ,
+				    PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 33ea73e..fff2e57 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -503,6 +503,30 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
 
 	return ret;
 }
+
+static int
+dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
+			   struct dlb2_start_domain_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_start_domain(&dlb2_dev->hw,
+				   handle->domain_id,
+				   cfg,
+				   &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -520,6 +544,7 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v5 17/23] event/dlb2: add enqueue and its burst variants

[Timothy McDaniel]

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst | 118 +++++++++
 drivers/event/dlb2/dlb2.c     | 578 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 696 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index bbd6ac8..aa8bf01 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -163,6 +163,124 @@ Flow ID
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Hardware Credits
+~~~~~~~~~~~~~~~~
+
+DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
+event storage, with each unit of storage represented by a credit. A port spends
+a credit to enqueue an event, and hardware refills the ports with credits as the
+events are scheduled to ports. Refills come from credit pools, and each port is
+a member of a load-balanced credit pool and a directed credit pool. The
+load-balanced credits are used to enqueue to load-balanced queues, and directed
+credits are used for directed queues.
+
+A DLB2 eventdev contains one load-balanced and one directed credit pool. These
+pools' sizes are controlled by the nb_events_limit field in struct
+rte_event_dev_config. The load-balanced pool is sized to contain
+nb_events_limit credits, and the directed pool is sized to contain
+nb_events_limit/4 credits. The directed pool size can be overridden with the
+num_dir_credits vdev argument, like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,num_dir_credits=<value>
+
+This can be used if the default allocation is too low or too high for the
+specific application needs. The PMD also supports a vdev arg that limits the
+max_num_events reported by rte_event_dev_info_get():
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,max_num_events=<value>
+
+By default, max_num_events is reported as the total available load-balanced
+credits. If multiple DLB2-based applications are being used, it may be desirable
+to control how many load-balanced credits each application uses, particularly
+when application(s) are written to configure nb_events_limit equal to the
+reported max_num_events.
+
+Each port is a member of both credit pools. A port's credit allocation is
+defined by its low watermark, high watermark, and refill quanta. These three
+parameters are calculated by the dlb PMD like so:
+
+- The load-balanced high watermark is set to the port's enqueue_depth.
+  The directed high watermark is set to the minimum of the enqueue_depth and
+  the directed pool size divided by the total number of ports.
+- The refill quanta is set to half the high watermark.
+- The low watermark is set to the minimum of 16 and the refill quanta.
+
+When the eventdev is started, each port is pre-allocated a high watermark's
+worth of credits. For example, if an eventdev contains four ports with enqueue
+depths of 32 and a load-balanced credit pool size of 4096, each port will start
+with 32 load-balanced credits, and there will be 3968 credits available to
+replenish the ports. Thus, a single port is not capable of enqueueing up to the
+nb_events_limit (without any events being dequeued), since the other ports are
+retaining their initial credit allocation; in short, all ports must enqueue in
+order to reach the limit.
+
+If a port attempts to enqueue and has no credits available, the enqueue
+operation will fail and the application must retry the enqueue. Credits are
+replenished asynchronously by the DLB2 hardware.
+
+Software Credits
+~~~~~~~~~~~~~~~~
+
+The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
+credit scheme on top of the hardware credit scheme in order to comply with
+the per-port backpressure described in the eventdev API.
+
+The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
+credit when it enqueues to a queue, and credits are later replenished after the
+events are dequeued and released.
+
+In the software credit scheme, a credit is consumed when a new (.op =
+RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
+replenished when the event is released from the system (either explicitly with
+RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
+
+In this model, an event is "in the system" from its first enqueue into eventdev
+until it is last dequeued. If the event goes through multiple event queues, it
+is still considered "in the system" while a worker thread is processing it.
+
+A port will fail to enqueue if the number of events in the system exceeds its
+``new_event_threshold`` (specified at port setup time). A port will also fail
+to enqueue if it lacks enough hardware credits to enqueue; load-balanced
+credits are used to enqueue to a load-balanced queue, and directed credits are
+used to enqueue to a directed queue.
+
+The out-of-credit situations are typically transient, and an eventdev
+application using the DLB2 ought to retry its enqueues if they fail.
+If enqueue fails, DLB2 PMD sets rte_errno as follows:
+
+- -ENOSPC: Credit exhaustion (either hardware or software)
+- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
+
+Depending on the pipeline the application has constructed, it's possible to
+enter a credit deadlock scenario wherein the worker thread lacks the credit
+to enqueue an event, and it must dequeue an event before it can recover the
+credit. If the worker thread retries its enqueue indefinitely, it will not
+make forward progress. Such deadlock is possible if the application has event
+"loops", in which an event in dequeued from queue A and later enqueued back to
+queue A.
+
+Due to this, workers should stop retrying after a time, release the events it
+is attempting to enqueue, and dequeue more events. It is important that the
+worker release the events and don't simply set them aside to retry the enqueue
+again later, because the port has limited history list size (by default, twice
+the port's dequeue_depth).
+
+Priority
+~~~~~~~~
+
+The DLB2 supports event priority and per-port queue service priority, as
+described in the eventdev header file. The DLB2 does not support 'global' event
+queue priority established at queue creation time.
+
+DLB2 supports 8 event and queue service priority levels. For both priority
+types, the PMD uses the upper three bits of the priority field to determine the
+DLB2 priority, discarding the 5 least significant bits. The 5 least significant
+event priority bits are not preserved when an event is enqueued.
+
 Reconfiguration
 ~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2d233bf..90a5c9d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2089,6 +2089,578 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
 	return 0;
 }
 
+static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
+	{
+		/* Load-balanced cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
+	},
+	{
+		/* Directed cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
+	},
+};
+
+static inline uint32_t
+dlb2_port_credits_get(struct dlb2_port *qm_port,
+		      enum dlb2_hw_queue_types type)
+{
+	uint32_t credits = *qm_port->credit_pool[type];
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	if (unlikely(credits < batch_size))
+		batch_size = credits;
+
+	if (likely(credits &&
+		   __atomic_compare_exchange_n(
+			qm_port->credit_pool[type],
+			&credits, credits - batch_size, false,
+			__ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
+		return batch_size;
+	else
+		return 0;
+}
+
+static inline void
+dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port)
+{
+	uint16_t quanta = ev_port->credit_update_quanta;
+
+	if (ev_port->inflight_credits >= quanta * 2) {
+		/* Replenish credits, saving one quanta for enqueues */
+		uint16_t val = ev_port->inflight_credits - quanta;
+
+		__atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits -= val;
+	}
+}
+
+static inline int
+dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
+						__ATOMIC_SEQ_CST);
+	const int num = 1;
+
+	if (unlikely(ev_port->inflight_max < sw_inflights)) {
+		DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
+		rte_errno = -ENOSPC;
+		return 1;
+	}
+
+	if (ev_port->inflight_credits < num) {
+		/* check if event enqueue brings ev_port over max threshold */
+		uint32_t credit_update_quanta = ev_port->credit_update_quanta;
+
+		if (sw_inflights + credit_update_quanta >
+				dlb2->new_event_limit) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_new_event_limit,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+
+		__atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
+				   __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits += (credit_update_quanta);
+
+		if (ev_port->inflight_credits < num) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_inflight_credits,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_ldb_credits == 0)) {
+		qm_port->cached_ldb_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_LDB_QUEUE);
+		if (unlikely(qm_port->cached_ldb_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
+			1);
+			DLB2_LOG_DBG("ldb credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_dir_credits == 0)) {
+		qm_port->cached_dir_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_DIR_QUEUE);
+		if (unlikely(qm_port->cached_dir_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
+			1);
+			DLB2_LOG_DBG("dir credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static __rte_always_inline void
+dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
+	      struct process_local_port_data *port_data)
+{
+	dlb2_movdir64b(port_data->pp_addr, qe4);
+}
+
+static inline int
+dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_cq_pop_qe *qe;
+
+	RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
+
+	qe = qm_port->consume_qe;
+
+	qe->tokens = num - 1;
+
+	/* No store fence needed since no pointer is being sent, and CQ token
+	 * pops can be safely reordered with other HCWs.
+	 */
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	dlb2_movntdq_single(port_data->pp_addr, qe);
+
+	DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
+
+	qm_port->owed_tokens = 0;
+
+	return 0;
+}
+
+static inline void
+dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
+		   bool do_sfence,
+		   struct process_local_port_data *port_data)
+{
+	/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
+	 * application writes complete before enqueueing the QE.
+	 */
+	if (do_sfence)
+		rte_wmb();
+
+	dlb2_pp_write(qm_port->qe4, port_data);
+}
+
+static inline void
+dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
+{
+	struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
+	int num = qm_port->owed_tokens;
+
+	qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
+	qe[idx].tokens = num - 1;
+
+	qm_port->owed_tokens = 0;
+}
+
+static inline void
+dlb2_event_build_hcws(struct dlb2_port *qm_port,
+		      const struct rte_event ev[],
+		      int num,
+		      uint8_t *sched_type,
+		      uint8_t *queue_id)
+{
+	struct dlb2_enqueue_qe *qe;
+	uint16_t sched_word[4];
+	__m128i sse_qe[2];
+	int i;
+
+	qe = qm_port->qe4;
+
+	sse_qe[0] = _mm_setzero_si128();
+	sse_qe[1] = _mm_setzero_si128();
+
+	switch (num) {
+	case 4:
+		/* Construct the metadata portion of two HCWs in one 128b SSE
+		 * register. HCW metadata is constructed in the SSE registers
+		 * like so:
+		 * sse_qe[0][63:0]:   qe[0]'s metadata
+		 * sse_qe[0][127:64]: qe[1]'s metadata
+		 * sse_qe[1][63:0]:   qe[2]'s metadata
+		 * sse_qe[1][127:64]: qe[3]'s metadata
+		 */
+
+		/* Convert the event operation into a command byte and store it
+		 * in the metadata:
+		 * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
+		 * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
+		 * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
+		 * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
+		 */
+#define DLB2_QE_CMD_BYTE 7
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[0].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[1].op],
+				DLB2_QE_CMD_BYTE + 8);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[2].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[3].op],
+				DLB2_QE_CMD_BYTE + 8);
+
+		/* Store priority, scheduling type, and queue ID in the sched
+		 * word array because these values are re-used when the
+		 * destination is a directed queue.
+		 */
+		sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
+				sched_type[0] << 8 |
+				queue_id[0];
+		sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
+				sched_type[1] << 8 |
+				queue_id[1];
+		sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
+				sched_type[2] << 8 |
+				queue_id[2];
+		sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
+				sched_type[3] << 8 |
+				queue_id[3];
+
+		/* Store the event priority, scheduling type, and queue ID in
+		 * the metadata:
+		 * sse_qe[0][31:16] = sched_word[0]
+		 * sse_qe[0][95:80] = sched_word[1]
+		 * sse_qe[1][31:16] = sched_word[2]
+		 * sse_qe[1][95:80] = sched_word[3]
+		 */
+#define DLB2_QE_QID_SCHED_WORD 1
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[0],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[1],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[2],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[3],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+
+		/* If the destination is a load-balanced queue, store the lock
+		 * ID. If it is a directed queue, DLB places this field in
+		 * bytes 10-11 of the received QE, so we format it accordingly:
+		 * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
+		 * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
+		 * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
+		 * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
+		 */
+#define DLB2_QE_LOCK_ID_WORD 2
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[0] == DLB2_SCHED_DIRECTED) ?
+					sched_word[0] : ev[0].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[1] == DLB2_SCHED_DIRECTED) ?
+					sched_word[1] : ev[1].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[2] == DLB2_SCHED_DIRECTED) ?
+					sched_word[2] : ev[2].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[3] == DLB2_SCHED_DIRECTED) ?
+					sched_word[3] : ev[3].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+
+		/* Store the event type and sub event type in the metadata:
+		 * sse_qe[0][15:0]  = flow_id[0]
+		 * sse_qe[0][79:64] = flow_id[1]
+		 * sse_qe[1][15:0]  = flow_id[2]
+		 * sse_qe[1][79:64] = flow_id[3]
+		 */
+#define DLB2_QE_EV_TYPE_WORD 0
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[0].sub_event_type << 8 |
+						ev[0].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[1].sub_event_type << 8 |
+						ev[1].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[2].sub_event_type << 8 |
+						ev[2].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[3].sub_event_type << 8 |
+						ev[3].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+
+		/* Store the metadata to memory (use the double-precision
+		 * _mm_storeh_pd because there is no integer function for
+		 * storing the upper 64b):
+		 * qe[0] metadata = sse_qe[0][63:0]
+		 * qe[1] metadata = sse_qe[0][127:64]
+		 * qe[2] metadata = sse_qe[1][63:0]
+		 * qe[3] metadata = sse_qe[1][127:64]
+		 */
+		_mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
+		_mm_storeh_pd((double *)&qe[1].u.opaque_data,
+			      (__m128d)sse_qe[0]);
+		_mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
+		_mm_storeh_pd((double *)&qe[3].u.opaque_data,
+			      (__m128d)sse_qe[1]);
+
+		qe[0].data = ev[0].u64;
+		qe[1].data = ev[1].u64;
+		qe[2].data = ev[2].u64;
+		qe[3].data = ev[3].u64;
+
+		break;
+	case 3:
+	case 2:
+	case 1:
+		/* At least one QE will be valid, so only zero out three */
+		qe[1].cmd_byte = 0;
+		qe[2].cmd_byte = 0;
+		qe[3].cmd_byte = 0;
+
+		for (i = 0; i < num; i++) {
+			qe[i].cmd_byte =
+				cmd_byte_map[qm_port->is_directed][ev[i].op];
+			qe[i].sched_type = sched_type[i];
+			qe[i].data = ev[i].u64;
+			qe[i].qid = queue_id[i];
+			qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
+			qe[i].lock_id = ev[i].flow_id;
+			if (sched_type[i] == DLB2_SCHED_DIRECTED) {
+				struct dlb2_msg_info *info =
+					(struct dlb2_msg_info *)&qe[i].lock_id;
+
+				info->qid = queue_id[i];
+				info->sched_type = DLB2_SCHED_DIRECTED;
+				info->priority = qe[i].priority;
+			}
+			qe[i].u.event_type.major = ev[i].event_type;
+			qe[i].u.event_type.sub = ev[i].sub_event_type;
+		}
+		break;
+	}
+}
+
+static inline int
+dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
+			struct dlb2_port *qm_port,
+			const struct rte_event ev[],
+			uint8_t *sched_type,
+			uint8_t *queue_id)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	uint16_t *cached_credits = NULL;
+	struct dlb2_queue *qm_queue;
+
+	ev_queue = &dlb2->ev_queues[ev->queue_id];
+	qm_queue = &ev_queue->qm_queue;
+	*queue_id = qm_queue->id;
+
+	/* Ignore sched_type and hardware credits on release events */
+	if (ev->op == RTE_EVENT_OP_RELEASE)
+		goto op_check;
+
+	if (!qm_queue->is_directed) {
+		/* Load balanced destination queue */
+
+		if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_ldb_credits;
+
+		switch (ev->sched_type) {
+		case RTE_SCHED_TYPE_ORDERED:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
+			if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
+				DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
+					     *queue_id);
+				rte_errno = -EINVAL;
+				return 1;
+			}
+			*sched_type = DLB2_SCHED_ORDERED;
+			break;
+		case RTE_SCHED_TYPE_ATOMIC:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
+			*sched_type = DLB2_SCHED_ATOMIC;
+			break;
+		case RTE_SCHED_TYPE_PARALLEL:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
+			if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
+				*sched_type = DLB2_SCHED_ORDERED;
+			else
+				*sched_type = DLB2_SCHED_UNORDERED;
+			break;
+		default:
+			DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
+			DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
+			rte_errno = -EINVAL;
+			return 1;
+		}
+	} else {
+		/* Directed destination queue */
+
+		if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_dir_credits;
+
+		DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
+
+		*sched_type = DLB2_SCHED_DIRECTED;
+	}
+
+op_check:
+	switch (ev->op) {
+	case RTE_EVENT_OP_NEW:
+		/* Check that a sw credit is available */
+		if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		ev_port->inflight_credits--;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_FORWARD:
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_RELEASE:
+		ev_port->inflight_credits++;
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+
+		/* Replenish s/w credits if enough are cached */
+		dlb2_replenish_sw_credits(dlb2, ev_port);
+		break;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+	if (ev->op != RTE_EVENT_OP_RELEASE) {
+		DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
+		DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
+	}
+#endif
+
+	return 0;
+}
+
+static inline uint16_t
+dlb2_event_enqueue_burst(void *event_port,
+			 const struct rte_event events[],
+			 uint16_t num)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct process_local_port_data *port_data;
+	int i, cnt;
+
+	RTE_ASSERT(ev_port->enq_configured);
+	RTE_ASSERT(events != NULL);
+
+	cnt = 0;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
+		int j = 0;
+
+		for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
+			const struct rte_event *ev = &events[i + j];
+
+			if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
+						    &sched_types[j],
+						    &queue_ids[j]))
+				break;
+		}
+
+		if (j == 0)
+			break;
+
+		dlb2_event_build_hcws(qm_port, &events[i], j,
+				      sched_types, queue_ids);
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+
+		if (j < DLB2_NUM_QES_PER_CACHE_LINE)
+			break;
+	}
+
+	return cnt;
+}
+
+static inline uint16_t
+dlb2_event_enqueue(void *event_port,
+		   const struct rte_event events[])
+{
+	return dlb2_event_enqueue_burst(event_port, events, 1);
+}
+
+static uint16_t
+dlb2_event_enqueue_new_burst(void *event_port,
+			     const struct rte_event events[],
+			     uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
+static uint16_t
+dlb2_event_enqueue_forward_burst(void *event_port,
+				 const struct rte_event events[],
+				 uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2112,7 +2684,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
 	};
 
+	/* Expose PMD's eventdev interface */
+
 	dev->dev_ops = &dlb2_eventdev_entry_ops;
+	dev->enqueue = dlb2_event_enqueue;
+	dev->enqueue_burst = dlb2_event_enqueue_burst;
+	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
+	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v5 18/23] event/dlb2: add dequeue and its burst variants

[Timothy McDaniel]

Add support for dequeue, dequeue_burst, ...

DLB2 does not currently support interrupts, but instead use
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB2 supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst |  21 ++
 drivers/event/dlb2/dlb2.c     | 761 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 782 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index aa8bf01..b9f57fd 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -314,6 +314,27 @@ The PMD does not support the following configuration sequences:
 This sequence is not supported because the event device must be reconfigured
 before its ports or queues can be.
 
+Deferred Scheduling
+~~~~~~~~~~~~~~~~~~~
+
+The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
+dequeued event before returning from rte_event_dequeue_burst(). This frees the
+corresponding entries in the CQ, which enables the DLB2 to schedule more events
+to it.
+
+To support applications seeking finer-grained scheduling control -- for example
+deferring scheduling to get the best possible priority scheduling and
+load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
+the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
+call. This mode only applies to load-balanced event ports with dequeue depth of
+1.
+
+To enable deferred scheduling, use the defer_sched vdev argument like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,defer_sched=on
+
 Atomic Inflights Allocation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 90a5c9d..bd96055 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2661,9 +2661,761 @@ dlb2_event_enqueue_forward_burst(void *event_port,
 	return dlb2_event_enqueue_burst(event_port, events, num);
 }
 
+static inline void
+dlb2_port_credits_inc(struct dlb2_port *qm_port, int num)
+{
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	/* increment port credits, and return to pool if exceeds threshold */
+	if (!qm_port->is_directed) {
+		qm_port->cached_ldb_credits += num;
+		if (qm_port->cached_ldb_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_LDB_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_ldb_credits -= batch_size;
+		}
+	} else {
+		qm_port->cached_dir_credits += num;
+		if (qm_port->cached_dir_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_DIR_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_dir_credits -= batch_size;
+		}
+	}
+}
+
+static inline bool
+dlb2_cq_is_empty(struct dlb2_port *qm_port)
+{
+	volatile struct dlb2_dequeue_qe *qe_ptr;
+	struct dlb2_dequeue_qe qe;
+
+	qe_ptr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	qe = qe_ptr[qm_port->cq_idx];
+
+	return (qe.cq_gen != qm_port->gen_bit);
+}
+
+static inline int
+dlb2_dequeue_wait(struct dlb2_eventdev *dlb2,
+		  struct dlb2_eventdev_port *ev_port,
+		  struct dlb2_port *qm_port,
+		  uint64_t timeout,
+		  uint64_t start_ticks)
+{
+	struct process_local_port_data *port_data;
+	uint64_t elapsed_ticks;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	elapsed_ticks = rte_get_timer_cycles() - start_ticks;
+
+	/* Wait/poll time expired */
+	if (elapsed_ticks >= timeout) {
+		return 1;
+	} else if (dlb2->umwait_allowed) {
+		volatile struct dlb2_dequeue_qe *cq_base;
+
+		cq_base = port_data->cq_base;
+
+		/* Block on cache line write to CQ. Note: it's
+		 * safe to access the per-process cq_base
+		 * address here, since the PMD has already
+		 * attempted at least one CQ dequeue.
+		 */
+		dlb2_umonitor(&cq_base[qm_port->cq_idx]);
+
+		/* Avoid race condition. Check if still empty */
+		if (dlb2_cq_is_empty(qm_port)) {
+			dlb2_umwait(RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE,
+				    timeout + start_ticks);
+			DLB2_INC_STAT(
+				ev_port->stats.traffic.rx_umonitor_umwait, 1);
+		}
+	} else {
+		uint64_t poll_interval = RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL;
+		uint64_t curr_ticks = rte_get_timer_cycles();
+		uint64_t init_ticks = curr_ticks;
+
+		while ((curr_ticks - start_ticks < timeout) &&
+		       (curr_ticks - init_ticks < poll_interval))
+			curr_ticks = rte_get_timer_cycles();
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
+			 struct dlb2_port *qm_port,
+			 struct rte_event *events,
+			 struct dlb2_dequeue_qe *qes,
+			 int cnt)
+{
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	int i, num, evq_id;
+
+	for (i = 0, num = 0; i < cnt; i++) {
+		struct dlb2_dequeue_qe *qe = &qes[i];
+		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
+			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+			[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+		};
+
+		/* Fill in event information.
+		 * Note that flow_id must be embedded in the data by
+		 * the app, such as the mbuf RSS hash field if the data
+		 * buffer is a mbuf.
+		 */
+		if (unlikely(qe->error)) {
+			DLB2_LOG_ERR("QE error bit ON\n");
+			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
+			dlb2_consume_qe_immediate(qm_port, 1);
+			continue; /* Ignore */
+		}
+
+		events[num].u64 = qe->data;
+		events[num].flow_id = qe->flow_id;
+		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
+		events[num].event_type = qe->u.event_type.major;
+		events[num].sub_event_type = qe->u.event_type.sub;
+		events[num].sched_type = sched_type_map[qe->sched_type];
+		events[num].impl_opaque = qe->qid_depth;
+
+		/* qid not preserved for directed queues */
+		if (qm_port->is_directed)
+			evq_id = ev_port->link[0].queue_id;
+		else
+			evq_id = qid_mappings[qe->qid];
+
+		events[num].queue_id = evq_id;
+		DLB2_INC_STAT(
+			ev_port->stats.queue[evq_id].qid_depth[qe->qid_depth],
+			1);
+		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type], 1);
+		num++;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num);
+
+	return num;
+}
+
+static inline int
+dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
+			      struct dlb2_port *qm_port,
+			      struct rte_event *events,
+			      struct dlb2_dequeue_qe *qes)
+{
+	int sched_type_map[] = {
+		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+	};
+	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	__m128i sse_evt[2];
+
+	/* In the unlikely case that any of the QE error bits are set, process
+	 * them one at a time.
+	 */
+	if (unlikely(qes[0].error || qes[1].error ||
+		     qes[2].error || qes[3].error))
+		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
+						 qes, num_events);
+
+	events[0].u64 = qes[0].data;
+	events[1].u64 = qes[1].data;
+	events[2].u64 = qes[2].data;
+	events[3].u64 = qes[3].data;
+
+	/* Construct the metadata portion of two struct rte_events
+	 * in one 128b SSE register. Event metadata is constructed in the SSE
+	 * registers like so:
+	 * sse_evt[0][63:0]:   event[0]'s metadata
+	 * sse_evt[0][127:64]: event[1]'s metadata
+	 * sse_evt[1][63:0]:   event[2]'s metadata
+	 * sse_evt[1][127:64]: event[3]'s metadata
+	 */
+	sse_evt[0] = _mm_setzero_si128();
+	sse_evt[1] = _mm_setzero_si128();
+
+	/* Convert the hardware queue ID to an event queue ID and store it in
+	 * the metadata:
+	 * sse_evt[0][47:40]   = qid_mappings[qes[0].qid]
+	 * sse_evt[0][111:104] = qid_mappings[qes[1].qid]
+	 * sse_evt[1][47:40]   = qid_mappings[qes[2].qid]
+	 * sse_evt[1][111:104] = qid_mappings[qes[3].qid]
+	 */
+#define DLB_EVENT_QUEUE_ID_BYTE 5
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[0].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[1].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[2].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[3].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+
+	/* Convert the hardware priority to an event priority and store it in
+	 * the metadata, while also returning the queue depth status
+	 * value captured by the hardware, storing it in impl_opaque, which can
+	 * be read by the application but not modified
+	 * sse_evt[0][55:48]   = DLB2_TO_EV_PRIO(qes[0].priority)
+	 * sse_evt[0][63:56]   = qes[0].qid_depth
+	 * sse_evt[0][119:112] = DLB2_TO_EV_PRIO(qes[1].priority)
+	 * sse_evt[0][127:120] = qes[1].qid_depth
+	 * sse_evt[1][55:48]   = DLB2_TO_EV_PRIO(qes[2].priority)
+	 * sse_evt[1][63:56]   = qes[2].qid_depth
+	 * sse_evt[1][119:112] = DLB2_TO_EV_PRIO(qes[3].priority)
+	 * sse_evt[1][127:120] = qes[3].qid_depth
+	 */
+#define DLB_EVENT_PRIO_IMPL_OPAQUE_WORD 3
+#define DLB_BYTE_SHIFT 8
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[0].priority) |
+			(qes[0].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[1].priority) |
+			(qes[1].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[2].priority) |
+			(qes[2].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[3].priority) |
+			(qes[3].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+
+	/* Write the event type, sub event type, and flow_id to the event
+	 * metadata.
+	 * sse_evt[0][31:0]   = qes[0].flow_id |
+	 *			qes[0].u.event_type.major << 28 |
+	 *			qes[0].u.event_type.sub << 20;
+	 * sse_evt[0][95:64]  = qes[1].flow_id |
+	 *			qes[1].u.event_type.major << 28 |
+	 *			qes[1].u.event_type.sub << 20;
+	 * sse_evt[1][31:0]   = qes[2].flow_id |
+	 *			qes[2].u.event_type.major << 28 |
+	 *			qes[2].u.event_type.sub << 20;
+	 * sse_evt[1][95:64]  = qes[3].flow_id |
+	 *			qes[3].u.event_type.major << 28 |
+	 *			qes[3].u.event_type.sub << 20;
+	 */
+#define DLB_EVENT_EV_TYPE_DW 0
+#define DLB_EVENT_EV_TYPE_SHIFT 28
+#define DLB_EVENT_SUB_EV_TYPE_SHIFT 20
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[0].flow_id |
+			qes[0].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[0].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[1].flow_id |
+			qes[1].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[1].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[2].flow_id |
+			qes[2].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[2].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[3].flow_id |
+			qes[3].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT  |
+			qes[3].u.event_type.sub << DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+
+	/* Write the sched type to the event metadata. 'op' and 'rsvd' are not
+	 * set:
+	 * sse_evt[0][39:32]  = sched_type_map[qes[0].sched_type] << 6
+	 * sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6
+	 * sse_evt[1][39:32]  = sched_type_map[qes[2].sched_type] << 6
+	 * sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6
+	 */
+#define DLB_EVENT_SCHED_TYPE_BYTE 4
+#define DLB_EVENT_SCHED_TYPE_SHIFT 6
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[0].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[1].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[2].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[3].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+
+	/* Store the metadata to the event (use the double-precision
+	 * _mm_storeh_pd because there is no integer function for storing the
+	 * upper 64b):
+	 * events[0].event = sse_evt[0][63:0]
+	 * events[1].event = sse_evt[0][127:64]
+	 * events[2].event = sse_evt[1][63:0]
+	 * events[3].event = sse_evt[1][127:64]
+	 */
+	_mm_storel_epi64((__m128i *)&events[0].event, sse_evt[0]);
+	_mm_storeh_pd((double *)&events[1].event, (__m128d) sse_evt[0]);
+	_mm_storel_epi64((__m128i *)&events[2].event, sse_evt[1]);
+	_mm_storeh_pd((double *)&events[3].event, (__m128d) sse_evt[1]);
+
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[0].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[1].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[2].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[3].sched_type], 1);
+
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[0].queue_id].
+			qid_depth[qes[0].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[1].queue_id].
+			qid_depth[qes[1].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[2].queue_id].
+			qid_depth[qes[2].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[3].queue_id].
+			qid_depth[qes[3].qid_depth],
+		1);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num_events);
+
+	return num_events;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe_sparse(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe)
+{
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	uint8_t xor_mask[2] = {0x0F, 0x00};
+	const uint8_t and_mask = 0x0F;
+	__m128i *qes = (__m128i *)qe;
+	uint8_t gen_bits, gen_bit;
+	uintptr_t addr[4];
+	uint16_t idx;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+
+	idx = qm_port->cq_idx;
+
+	/* Load the next 4 QEs */
+	addr[0] = (uintptr_t)&cq_addr[idx];
+	addr[1] = (uintptr_t)&cq_addr[(idx +  4) & qm_port->cq_depth_mask];
+	addr[2] = (uintptr_t)&cq_addr[(idx +  8) & qm_port->cq_depth_mask];
+	addr[3] = (uintptr_t)&cq_addr[(idx + 12) & qm_port->cq_depth_mask];
+
+	/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */
+	rte_prefetch0(&cq_addr[(idx + 16) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 20) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 24) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 28) & qm_port->cq_depth_mask]);
+
+	/* Correct the xor_mask for wrap-around QEs */
+	gen_bit = qm_port->gen_bit;
+	xor_mask[gen_bit] ^= !!((idx +  4) > qm_port->cq_depth_mask) << 1;
+	xor_mask[gen_bit] ^= !!((idx +  8) > qm_port->cq_depth_mask) << 2;
+	xor_mask[gen_bit] ^= !!((idx + 12) > qm_port->cq_depth_mask) << 3;
+
+	/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is
+	 * valid, then QEs[0:N-1] are too.
+	 */
+	qes[3] = _mm_load_si128((__m128i *)(void *)addr[3]);
+	rte_compiler_barrier();
+	qes[2] = _mm_load_si128((__m128i *)(void *)addr[2]);
+	rte_compiler_barrier();
+	qes[1] = _mm_load_si128((__m128i *)(void *)addr[1]);
+	rte_compiler_barrier();
+	qes[0] = _mm_load_si128((__m128i *)(void *)addr[0]);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[gen_bit];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask;
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline void
+dlb2_inc_cq_idx(struct dlb2_port *qm_port, int cnt)
+{
+	uint16_t idx = qm_port->cq_idx_unmasked + cnt;
+
+	qm_port->cq_idx_unmasked = idx;
+	qm_port->cq_idx = idx & qm_port->cq_depth_mask;
+	qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) & 0x1;
+}
+
+static int
+dlb2_event_release(struct dlb2_eventdev *dlb2,
+		   uint8_t port_id,
+		   int n)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_eventdev_port *ev_port;
+	struct dlb2_port *qm_port;
+	int i, cnt;
+
+	if (port_id > dlb2->num_ports) {
+		DLB2_LOG_ERR("Invalid port id %d in dlb2-event_release\n",
+			     port_id);
+		rte_errno = -EINVAL;
+		return rte_errno;
+	}
+
+	ev_port = &dlb2->ev_ports[port_id];
+	qm_port = &ev_port->qm_port;
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	cnt = 0;
+
+	if (qm_port->is_directed) {
+		cnt = n;
+		goto sw_credit_update;
+	}
+
+	for (i = 0; i < n; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		int j;
+
+		/* Zero-out QEs */
+		qm_port->qe4[0].cmd_byte = 0;
+		qm_port->qe4[1].cmd_byte = 0;
+		qm_port->qe4[2].cmd_byte = 0;
+		qm_port->qe4[3].cmd_byte = 0;
+
+		for (j = 0; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++)
+			qm_port->qe4[j].cmd_byte = DLB2_COMP_CMD_BYTE;
+
+		qm_port->issued_releases += j;
+
+		if (j == 0)
+			break;
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+	}
+
+sw_credit_update:
+	/* each release returns one credit */
+	if (!ev_port->outstanding_releases) {
+		DLB2_LOG_ERR("Unrecoverable application error. Outstanding releases underflowed.\n");
+		rte_errno = -ENOTRECOVERABLE;
+		return rte_errno;
+	}
+
+	ev_port->outstanding_releases -= cnt;
+	ev_port->inflight_credits += cnt;
+
+	/* Replenish s/w credits if enough releases are performed */
+	dlb2_replenish_sw_credits(dlb2, ev_port);
+	return 0;
+}
+
+static inline int16_t
+dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
+		       struct dlb2_eventdev_port *ev_port,
+		       struct rte_event *events,
+		       uint16_t max_num,
+		       uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe_sparse(qm_port, qes);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail << 2);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							      qm_port,
+							      &events[num],
+							      &qes[0]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							 qm_port,
+							 &events[num],
+							 &qes[0],
+							 num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe,
+	     uint8_t *offset)
+{
+	uint8_t xor_mask[2][4] = { {0x0F, 0x0E, 0x0C, 0x08},
+				   {0x00, 0x01, 0x03, 0x07} };
+	uint8_t and_mask[4] = {0x0F, 0x0E, 0x0C, 0x08};
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	__m128i *qes = (__m128i *)qe;
+	uint64_t *cache_line_base;
+	uint8_t gen_bits;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	cq_addr = &cq_addr[qm_port->cq_idx];
+
+	cache_line_base = (void *)(((uintptr_t)cq_addr) & ~0x3F);
+	*offset = ((uintptr_t)cq_addr & 0x30) >> 4;
+
+	/* Load the next CQ cache line from memory. Pack these reads as tight
+	 * as possible to reduce the chance that DLB invalidates the line while
+	 * the CPU is reading it. Read the cache line backwards to ensure that
+	 * if QE[N] (N > 0) is valid, then QEs[0:N-1] are too.
+	 *
+	 * (Valid QEs start at &qe[offset])
+	 */
+	qes[3] = _mm_load_si128((__m128i *)&cache_line_base[6]);
+	qes[2] = _mm_load_si128((__m128i *)&cache_line_base[4]);
+	qes[1] = _mm_load_si128((__m128i *)&cache_line_base[2]);
+	qes[0] = _mm_load_si128((__m128i *)&cache_line_base[0]);
+
+	/* Evict the cache line ASAP */
+	dlb2_cldemote(cache_line_base);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[qm_port->gen_bit][*offset];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask[*offset];
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline int16_t
+dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
+		struct dlb2_eventdev_port *ev_port,
+		struct rte_event *events,
+		uint16_t max_num,
+		uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t offset;
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe(qm_port, qes, &offset);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							     qm_port,
+							     &events[num],
+							     &qes[offset]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							qm_port,
+							&events[num],
+							&qes[offset],
+							num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static uint16_t
+dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
+			 uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue(void *event_port, struct rte_event *ev, uint64_t wait)
+{
+	return dlb2_event_dequeue_burst(event_port, ev, 1, wait);
+}
+
+static uint16_t
+dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
+				uint16_t num, uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
+			  uint64_t wait)
+{
+	return dlb2_event_dequeue_burst_sparse(event_port, ev, 1, wait);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
+	struct dlb2_eventdev *dlb2;
+
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
@@ -2691,6 +3443,15 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	dev->enqueue_burst = dlb2_event_enqueue_burst;
 	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
 	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
+
+	dlb2 = dev->data->dev_private;
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE) {
+		dev->dequeue = dlb2_event_dequeue_sparse;
+		dev->dequeue_burst = dlb2_event_dequeue_burst_sparse;
+	} else {
+		dev->dequeue = dlb2_event_dequeue;
+		dev->dequeue_burst = dlb2_event_dequeue_burst;
+	}
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v5 19/23] event/dlb2: add eventdev stop and close

[Timothy McDaniel]

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 257 +++++++++++++++++++++++++++--
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c |  91 ++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  52 ++++++
 5 files changed, 396 insertions(+), 16 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index bd96055..b02a70c 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,21 +70,6 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
-/*
- * DUMMY - added so that xstats path will compile/link.
- * Will be replaced by real version in a subsequent
- * patch.
- */
-uint32_t
-dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
-		     struct dlb2_eventdev_queue *queue)
-{
-	RTE_SET_USED(dlb2);
-	RTE_SET_USED(queue);
-
-	return 0;
-}
-
 static void
 dlb2_free_qe_mem(struct dlb2_port *qm_port)
 {
@@ -1877,7 +1862,6 @@ dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
 		return 0; /* Ignore and return success */
 	}
 
-	/* FIXME: How to handle unlink on a directed port? */
 	if (ev_port->qm_port.is_directed) {
 		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
 			     ev_port->id);
@@ -3412,6 +3396,245 @@ dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
 }
 
 static void
+dlb2_flush_port(struct rte_eventdev *dev, int port_id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	eventdev_stop_flush_t flush;
+	struct rte_event ev;
+	uint8_t dev_id;
+	void *arg;
+	int i;
+
+	flush = dev->dev_ops->dev_stop_flush;
+	dev_id = dev->data->dev_id;
+	arg = dev->data->dev_stop_flush_arg;
+
+	while (rte_event_dequeue_burst(dev_id, port_id, &ev, 1, 0)) {
+		if (flush)
+			flush(dev_id, ev, arg);
+
+		if (dlb2->ev_ports[port_id].qm_port.is_directed)
+			continue;
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+	}
+
+	/* Enqueue any additional outstanding releases */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = dlb2->ev_ports[port_id].outstanding_releases; i > 0; i--)
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+}
+
+static uint32_t
+dlb2_get_ldb_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_ldb_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_ldb_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_ldb_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static uint32_t
+dlb2_get_dir_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_dir_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_dir_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_dir_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	if (queue->qm_queue.is_directed)
+		return dlb2_get_dir_queue_depth(dlb2, queue);
+	else
+		return dlb2_get_ldb_queue_depth(dlb2, queue);
+}
+
+static bool
+dlb2_queue_is_empty(struct dlb2_eventdev *dlb2,
+		    struct dlb2_eventdev_queue *queue)
+{
+	return dlb2_get_queue_depth(dlb2, queue) == 0;
+}
+
+static bool
+dlb2_linked_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0)
+			continue;
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static bool
+dlb2_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static void
+dlb2_drain(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_port *ev_port = NULL;
+	uint8_t dev_id;
+	int i;
+
+	dev_id = dev->data->dev_id;
+
+	while (!dlb2_linked_queues_empty(dlb2)) {
+		/* Flush all the ev_ports, which will drain all their connected
+		 * queues.
+		 */
+		for (i = 0; i < dlb2->num_ports; i++)
+			dlb2_flush_port(dev, i);
+	}
+
+	/* The queues are empty, but there may be events left in the ports. */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_flush_port(dev, i);
+
+	/* If the domain's queues are empty, we're done. */
+	if (dlb2_queues_empty(dlb2))
+		return;
+
+	/* Else, there must be at least one unlinked load-balanced queue.
+	 * Select a load-balanced port with which to drain the unlinked
+	 * queue(s).
+	 */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		if (!ev_port->qm_port.is_directed)
+			break;
+	}
+
+	if (i == dlb2->num_ports) {
+		DLB2_LOG_ERR("internal error: no LDB ev_ports\n");
+		return;
+	}
+
+	rte_errno = 0;
+	rte_event_port_unlink(dev_id, ev_port->id, NULL, 0);
+
+	if (rte_errno) {
+		DLB2_LOG_ERR("internal error: failed to unlink ev_port %d\n",
+			     ev_port->id);
+		return;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		uint8_t qid, prio;
+		int ret;
+
+		if (dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			continue;
+
+		qid = i;
+		prio = 0;
+
+		/* Link the ev_port to the queue */
+		ret = rte_event_port_link(dev_id, ev_port->id, &qid, &prio, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to link ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+
+		/* Flush the queue */
+		while (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			dlb2_flush_port(dev, ev_port->id);
+
+		/* Drain any extant events in the ev_port. */
+		dlb2_flush_port(dev, ev_port->id);
+
+		/* Unlink the ev_port from the queue */
+		ret = rte_event_port_unlink(dev_id, ev_port->id, &qid, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to unlink ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+	}
+}
+
+static void
+dlb2_eventdev_stop(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_DBG("Internal error: already stopped\n");
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	} else if (dlb2->run_state != DLB2_RUN_STATE_STARTED) {
+		DLB2_LOG_ERR("Internal error: bad state %d for dev_stop\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPING;
+
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	dlb2_drain(dev);
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPED;
+}
+
+static int
+dlb2_eventdev_close(struct rte_eventdev *dev)
+{
+	dlb2_hw_reset_sched_domain(dev, false);
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3421,6 +3644,8 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.dev_start        = dlb2_eventdev_start,
+		.dev_stop         = dlb2_eventdev_stop,
+		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a86191d..5471dd8 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -66,3 +66,9 @@ int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				     struct dlb2_start_domain_args *cfg);
+
+int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 7ead374..b508eb0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -65,4 +65,10 @@ extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				struct dlb2_start_domain_args *cfg);
+
+extern int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+extern int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index c14a2f3..ae5ef2f 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5934,3 +5934,94 @@ dlb2_hw_start_domain(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_dir_queue_depth(hw, queue);
+
+	return 0;
+}
+
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+
+	dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index fff2e57..632c4e0 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -527,6 +527,56 @@ dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_ldb_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_ldb_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_dir_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret = 0;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_dir_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -544,6 +594,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth;
+	dlb2_iface_get_dir_queue_depth = dlb2_pf_get_dir_queue_depth;
 	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v5 20/23] event/dlb2: add PMD's token pop public interface

[Timothy McDaniel]

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/api/doxy-api-index.md         |  1 +
 drivers/event/dlb2/dlb2.c         | 53 ++++++++++++++++++++++++----
 drivers/event/dlb2/dlb2_priv.h    |  3 ++
 drivers/event/dlb2/meson.build    |  5 ++-
 drivers/event/dlb2/rte_pmd_dlb2.c | 39 +++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2.h | 72 +++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/version.map    |  6 ++++
 7 files changed, 172 insertions(+), 7 deletions(-)
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index a9c12d1..00f13e2 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -52,6 +52,7 @@ The public API headers are grouped by topics:
   [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
   [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
+  [dlb2]	       (@ref rte_pmd_dlb2.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index b02a70c..b25b213 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1196,7 +1196,7 @@ dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
 	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
 
 	qm_port->dequeue_depth = dequeue_depth;
-
+	qm_port->token_pop_thresh = dequeue_depth;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -1364,6 +1364,8 @@ dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
 
 	qm_port->dequeue_depth = dequeue_depth;
 
+	/* Directed ports are auto-pop, by default. */
+	qm_port->token_pop_mode = AUTO_POP;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -2611,6 +2613,14 @@ dlb2_event_enqueue_burst(void *event_port,
 		dlb2_event_build_hcws(qm_port, &events[i], j,
 				      sched_types, queue_ids);
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
@@ -2619,6 +2629,11 @@ dlb2_event_enqueue_burst(void *event_port,
 			break;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
 	return cnt;
 }
 
@@ -3101,11 +3116,25 @@ dlb2_event_release(struct dlb2_eventdev *dlb2,
 		if (j == 0)
 			break;
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
 sw_credit_update:
 	/* each release returns one credit */
 	if (!ev_port->outstanding_releases) {
@@ -3189,8 +3218,8 @@ dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3316,8 +3345,8 @@ dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3332,6 +3361,7 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 			 uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3347,6 +3377,9 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3366,6 +3399,7 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 				uint16_t num, uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3381,6 +3415,9 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3685,7 +3722,7 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    struct dlb2_devargs *dlb2_args)
 {
 	struct dlb2_eventdev *dlb2;
-	int err;
+	int err, i;
 
 	dlb2 = dev->data->dev_private;
 
@@ -3735,6 +3772,10 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Initialize each port's token pop mode */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
+		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 61567a6..b73cf3f 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -12,6 +12,7 @@
 #include <rte_config.h>
 #include "dlb2_user.h"
 #include "dlb2_log.h"
+#include "rte_pmd_dlb2.h"
 
 #ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
 #define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
@@ -290,6 +291,7 @@ struct dlb2_port {
 	bool gen_bit;
 	uint16_t dir_credits;
 	uint32_t dequeue_depth;
+	enum dlb2_token_pop_mode token_pop_mode;
 	union dlb2_port_config cfg;
 	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
 	uint16_t cached_ldb_credits;
@@ -298,6 +300,7 @@ struct dlb2_port {
 	bool int_armed;
 	uint16_t owed_tokens;
 	int16_t issued_releases;
+	int16_t token_pop_thresh;
 	int cq_depth;
 	uint16_t cq_idx;
 	uint16_t cq_idx_unmasked;
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index dc16c96..b734ff6 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,3 +1,4 @@
+
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
@@ -12,7 +13,9 @@ sources = files('dlb2.c',
 		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
-		'pf/base/dlb2_resource.c'
+		'pf/base/dlb2_resource.c',
+		'rte_pmd_dlb2.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
+install_headers('rte_pmd_dlb2.h')
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
new file mode 100644
index 0000000..b09b585
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "rte_pmd_dlb2.h"
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode)
+{
+	struct dlb2_eventdev *dlb2;
+	struct rte_eventdev *dev;
+
+	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+	dev = &rte_eventdevs[dev_id];
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (mode >= NUM_TOKEN_POP_MODES)
+		return -EINVAL;
+
+	/* The event device must be configured, but not yet started */
+	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
+		return -EINVAL;
+
+	/* The token pop mode must be set before configuring the port */
+	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
+		return -EINVAL;
+
+	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
new file mode 100644
index 0000000..74399db
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+/*!
+ *  @file      rte_pmd_dlb2.h
+ *
+ *  @brief     DLB PMD-specific functions
+ */
+
+#ifndef _RTE_PMD_DLB2_H_
+#define _RTE_PMD_DLB2_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Selects the token pop mode for a DLB2 port.
+ */
+enum dlb2_token_pop_mode {
+	/* Pop the CQ tokens immediately after dequeueing. */
+	AUTO_POP,
+	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
+	 * Supported on load-balanced ports only.
+	 */
+	DELAYED_POP,
+	/* Pop the CQ tokens during next dequeue operation. */
+	DEFERRED_POP,
+
+	/* NUM_TOKEN_POP_MODES must be last */
+	NUM_TOKEN_POP_MODES
+};
+
+/*!
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Configure the token pop mode for a DLB2 port. By default, all ports use
+ * AUTO_POP. This function must be called before calling rte_event_port_setup()
+ * for the port, but after calling rte_event_dev_configure().
+ *
+ * @param dev_id
+ *    The identifier of the event device.
+ * @param port_id
+ *    The identifier of the event port.
+ * @param mode
+ *    The token pop mode.
+ *
+ * @return
+ * - 0: Success
+ * - EINVAL: Invalid dev_id, port_id, or mode
+ * - EINVAL: The DLB2 is not configured, is already running, or the port is
+ *   already setup
+ */
+
+__rte_experimental
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_DLB2_H_ */
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
index 4a76d1d..b1e4dff 100644
--- a/drivers/event/dlb2/version.map
+++ b/drivers/event/dlb2/version.map
@@ -1,3 +1,9 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmd_dlb2_set_token_pop_mode;
+};
-- 
2.6.4

[dpdk-dev] [PATCH v5 21/23] event/dlb2: add PMD self-tests

[Timothy McDaniel]

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 app/test/test_eventdev.c           |    7 +
 drivers/event/dlb2/dlb2.c          |    1 +
 drivers/event/dlb2/dlb2_selftest.c | 1570 ++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build     |    3 +-
 4 files changed, 1580 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c

diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
index 62019c1..bcfaa53 100644
--- a/app/test/test_eventdev.c
+++ b/app/test/test_eventdev.c
@@ -1030,6 +1030,12 @@ test_eventdev_selftest_dpaa2(void)
 	return test_eventdev_selftest_impl("event_dpaa2", "");
 }
 
+static int
+test_eventdev_selftest_dlb2(void)
+{
+	return test_eventdev_selftest_impl("dlb2_event", "");
+}
+
 REGISTER_TEST_COMMAND(eventdev_common_autotest, test_eventdev_common);
 REGISTER_TEST_COMMAND(eventdev_selftest_sw, test_eventdev_selftest_sw);
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
@@ -1037,3 +1043,4 @@ REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
 		test_eventdev_selftest_octeontx2);
 REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2, test_eventdev_selftest_dpaa2);
+REGISTER_TEST_COMMAND(eventdev_selftest_dlb2, test_eventdev_selftest_dlb2);
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index b25b213..113fedf 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3696,6 +3696,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
 		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+		.dev_selftest     = test_dlb2_eventdev,
 	};
 
 	/* Expose PMD's eventdev interface */
diff --git a/drivers/event/dlb2/dlb2_selftest.c b/drivers/event/dlb2/dlb2_selftest.c
new file mode 100644
index 0000000..418a53b
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_selftest.c
@@ -0,0 +1,1570 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <unistd.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_eventdev.h>
+#include <rte_pause.h>
+
+#include "dlb2_priv.h"
+#include "rte_pmd_dlb2.h"
+
+#define MAX_PORTS 32
+#define MAX_QIDS 32
+#define DEFAULT_NUM_SEQ_NUMS 64
+
+static struct rte_mempool *eventdev_func_mempool;
+static int evdev;
+
+struct test {
+	struct rte_mempool *mbuf_pool;
+	int nb_qids;
+};
+
+/* initialization and config */
+static inline int
+init(struct test *t, int nb_queues, int nb_ports)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	memset(t, 0, sizeof(*t));
+
+	t->mbuf_pool = eventdev_func_mempool;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = nb_queues;
+	config.nb_event_ports = nb_ports;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0)
+		printf("%d: Error configuring device\n", __LINE__);
+
+	return ret;
+}
+
+static inline int
+create_ports(int num_ports)
+{
+	int i;
+
+	if (num_ports > MAX_PORTS)
+		return -1;
+
+	for (i = 0; i < num_ports; i++) {
+		struct rte_event_port_conf conf;
+
+		if (rte_event_port_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default port conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		if (rte_event_port_setup(evdev, i, &conf) < 0) {
+			printf("%d: Error setting up port %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+create_lb_qids(struct test *t, int num_qids, uint32_t flags)
+{
+	int i;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		struct rte_event_queue_conf conf;
+
+		if (rte_event_queue_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default queue conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		conf.schedule_type = flags;
+
+		if (conf.schedule_type == RTE_SCHED_TYPE_PARALLEL)
+			conf.nb_atomic_order_sequences = 0;
+		else
+			conf.nb_atomic_order_sequences = DEFAULT_NUM_SEQ_NUMS;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+static inline int
+create_atomic_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ATOMIC);
+}
+
+static inline int
+create_ordered_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ORDERED);
+}
+
+static inline int
+create_unordered_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_PARALLEL);
+}
+
+static inline int
+create_directed_qids(struct test *t, int num_qids, const uint8_t ports[])
+{
+	static struct rte_event_queue_conf conf;
+	int i;
+
+	conf.priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
+	conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		uint8_t queue_id;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, ports[i - t->nb_qids],
+					&queue_id, NULL, 1) != 1) {
+			printf("%d: error creating link for qid %d\n",
+			       __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+/* destruction */
+static inline void
+cleanup(void)
+{
+	int ret = 0;
+
+	rte_event_dev_stop(evdev);
+	ret = rte_event_dev_close(evdev);
+
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+};
+
+static inline int
+enqueue_timeout(uint8_t port_id, struct rte_event *ev, uint64_t tmo_us)
+{
+	const uint64_t start = rte_get_timer_cycles();
+	const uint64_t ticks = (tmo_us * rte_get_timer_hz()) / 1E6;
+
+	while ((rte_get_timer_cycles() - start) < ticks) {
+		if (rte_event_enqueue_burst(evdev, port_id, ev, 1) == 1)
+			return 0;
+
+		if (rte_errno != -ENOSPC) {
+			printf("enqueue_burst returned rte_errno %d\n",
+			       rte_errno);
+			return -1;
+		}
+	}
+	printf("%s time out\n", __func__);
+	return -1;
+}
+
+static void
+flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused)
+{
+	rte_pktmbuf_free(event.mbuf);
+}
+
+static int
+test_stop_flush(struct test *t) /* test to check we can properly flush events */
+{
+	struct rte_event ev;
+	uint32_t dequeue_depth;
+	unsigned int i, count;
+	uint8_t queue_id;
+
+	ev.op = RTE_EVENT_OP_NEW;
+
+	if (init(t, 2, 1) < 0 ||
+	    create_ports(1) < 0 ||
+	    create_atomic_qids(t, 2) < 0) {
+		printf("%d: Error initializing device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_port_link(evdev, 0, NULL, NULL, 0) != 2) {
+		printf("%d: Error linking queues to the port\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: Error with start call\n", __LINE__);
+		goto err;
+	}
+
+	/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked
+	 * queue.
+	 */
+	queue_id = 1;
+
+	if (rte_event_port_unlink(evdev, 0, &queue_id, 1) != 1) {
+		printf("%d: Error unlinking queue 1 from port\n", __LINE__);
+		goto err;
+	}
+
+	count = rte_mempool_avail_count(t->mbuf_pool);
+
+	if (rte_event_port_attr_get(evdev,
+				    0,
+				    RTE_EVENT_PORT_ATTR_DEQ_DEPTH,
+				    &dequeue_depth)) {
+		printf("%d: Error retrieveing dequeue depth\n", __LINE__);
+		goto err;
+	}
+
+	/* Send QEs to queue 0 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 0;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Send QEs to queue 1 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 1;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Now the DLB is scheduling events from the port to the IQ, and at
+	 * least one event should be remaining in each queue.
+	 */
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, flush, NULL)) {
+		printf("%d: Error installing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+
+	if (count != rte_mempool_avail_count(t->mbuf_pool)) {
+		printf("%d: Error executing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, NULL, NULL)) {
+		printf("%d: Error uninstalling the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	return 0;
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_single_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Create a directed port */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) == 0) {
+		printf("%d: port 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Create a load-balanced port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a directed queue */
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed queue */
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) == 0) {
+		printf("%d: queue 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a load-balanced queue */
+	queue_conf.event_queue_cfg = 0;
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to link directed and load-balanced resources */
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 0 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 1 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Link ports to queues */
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+#define NUM_LDB_PORTS 64
+#define NUM_LDB_QUEUES 32
+
+static int
+test_info_get(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	config.nb_event_ports = info.max_event_ports;
+	config.nb_event_queues = NUM_LDB_QUEUES + info.max_event_ports / 2;
+	config.nb_single_link_event_port_queues = info.max_event_ports / 2;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		goto err;
+	}
+
+	/* The DLB2 PMD only reports load-balanced ports and queues in its
+	 * info_get function. Confirm that these values don't include the
+	 * directed port or queue counts.
+	 */
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret) {
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+		return -1;
+	}
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+static int
+test_reconfiguration_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 2 LDB queues */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+			printf("%d: P0->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P1->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+			printf("%d: P1->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Configure device with 2 DIR ports and 2 DIR queues */
+	config.nb_single_link_event_port_queues = 2;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q1 */
+	queue_id = 1;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: P0->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Link P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: P1->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_queues = 5;
+	config.nb_event_ports = 5;
+	config.nb_single_link_event_port_queues = 1;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < config.nb_event_queues - 1; i++) {
+		port_conf.event_port_cfg = 0;
+		queue_conf.event_queue_cfg = 0;
+
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: P%d->Q%d link expected to succeed\n",
+			       __LINE__, i, i);
+			goto err;
+		}
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+		printf("%d: port %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+		printf("%d: queue %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	queue_id = i;
+
+	if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+		printf("%d: P%d->Q%d link expected to succeed\n",
+		       __LINE__, i, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_ports += 1;
+
+	/* Reconfigure device with 1 more load-balanced port */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Configure the new port */
+	if (rte_event_port_setup(evdev, config.nb_event_ports - 1,
+				 &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_load_balanced_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_directed_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 DIR port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (ev.queue_id != 0) {
+		printf("%d: invalid dequeued event queue ID (%d)\n",
+		       __LINE__, ev.queue_id);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_deferred_sched(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	const int num_events = 128;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 1 queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 1, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.dequeue_depth = 1;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.schedule_type = RTE_SCHED_TYPE_PARALLEL;
+	queue_conf.nb_atomic_order_sequences = 0;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 128 NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_PARALLEL;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue one event from port 0 */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue (and release) all other events from port 1. Deferred
+	 * scheduling ensures no other events are scheduled to port 0 without a
+	 * subsequent rte_event_dequeue_burst() call.
+	 */
+	for (i = 0; i < num_events - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 1, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		if (rte_event_enqueue_burst(evdev, 1, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_delayed_pop(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	int ret, i, num_events;
+	struct rte_event ev;
+	uint8_t queue_id;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DELAYED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	num_events = 2 * port_conf.dequeue_depth;
+
+	/* Enqueue 2 * dequeue_depth NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue dequeue_depth events but only release dequeue_depth - 2.
+	 * Delayed pop won't perform the pop and no more events will be
+	 * scheduled.
+	 */
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = 0; i < port_conf.dequeue_depth - 2; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	ret = rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout);
+	if (ret != 0) {
+		printf("%d: event dequeue expected to fail (ret = %d)\n",
+		       __LINE__, ret);
+		goto err;
+	}
+
+	/* Release one more event. This will trigger the token pop, and
+	 * dequeue_depth - 1 more events will be scheduled to the device.
+	 */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: RELEASE enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 0) {
+		printf("%d: event dequeue expected to fail\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+do_selftest(void)
+{
+	struct test t;
+	int ret;
+
+	/* Only create mbuf pool once, reuse for each test run */
+	if (!eventdev_func_mempool) {
+		eventdev_func_mempool =
+			rte_pktmbuf_pool_create("EVENTDEV_DLB2_ST_POOL",
+						(1 << 12), /* 4k buffers */
+						32 /*MBUF_CACHE_SIZE*/,
+						0,
+						512, /* use very small mbufs */
+						rte_socket_id());
+		if (!eventdev_func_mempool) {
+			printf("ERROR creating mempool\n");
+			goto test_fail;
+		}
+	}
+	t.mbuf_pool = eventdev_func_mempool;
+
+	printf("*** Running Stop Flush test...\n");
+	ret = test_stop_flush(&t);
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Single Link test...\n");
+	ret = test_single_link();
+	if (ret != 0) {
+		printf("ERROR - Single Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Info Get test...\n");
+	ret = test_info_get();
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Reconfiguration Link test...\n");
+	ret = test_reconfiguration_link();
+	if (ret != 0) {
+		printf("ERROR - Reconfiguration Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Load-Balanced Traffic test...\n");
+	ret = test_load_balanced_traffic();
+	if (ret != 0) {
+		printf("ERROR - Load-Balanced Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Directed Traffic test...\n");
+	ret = test_directed_traffic();
+	if (ret != 0) {
+		printf("ERROR - Directed Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Deferred Scheduling test...\n");
+	ret = test_deferred_sched();
+	if (ret != 0) {
+		printf("ERROR - Deferred Scheduling test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Delayed Pop test...\n");
+	ret = test_delayed_pop();
+	if (ret != 0) {
+		printf("ERROR - Delayed Pop test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	return 0;
+
+test_fail:
+	return -1;
+}
+
+int
+test_dlb2_eventdev(void)
+{
+	const char *dlb2_eventdev_name = "dlb2_event";
+	uint8_t num_evdevs = rte_event_dev_count();
+	int i, ret = 0;
+	int found = 0, skipped = 0, passed = 0, failed = 0;
+	struct rte_event_dev_info info;
+
+	for (i = 0; found + skipped < num_evdevs && i < RTE_EVENT_MAX_DEVS;
+	     i++) {
+		ret = rte_event_dev_info_get(i, &info);
+		if (ret < 0)
+			continue;
+
+		/* skip non-dlb2 event devices */
+		if (strncmp(info.driver_name, dlb2_eventdev_name,
+			    sizeof(*info.driver_name)) != 0) {
+			skipped++;
+			continue;
+		}
+
+		evdev = rte_event_dev_get_dev_id(info.driver_name);
+		if (evdev < 0) {
+			printf("Could not get dev_id for eventdev with name %s, i=%d\n",
+			       info.driver_name, i);
+			skipped++;
+			continue;
+		}
+		found++;
+		printf("Running selftest on eventdev %s\n", info.driver_name);
+		ret = do_selftest();
+		if (ret == 0) {
+			passed++;
+			printf("Selftest passed for eventdev %s\n",
+			       info.driver_name);
+		} else {
+			failed++;
+			printf("Selftest failed for eventdev %s, err=%d\n",
+			       info.driver_name, ret);
+		}
+	}
+
+	printf("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n",
+	       found, skipped, passed, failed);
+	return ret;
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index b734ff6..340bc57 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -14,7 +14,8 @@ sources = files('dlb2.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c',
-		'rte_pmd_dlb2.c'
+		'rte_pmd_dlb2.c',
+		'dlb2_selftest.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v5 22/23] event/dlb2: add queue and port release

[Timothy McDaniel]

DLB does not support reconfiguring individual queues
or ports on the fly. The entire device must be reconfigured.
Previously allocated port QE and memzone memory
is freed in this patch.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 113fedf..4b4719f 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3672,6 +3672,28 @@ dlb2_eventdev_close(struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(id);
+
+	/* This function intentionally left blank. */
+}
+
+static void
+dlb2_eventdev_port_release(void *port)
+{
+	struct dlb2_eventdev_port *ev_port = port;
+	struct dlb2_port *qm_port;
+
+	if (ev_port) {
+		qm_port = &ev_port->qm_port;
+		if (qm_port->config_state == DLB2_CONFIGURED)
+			dlb2_free_qe_mem(qm_port);
+	}
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3685,8 +3707,10 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
+		.queue_release    = dlb2_eventdev_queue_release,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_release     = dlb2_eventdev_port_release,
 		.port_link        = dlb2_eventdev_port_link,
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
-- 
2.6.4

[dpdk-dev] [PATCH v5 23/23] event/dlb2: add timeout ticks entry point

[Timothy McDaniel]

Adds the timeout ticks conversion function.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 4b4719f..dea61fc 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3693,6 +3693,18 @@ dlb2_eventdev_port_release(void *port)
 	}
 }
 
+static int
+dlb2_eventdev_timeout_ticks(struct rte_eventdev *dev, uint64_t ns,
+			    uint64_t *timeout_ticks)
+{
+	RTE_SET_USED(dev);
+	uint64_t cycles_per_ns = rte_get_timer_hz() / 1E9;
+
+	*timeout_ticks = ns * cycles_per_ns;
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -3715,6 +3727,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
 				    dlb2_eventdev_port_unlinks_in_progress,
+		.timeout_ticks    = dlb2_eventdev_timeout_ticks,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[Thomas Monjalon]

30/10/2020 10:43, Timothy McDaniel:
> - note that the code still uses its private byte-encoded versions of
>   umonitor/umwait, rather than the new functions in the power
>   patch that are built on top of those intrinsics. This is intentional.

Why? Now these intrinsics are available in the main branch.
We should avoid duplicating such code.

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[McDaniel, Timothy]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Friday, October 30, 2020 5:02 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com; david.marchand@redhat.com
> Subject: Re: [PATCH v5 00/23] Add DLB2 PMD
> 
> 30/10/2020 10:43, Timothy McDaniel:
> > - note that the code still uses its private byte-encoded versions of
> >   umonitor/umwait, rather than the new functions in the power
> >   patch that are built on top of those intrinsics. This is intentional.
> 
> Why? Now these intrinsics are available in the main branch.
> We should avoid duplicating such code.
> 
> 

I had asked that the low level intrinsics (UMWAIT/UMONITOR) be split out so that DLB/DLB2 could use them instead of its own private byte-encoded versions, but instead we have these wrappers that call the low level intrinsics. Those wrappers
introduce additional overhead that is not required for DLB/DLB2. I have a meeting with Ma Liang on Monday to discuss.

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[Jerin Jacob]

On Fri, Oct 30, 2020 at 3:46 PM McDaniel, Timothy
<timothy.mcdaniel@intel.com> wrote:
>
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Friday, October 30, 2020 5:02 AM
> > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> > <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> > jerinj@marvell.com; david.marchand@redhat.com
> > Subject: Re: [PATCH v5 00/23] Add DLB2 PMD
> >
> > 30/10/2020 10:43, Timothy McDaniel:
> > > - note that the code still uses its private byte-encoded versions of
> > >   umonitor/umwait, rather than the new functions in the power
> > >   patch that are built on top of those intrinsics. This is intentional.
> >
> > Why? Now these intrinsics are available in the main branch.
> > We should avoid duplicating such code.
> >
> >
>
> I had asked that the low level intrinsics (UMWAIT/UMONITOR) be split out so that DLB/DLB2 could use them instead of its own private byte-encoded versions, but instead we have these wrappers that call the low level intrinsics. Those wrappers
> introduce additional overhead that is not required for DLB/DLB2. I have a meeting with Ma Liang on Monday to discuss.

Then why we merged the EAL patches? The all-purpose was to use this by
other subsystems. If it is only for the power library then we should
make specific to the power library.

Thomas, Should I take this series in eventdev or I need to wait to
sort out this?

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[Thomas Monjalon]

30/10/2020 11:32, Jerin Jacob:
> McDaniel, Timothy <timothy.mcdaniel@intel.com> wrote:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 30/10/2020 10:43, Timothy McDaniel:
> > > > - note that the code still uses its private byte-encoded versions of
> > > >   umonitor/umwait, rather than the new functions in the power
> > > >   patch that are built on top of those intrinsics. This is intentional.
> > >
> > > Why? Now these intrinsics are available in the main branch.
> > > We should avoid duplicating such code.
> >
> > I had asked that the low level intrinsics (UMWAIT/UMONITOR)
> > be split out so that DLB/DLB2 could use them instead
> > of its own private byte-encoded versions,
> > but instead we have these wrappers that call the low level
> > intrinsics. Those wrappers introduce additional overhead
> > that is not required for DLB/DLB2.
> > I have a meeting with Ma Liang on Monday to discuss.

Why did not you tell it on the mailing list?
It would have prevented from merging a wrong/useless API.

I am now convinced that the hard push to get those intrinsics
which started with a lack of communication (no roadmap, no Cc, no reply)
is really a bad story in the community process.

> Then why we merged the EAL patches? The all-purpose was to use this by
> other subsystems. If it is only for the power library then we should
> make specific to the power library.

I agree with you Jerin.

> Thomas, Should I take this series in eventdev
> or I need to wait to sort out this?

I think you could merge those patches and I revert the EAL ones.
In any case, I won't merge any more patch about power monitor
in this release. I don't like swimming in the fog.

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[McDaniel, Timothy]

> -----Original Message-----
> From: McDaniel, Timothy
> Sent: Friday, October 30, 2020 5:17 AM
> To: Thomas Monjalon <thomas@monjalon.net>
> Cc: dev@dpdk.org; Carrillo, Erik G <Erik.G.Carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com; david.marchand@redhat.com
> Subject: RE: [PATCH v5 00/23] Add DLB2 PMD
> 
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Friday, October 30, 2020 5:02 AM
> > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> > <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> > jerinj@marvell.com; david.marchand@redhat.com
> > Subject: Re: [PATCH v5 00/23] Add DLB2 PMD
> >
> > 30/10/2020 10:43, Timothy McDaniel:
> > > - note that the code still uses its private byte-encoded versions of
> > >   umonitor/umwait, rather than the new functions in the power
> > >   patch that are built on top of those intrinsics. This is intentional.
> >
> > Why? Now these intrinsics are available in the main branch.
> > We should avoid duplicating such code.
> >
> >
> 
> I had asked that the low level intrinsics (UMWAIT/UMONITOR) be split out so
> that DLB/DLB2 could use them instead of its own private byte-encoded versions,
> but instead we have these wrappers that call the low level intrinsics. Those
> wrappers
> introduce additional overhead that is not required for DLB/DLB2. I have a
> meeting with Ma Liang on Monday to discuss.

I thought the ask of DLB was to just substitute the low level umwait/umonitor byte
encoded instructions DLB has defined privately with similar byte-encoded instructions defined in the power
patch. The power patch does not directly expose those, which is why I did not update DLB/DLB2.
The power patch does have the advantage of centralizing the race avoidance
logic, which is a good thing for any PMD that wishes to take advantage of umwait/umonitor.  

Sorry for the confusion. I just misunderstood what was being asked of DLB in regard to switching over..  That being said, 
I am willing to convert DLB/DLB2 to use  rte_power_monitor(...) in a future patch-set.

Thanks,
Tim

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[Thomas Monjalon]

30/10/2020 12:58, McDaniel, Timothy:
> From: McDaniel, Timothy
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 30/10/2020 10:43, Timothy McDaniel:
> > > > - note that the code still uses its private byte-encoded versions of
> > > >   umonitor/umwait, rather than the new functions in the power
> > > >   patch that are built on top of those intrinsics. This is intentional.
> > >
> > > Why? Now these intrinsics are available in the main branch.
> > > We should avoid duplicating such code.
> > >
> > >
> > 
> > I had asked that the low level intrinsics (UMWAIT/UMONITOR) be split out so
> > that DLB/DLB2 could use them instead of its own private byte-encoded versions,
> > but instead we have these wrappers that call the low level intrinsics. Those
> > wrappers
> > introduce additional overhead that is not required for DLB/DLB2. I have a
> > meeting with Ma Liang on Monday to discuss.
> 
> I thought the ask of DLB was to just substitute the low level umwait/umonitor byte
> encoded instructions DLB has defined privately with similar byte-encoded instructions defined in the power
> patch. The power patch does not directly expose those, which is why I did not update DLB/DLB2.
> The power patch does have the advantage of centralizing the race avoidance
> logic, which is a good thing for any PMD that wishes to take advantage of umwait/umonitor.

So you mean the overhead is a good thing?

> Sorry for the confusion. I just misunderstood what was being asked of DLB in regard to switching over..  That being said, 
> I am willing to convert DLB/DLB2 to use  rte_power_monitor(...) in a future patch-set.

Why not now?

Indeed there is a confusion and it looks like a lot of novlang
to exit from the situation.
We'll wait a clear decision with facts.

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[McDaniel, Timothy]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Friday, October 30, 2020 8:15 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: 'dev@dpdk.org' <dev@dpdk.org>; Carrillo, Erik G
> <erik.g.carrillo@intel.com>; Eads, Gage <gage.eads@intel.com>; Van Haaren,
> Harry <harry.van.haaren@intel.com>; 'jerinj@marvell.com'
> <jerinj@marvell.com>; 'david.marchand@redhat.com'
> <david.marchand@redhat.com>
> Subject: Re: [PATCH v5 00/23] Add DLB2 PMD
> 
> 30/10/2020 12:58, McDaniel, Timothy:
> > From: McDaniel, Timothy
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > 30/10/2020 10:43, Timothy McDaniel:
> > > > > - note that the code still uses its private byte-encoded versions of
> > > > >   umonitor/umwait, rather than the new functions in the power
> > > > >   patch that are built on top of those intrinsics. This is intentional.
> > > >
> > > > Why? Now these intrinsics are available in the main branch.
> > > > We should avoid duplicating such code.
> > > >
> > > >
> > >
> > > I had asked that the low level intrinsics (UMWAIT/UMONITOR) be split out so
> > > that DLB/DLB2 could use them instead of its own private byte-encoded
> versions,
> > > but instead we have these wrappers that call the low level intrinsics. Those
> > > wrappers
> > > introduce additional overhead that is not required for DLB/DLB2. I have a
> > > meeting with Ma Liang on Monday to discuss.
> >
> > I thought the ask of DLB was to just substitute the low level umwait/umonitor
> byte
> > encoded instructions DLB has defined privately with similar byte-encoded
> instructions defined in the power
> > patch. The power patch does not directly expose those, which is why I did not
> update DLB/DLB2.
> > The power patch does have the advantage of centralizing the race avoidance
> > logic, which is a good thing for any PMD that wishes to take advantage of
> umwait/umonitor.
> 
> So you mean the overhead is a good thing?
> 
> > Sorry for the confusion. I just misunderstood what was being asked of DLB in
> regard to switching over..  That being said,
> > I am willing to convert DLB/DLB2 to use  rte_power_monitor(...) in a future
> patch-set.
> 
> Why not now?
> 
> Indeed there is a confusion and it looks like a lot of novlang
> to exit from the situation.
> We'll wait a clear decision with facts.
> 

Hi Thomas,

I have updated DLB and DLB2 to use rte_power_monitor(...), and those patches are
ready if you are willing to accept them and the 3 power patches.

For the sake of consistency, I see the benefit of using the power patch, even if it is 
slightly less efficient that the DLB-specific implementation that I currently have.
We have already encountered an empty queue, so this is no longer fast path for the PMD.

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[Thomas Monjalon]

30/10/2020 16:35, McDaniel, Timothy:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 30/10/2020 12:58, McDaniel, Timothy:
> > > From: McDaniel, Timothy
> > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > > 30/10/2020 10:43, Timothy McDaniel:
> > > > > > - note that the code still uses its private byte-encoded versions of
> > > > > >   umonitor/umwait, rather than the new functions in the power
> > > > > >   patch that are built on top of those intrinsics. This is intentional.
> > > > >
> > > > > Why? Now these intrinsics are available in the main branch.
> > > > > We should avoid duplicating such code.
> > > > >
> > > > >
> > > >
> > > > I had asked that the low level intrinsics (UMWAIT/UMONITOR) be split out so
> > > > that DLB/DLB2 could use them instead of its own private byte-encoded
> > versions,
> > > > but instead we have these wrappers that call the low level intrinsics. Those
> > > > wrappers
> > > > introduce additional overhead that is not required for DLB/DLB2. I have a
> > > > meeting with Ma Liang on Monday to discuss.
> > >
> > > I thought the ask of DLB was to just substitute the low level umwait/umonitor
> > byte
> > > encoded instructions DLB has defined privately with similar byte-encoded
> > instructions defined in the power
> > > patch. The power patch does not directly expose those, which is why I did not
> > update DLB/DLB2.
> > > The power patch does have the advantage of centralizing the race avoidance
> > > logic, which is a good thing for any PMD that wishes to take advantage of
> > umwait/umonitor.
> > 
> > So you mean the overhead is a good thing?
> > 
> > > Sorry for the confusion. I just misunderstood what was being asked of DLB in
> > regard to switching over..  That being said,
> > > I am willing to convert DLB/DLB2 to use  rte_power_monitor(...) in a future
> > patch-set.
> > 
> > Why not now?
> > 
> > Indeed there is a confusion and it looks like a lot of novlang
> > to exit from the situation.
> > We'll wait a clear decision with facts.
> > 
> 
> Hi Thomas,
> 
> I have updated DLB and DLB2 to use rte_power_monitor(...), and those patches are
> ready if you are willing to accept them and the 3 power patches.
> 
> For the sake of consistency, I see the benefit of using the power patch, even if it is 
> slightly less efficient that the DLB-specific implementation that I currently have.
> We have already encountered an empty queue, so this is no longer fast path for the PMD.

I am really concerned that the API in EAL is not the most efficient.
Why is that? Can we improve the EAL API?

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[McDaniel, Timothy]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Friday, October 30, 2020 10:48 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: 'dev@dpdk.org' <dev@dpdk.org>; Carrillo, Erik G
> <erik.g.carrillo@intel.com>; Eads, Gage <gage.eads@intel.com>; Van Haaren,
> Harry <harry.van.haaren@intel.com>; 'jerinj@marvell.com'
> <jerinj@marvell.com>; 'david.marchand@redhat.com'
> <david.marchand@redhat.com>
> Subject: Re: [PATCH v5 00/23] Add DLB2 PMD
> 
> 30/10/2020 16:35, McDaniel, Timothy:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 30/10/2020 12:58, McDaniel, Timothy:
> > > > From: McDaniel, Timothy
> > > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > > > 30/10/2020 10:43, Timothy McDaniel:
> > > > > > > - note that the code still uses its private byte-encoded versions of
> > > > > > >   umonitor/umwait, rather than the new functions in the power
> > > > > > >   patch that are built on top of those intrinsics. This is intentional.
> > > > > >
> > > > > > Why? Now these intrinsics are available in the main branch.
> > > > > > We should avoid duplicating such code.
> > > > > >
> > > > > >
> > > > >
> > > > > I had asked that the low level intrinsics (UMWAIT/UMONITOR) be split
> out so
> > > > > that DLB/DLB2 could use them instead of its own private byte-encoded
> > > versions,
> > > > > but instead we have these wrappers that call the low level intrinsics.
> Those
> > > > > wrappers
> > > > > introduce additional overhead that is not required for DLB/DLB2. I have a
> > > > > meeting with Ma Liang on Monday to discuss.
> > > >
> > > > I thought the ask of DLB was to just substitute the low level
> umwait/umonitor
> > > byte
> > > > encoded instructions DLB has defined privately with similar byte-encoded
> > > instructions defined in the power
> > > > patch. The power patch does not directly expose those, which is why I did
> not
> > > update DLB/DLB2.
> > > > The power patch does have the advantage of centralizing the race
> avoidance
> > > > logic, which is a good thing for any PMD that wishes to take advantage of
> > > umwait/umonitor.
> > >
> > > So you mean the overhead is a good thing?
> > >
> > > > Sorry for the confusion. I just misunderstood what was being asked of DLB
> in
> > > regard to switching over..  That being said,
> > > > I am willing to convert DLB/DLB2 to use  rte_power_monitor(...) in a future
> > > patch-set.
> > >
> > > Why not now?
> > >
> > > Indeed there is a confusion and it looks like a lot of novlang
> > > to exit from the situation.
> > > We'll wait a clear decision with facts.
> > >
> >
> > Hi Thomas,
> >
> > I have updated DLB and DLB2 to use rte_power_monitor(...), and those
> patches are
> > ready if you are willing to accept them and the 3 power patches.
> >
> > For the sake of consistency, I see the benefit of using the power patch, even if
> it is
> > slightly less efficient that the DLB-specific implementation that I currently
> have.
> > We have already encountered an empty queue, so this is no longer fast path
> for the PMD.
> 
> I am really concerned that the API in EAL is not the most efficient.
> Why is that? Can we improve the EAL API?
> 

From an efficiency perspective, I only noticed 2 things.
1) The size check and associated logic. This could be avoided if we had _8, _16, _32, _64 variants instead of 1 function that handled all possibilities, but even that may be a wash
with branch prediction 
2) The spinlock - but this observation was my mistake, and was flawed. I was looking at the rte_power_monitor_sync(...), and not rte_power_monitor(...), the latter of which does not take a spinlock.

In summary, I am no longer concerned about efficiency and suitability for DLB/DLB2.

Thanks,
Tim

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[Thomas Monjalon]

30/10/2020 17:02, McDaniel, Timothy:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 30/10/2020 16:35, McDaniel, Timothy:
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > 30/10/2020 12:58, McDaniel, Timothy:
> > > > > From: McDaniel, Timothy
> > > > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > > > > 30/10/2020 10:43, Timothy McDaniel:
> > > > > > > > - note that the code still uses its private byte-encoded versions of
> > > > > > > >   umonitor/umwait, rather than the new functions in the power
> > > > > > > >   patch that are built on top of those intrinsics. This is intentional.
> > > > > > >
> > > > > > > Why? Now these intrinsics are available in the main branch.
> > > > > > > We should avoid duplicating such code.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > I had asked that the low level intrinsics (UMWAIT/UMONITOR) be split
> > out so
> > > > > > that DLB/DLB2 could use them instead of its own private byte-encoded
> > > > versions,
> > > > > > but instead we have these wrappers that call the low level intrinsics.
> > Those
> > > > > > wrappers
> > > > > > introduce additional overhead that is not required for DLB/DLB2. I have a
> > > > > > meeting with Ma Liang on Monday to discuss.
> > > > >
> > > > > I thought the ask of DLB was to just substitute the low level
> > umwait/umonitor
> > > > byte
> > > > > encoded instructions DLB has defined privately with similar byte-encoded
> > > > instructions defined in the power
> > > > > patch. The power patch does not directly expose those, which is why I did
> > not
> > > > update DLB/DLB2.
> > > > > The power patch does have the advantage of centralizing the race
> > avoidance
> > > > > logic, which is a good thing for any PMD that wishes to take advantage of
> > > > umwait/umonitor.
> > > >
> > > > So you mean the overhead is a good thing?
> > > >
> > > > > Sorry for the confusion. I just misunderstood what was being asked of DLB
> > in
> > > > regard to switching over..  That being said,
> > > > > I am willing to convert DLB/DLB2 to use  rte_power_monitor(...) in a future
> > > > patch-set.
> > > >
> > > > Why not now?
> > > >
> > > > Indeed there is a confusion and it looks like a lot of novlang
> > > > to exit from the situation.
> > > > We'll wait a clear decision with facts.
> > > >
> > >
> > > Hi Thomas,
> > >
> > > I have updated DLB and DLB2 to use rte_power_monitor(...), and those
> > patches are
> > > ready if you are willing to accept them and the 3 power patches.
> > >
> > > For the sake of consistency, I see the benefit of using the power patch, even if
> > it is
> > > slightly less efficient that the DLB-specific implementation that I currently
> > have.
> > > We have already encountered an empty queue, so this is no longer fast path
> > for the PMD.
> > 
> > I am really concerned that the API in EAL is not the most efficient.
> > Why is that? Can we improve the EAL API?
> > 
> 
> From an efficiency perspective, I only noticed 2 things.
> 1) The size check and associated logic. This could be avoided if we had _8, _16, _32, _64 variants instead of 1 function that handled all possibilities, but even that may be a wash
> with branch prediction 
> 2) The spinlock - but this observation was my mistake, and was flawed. I was looking at the rte_power_monitor_sync(...), and not rte_power_monitor(...), the latter of which does not take a spinlock.
> 
> In summary, I am no longer concerned about efficiency and suitability for DLB/DLB2.

OK, so let's go ahead with a DLB PMD using this API.

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[Jerin Jacob]

On Fri, Oct 30, 2020 at 3:19 PM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:

>
> Timothy McDaniel (23):
>   event/dlb2: add documentation and meson build infrastructure
>   event/dlb2: add dynamic logging
>   event/dlb2: add private data structures and constants
>   event/dlb2: add definitions shared with LKM or shared code
>   event/dlb2: add inline functions
>   event/dlb2: add eventdev probe

There is build error with clang  and static build here.
Please send the next version with fix.

meson  -Dexamples=l3fwd --buildtype=debugoptimized --werror
--default-library=static /export/dpdk-next-eventdev/devtools/..
./build-clang-static
The Meson build system
Version: 0.55.3
Source dir: /export/dpdk-next-eventdev
Build dir: /export/dpdk-next-eventdev/build-clang-static
Build type: native build
Program cat found: YES
Using 'PKG_CONFIG_PATH' from environment with value: ''
Using 'PKG_CONFIG_PATH' from environment with value: ''
Project name: DPDK
Project version: 20.11.0-rc1
Using 'CC' from environment with value: 'ccache clang'
Using 'CFLAGS' from environment with value: ''
Using 'LDFLAGS' from environment with value: ''
Using 'CPPFLAGS' from environment with value: ''
Using 'CC' from environment with value: 'ccache clang'
Using 'CFLAGS' from environment with value: ''
Using 'LDFLAGS' from environment with value: ''
Using 'CPPFLAGS' from environment with value: ''
C compiler for the host machine: ccache clang (clang 10.0.1 "clang
version 10.0.1 ")
C linker for the host machine: clang ld.bfd 2.35.1
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program pkg-config found: YES
Program gen-pmdinfo-cfile.sh found: YES
Program list-dir-globs.py found: YES
Program check-symbols.sh found: YES
Program options-ibverbs-static.sh found: YES
Program binutils-avx512-check.sh found: YES
Program python3 found: YES (/usr/bin/python)



ccache clang -Idrivers/libtmp_rte_event_dlb2.a.p -Idrivers
-I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
-Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
-I../config -Ilib/librte_eal/include -I../lib/librte_eal/i
nclude -Ilib/librte_eal/linux/include
-I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
-I../lib/librte_eal/x86/include -Ilib/librte_eal/common
-I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
-Ilib/librte_kvargs
 -I../lib/librte_kvargs -Ilib/librte_metrics -I../lib/librte_metrics
-Ilib/librte_telemetry -I../lib/librte_telemetry -Ilib/librte_ring
-I../lib/librte_ring -Ilib/librte_ethdev -I../lib/librte_ethdev
-Ilib/librte_net -I../lib/librte_net -Il
ib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
-I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
-Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_rcu
-I../lib/librte_rcu -Ilib/librte_timer -I../lib/librte_timer -
Ilib/librte_cryptodev -I../lib/librte_cryptodev -Ilib/librte_pci
-I../lib/librte_pci -Idrivers/bus/pci -I../drivers/bus/pci
-I../drivers/bus/pci/linux -Xclang -fcolor-diagnostics -pipe
-D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2
-g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
-Wformat-nonliteral -Wformat-security -Wmissing-declarations
-Wmissing-prototypes -Wnested-externs -Wold-style-definition
-Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -
Wwrite-strings -Wno-address-of-packed-member
-Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
-DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o -MF
drivers/l
ibtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o.d -o
drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o -c
../drivers/event/dlb2/pf/dlb2_main.c
In file included from ../drivers/event/dlb2/pf/dlb2_main.c:22:
../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: error: use of
unknown builtin '__builtin_ia32_movntdq'
[-Wimplicit-function-declaration]
        __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
        ^
../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: note: did you mean
'__builtin_ia32_movntq'?
/usr/lib/clang/10.0.1/include/xmmintrin.h:2122:3: note:
'__builtin_ia32_movntq' declared here
  __builtin_ia32_movntq(__p, __a);

  [2010/2491] Compiling C object
drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o
FAILED: drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o
ccache clang -Idrivers/libtmp_rte_event_dlb2.a.p -Idrivers
-I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
-Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
-I../config -Ilib/librte_eal/include -I../lib/librte_eal/i
nclude -Ilib/librte_eal/linux/include
-I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
-I../lib/librte_eal/x86/include -Ilib/librte_eal/common
-I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
-Ilib/librte_kvargs
 -I../lib/librte_kvargs -Ilib/librte_metrics -I../lib/librte_metrics
-Ilib/librte_telemetry -I../lib/librte_telemetry -Ilib/librte_ring
-I../lib/librte_ring -Ilib/librte_ethdev -I../lib/librte_ethdev
-Ilib/librte_net -I../lib/librte_net -Il
ib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
-I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
-Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_rcu
-I../lib/librte_rcu -Ilib/librte_timer -I../lib/librte_timer -
Ilib/librte_cryptodev -I../lib/librte_cryptodev -Ilib/librte_pci
-I../lib/librte_pci -Idrivers/bus/pci -I../drivers/bus/pci
-I../drivers/bus/pci/linux -Xclang -fcolor-diagnostics -pipe
-D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2
-g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
-Wformat-nonliteral -Wformat-security -Wmissing-declarations
-Wmissing-prototypes -Wnested-externs -Wold-style-definition
-Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -
Wwrite-strings -Wno-address-of-packed-member
-Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
-DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -MF
drivers/lib
tmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o.d -o
drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -c
../drivers/event/dlb2/pf/dlb2_pf.c
In file included from ../drivers/event/dlb2/pf/dlb2_pf.c:35:
../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: error: use of
unknown builtin '__builtin_ia32_movntdq'
[-Wimplicit-function-declaration]
        __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Friday, October 30, 2020 9:22 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>; Thomas
> Monjalon <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD
> 
> On Fri, Oct 30, 2020 at 3:19 PM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> 
> >
> > Timothy McDaniel (23):
> >   event/dlb2: add documentation and meson build infrastructure
> >   event/dlb2: add dynamic logging
> >   event/dlb2: add private data structures and constants
> >   event/dlb2: add definitions shared with LKM or shared code
> >   event/dlb2: add inline functions
> >   event/dlb2: add eventdev probe
> 
> There is build error with clang  and static build here.
> Please send the next version with fix.
> 
> meson  -Dexamples=l3fwd --buildtype=debugoptimized --werror
> --default-library=static /export/dpdk-next-eventdev/devtools/..
> ./build-clang-static
> The Meson build system
> Version: 0.55.3
> Source dir: /export/dpdk-next-eventdev
> Build dir: /export/dpdk-next-eventdev/build-clang-static
> Build type: native build
> Program cat found: YES
> Using 'PKG_CONFIG_PATH' from environment with value: ''
> Using 'PKG_CONFIG_PATH' from environment with value: ''
> Project name: DPDK
> Project version: 20.11.0-rc1
> Using 'CC' from environment with value: 'ccache clang'
> Using 'CFLAGS' from environment with value: ''
> Using 'LDFLAGS' from environment with value: ''
> Using 'CPPFLAGS' from environment with value: ''
> Using 'CC' from environment with value: 'ccache clang'
> Using 'CFLAGS' from environment with value: ''
> Using 'LDFLAGS' from environment with value: ''
> Using 'CPPFLAGS' from environment with value: ''
> C compiler for the host machine: ccache clang (clang 10.0.1 "clang
> version 10.0.1 ")
> C linker for the host machine: clang ld.bfd 2.35.1
> Host machine cpu family: x86_64
> Host machine cpu: x86_64
> Program pkg-config found: YES
> Program gen-pmdinfo-cfile.sh found: YES
> Program list-dir-globs.py found: YES
> Program check-symbols.sh found: YES
> Program options-ibverbs-static.sh found: YES
> Program binutils-avx512-check.sh found: YES
> Program python3 found: YES (/usr/bin/python)
> 
> 
> 
> ccache clang -Idrivers/libtmp_rte_event_dlb2.a.p -Idrivers
> -I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
> -Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
> -I../config -Ilib/librte_eal/include -I../lib/librte_eal/i
> nclude -Ilib/librte_eal/linux/include
> -I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
> -I../lib/librte_eal/x86/include -Ilib/librte_eal/common
> -I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
> -Ilib/librte_kvargs
>  -I../lib/librte_kvargs -Ilib/librte_metrics -I../lib/librte_metrics
> -Ilib/librte_telemetry -I../lib/librte_telemetry -Ilib/librte_ring
> -I../lib/librte_ring -Ilib/librte_ethdev -I../lib/librte_ethdev
> -Ilib/librte_net -I../lib/librte_net -Il
> ib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
> -I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
> -Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_rcu
> -I../lib/librte_rcu -Ilib/librte_timer -I../lib/librte_timer -
> Ilib/librte_cryptodev -I../lib/librte_cryptodev -Ilib/librte_pci
> -I../lib/librte_pci -Idrivers/bus/pci -I../drivers/bus/pci
> -I../drivers/bus/pci/linux -Xclang -fcolor-diagnostics -pipe
> -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2
> -g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
> -Wformat-nonliteral -Wformat-security -Wmissing-declarations
> -Wmissing-prototypes -Wnested-externs -Wold-style-definition
> -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -
> Wwrite-strings -Wno-address-of-packed-member
> -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
> -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
> drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o -MF
> drivers/l
> ibtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o.d -o
> drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o -c
> ../drivers/event/dlb2/pf/dlb2_main.c
> In file included from ../drivers/event/dlb2/pf/dlb2_main.c:22:
> ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: error: use of
> unknown builtin '__builtin_ia32_movntdq'
> [-Wimplicit-function-declaration]
>         __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
>         ^
> ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: note: did you mean
> '__builtin_ia32_movntq'?
> /usr/lib/clang/10.0.1/include/xmmintrin.h:2122:3: note:
> '__builtin_ia32_movntq' declared here
>   __builtin_ia32_movntq(__p, __a);
> 
>   [2010/2491] Compiling C object
> drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o
> FAILED: drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o
> ccache clang -Idrivers/libtmp_rte_event_dlb2.a.p -Idrivers
> -I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
> -Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
> -I../config -Ilib/librte_eal/include -I../lib/librte_eal/i
> nclude -Ilib/librte_eal/linux/include
> -I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
> -I../lib/librte_eal/x86/include -Ilib/librte_eal/common
> -I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
> -Ilib/librte_kvargs
>  -I../lib/librte_kvargs -Ilib/librte_metrics -I../lib/librte_metrics
> -Ilib/librte_telemetry -I../lib/librte_telemetry -Ilib/librte_ring
> -I../lib/librte_ring -Ilib/librte_ethdev -I../lib/librte_ethdev
> -Ilib/librte_net -I../lib/librte_net -Il
> ib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
> -I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
> -Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_rcu
> -I../lib/librte_rcu -Ilib/librte_timer -I../lib/librte_timer -
> Ilib/librte_cryptodev -I../lib/librte_cryptodev -Ilib/librte_pci
> -I../lib/librte_pci -Idrivers/bus/pci -I../drivers/bus/pci
> -I../drivers/bus/pci/linux -Xclang -fcolor-diagnostics -pipe
> -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2
> -g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
> -Wformat-nonliteral -Wformat-security -Wmissing-declarations
> -Wmissing-prototypes -Wnested-externs -Wold-style-definition
> -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -
> Wwrite-strings -Wno-address-of-packed-member
> -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
> -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
> drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -MF
> drivers/lib
> tmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o.d -o
> drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -c
> ../drivers/event/dlb2/pf/dlb2_pf.c
> In file included from ../drivers/event/dlb2/pf/dlb2_pf.c:35:
> ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: error: use of
> unknown builtin '__builtin_ia32_movntdq'
> [-Wimplicit-function-declaration]
>         __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);

Not sure why this builds for me, but I do not see this error.
According to information online, '__builtin_ia32_movntdq' should be available if -msse2 is set.
According to the following snippet from config/x86/meson.build, it looks like msse4 is defined.
<snippet>
# we require SSE4.2 for DPDK
if cc.get_define('__SSE4_2__', args: machine_args) == ''
        message('SSE 4.2 not enabled by default, explicitly enabling')
        machine_args += '-msse4'
endif
<end snippet>

I realize this is from clang, not gcc, but why are they out of sync?
such that __builtin_ia32_movntdq is not available in clang, but is available in gcc.

Should I convert to _mm_stream_si128 (__m128i *__A, __m128i __B) ?

Any guidance on how to get past this would be greatly appreciated.

Thanks,
Tim

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[Jerin Jacob]

+ @Richardson, Bruce  @Ananyev, Konstantin

On Fri, Oct 30, 2020 at 8:55 PM McDaniel, Timothy
<timothy.mcdaniel@intel.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Jerin Jacob <jerinjacobk@gmail.com>
> > Sent: Friday, October 30, 2020 9:22 AM
> > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> > Gage <gage.eads@intel.com>; Van Haaren, Harry
> > <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>; Thomas
> > Monjalon <thomas@monjalon.net>
> > Subject: Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD
> >
> > On Fri, Oct 30, 2020 at 3:19 PM Timothy McDaniel
> > <timothy.mcdaniel@intel.com> wrote:
> >
> > >
> > > Timothy McDaniel (23):
> > >   event/dlb2: add documentation and meson build infrastructure
> > >   event/dlb2: add dynamic logging
> > >   event/dlb2: add private data structures and constants
> > >   event/dlb2: add definitions shared with LKM or shared code
> > >   event/dlb2: add inline functions
> > >   event/dlb2: add eventdev probe
> >
> > There is build error with clang  and static build here.
> > Please send the next version with fix.
> >
> > meson  -Dexamples=l3fwd --buildtype=debugoptimized --werror
> > --default-library=static /export/dpdk-next-eventdev/devtools/..
> > ./build-clang-static
> > The Meson build system
> > Version: 0.55.3
> > Source dir: /export/dpdk-next-eventdev
> > Build dir: /export/dpdk-next-eventdev/build-clang-static
> > Build type: native build
> > Program cat found: YES
> > Using 'PKG_CONFIG_PATH' from environment with value: ''
> > Using 'PKG_CONFIG_PATH' from environment with value: ''
> > Project name: DPDK
> > Project version: 20.11.0-rc1
> > Using 'CC' from environment with value: 'ccache clang'
> > Using 'CFLAGS' from environment with value: ''
> > Using 'LDFLAGS' from environment with value: ''
> > Using 'CPPFLAGS' from environment with value: ''
> > Using 'CC' from environment with value: 'ccache clang'
> > Using 'CFLAGS' from environment with value: ''
> > Using 'LDFLAGS' from environment with value: ''
> > Using 'CPPFLAGS' from environment with value: ''
> > C compiler for the host machine: ccache clang (clang 10.0.1 "clang
> > version 10.0.1 ")
> > C linker for the host machine: clang ld.bfd 2.35.1
> > Host machine cpu family: x86_64
> > Host machine cpu: x86_64
> > Program pkg-config found: YES
> > Program gen-pmdinfo-cfile.sh found: YES
> > Program list-dir-globs.py found: YES
> > Program check-symbols.sh found: YES
> > Program options-ibverbs-static.sh found: YES
> > Program binutils-avx512-check.sh found: YES
> > Program python3 found: YES (/usr/bin/python)
> >
> >
> >
> > ccache clang -Idrivers/libtmp_rte_event_dlb2.a.p -Idrivers
> > -I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
> > -Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
> > -I../config -Ilib/librte_eal/include -I../lib/librte_eal/i
> > nclude -Ilib/librte_eal/linux/include
> > -I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
> > -I../lib/librte_eal/x86/include -Ilib/librte_eal/common
> > -I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
> > -Ilib/librte_kvargs
> >  -I../lib/librte_kvargs -Ilib/librte_metrics -I../lib/librte_metrics
> > -Ilib/librte_telemetry -I../lib/librte_telemetry -Ilib/librte_ring
> > -I../lib/librte_ring -Ilib/librte_ethdev -I../lib/librte_ethdev
> > -Ilib/librte_net -I../lib/librte_net -Il
> > ib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
> > -I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
> > -Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_rcu
> > -I../lib/librte_rcu -Ilib/librte_timer -I../lib/librte_timer -
> > Ilib/librte_cryptodev -I../lib/librte_cryptodev -Ilib/librte_pci
> > -I../lib/librte_pci -Idrivers/bus/pci -I../drivers/bus/pci
> > -I../drivers/bus/pci/linux -Xclang -fcolor-diagnostics -pipe
> > -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2
> > -g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
> > -Wformat-nonliteral -Wformat-security -Wmissing-declarations
> > -Wmissing-prototypes -Wnested-externs -Wold-style-definition
> > -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -
> > Wwrite-strings -Wno-address-of-packed-member
> > -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
> > -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
> > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o -MF
> > drivers/l
> > ibtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o.d -o
> > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o -c
> > ../drivers/event/dlb2/pf/dlb2_main.c
> > In file included from ../drivers/event/dlb2/pf/dlb2_main.c:22:
> > ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: error: use of
> > unknown builtin '__builtin_ia32_movntdq'
> > [-Wimplicit-function-declaration]
> >         __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
> >         ^
> > ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: note: did you mean
> > '__builtin_ia32_movntq'?
> > /usr/lib/clang/10.0.1/include/xmmintrin.h:2122:3: note:
> > '__builtin_ia32_movntq' declared here
> >   __builtin_ia32_movntq(__p, __a);
> >
> >   [2010/2491] Compiling C object
> > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o
> > FAILED: drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o
> > ccache clang -Idrivers/libtmp_rte_event_dlb2.a.p -Idrivers
> > -I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
> > -Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
> > -I../config -Ilib/librte_eal/include -I../lib/librte_eal/i
> > nclude -Ilib/librte_eal/linux/include
> > -I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
> > -I../lib/librte_eal/x86/include -Ilib/librte_eal/common
> > -I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
> > -Ilib/librte_kvargs
> >  -I../lib/librte_kvargs -Ilib/librte_metrics -I../lib/librte_metrics
> > -Ilib/librte_telemetry -I../lib/librte_telemetry -Ilib/librte_ring
> > -I../lib/librte_ring -Ilib/librte_ethdev -I../lib/librte_ethdev
> > -Ilib/librte_net -I../lib/librte_net -Il
> > ib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
> > -I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
> > -Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_rcu
> > -I../lib/librte_rcu -Ilib/librte_timer -I../lib/librte_timer -
> > Ilib/librte_cryptodev -I../lib/librte_cryptodev -Ilib/librte_pci
> > -I../lib/librte_pci -Idrivers/bus/pci -I../drivers/bus/pci
> > -I../drivers/bus/pci/linux -Xclang -fcolor-diagnostics -pipe
> > -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2
> > -g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
> > -Wformat-nonliteral -Wformat-security -Wmissing-declarations
> > -Wmissing-prototypes -Wnested-externs -Wold-style-definition
> > -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -
> > Wwrite-strings -Wno-address-of-packed-member
> > -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
> > -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
> > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -MF
> > drivers/lib
> > tmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o.d -o
> > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -c
> > ../drivers/event/dlb2/pf/dlb2_pf.c
> > In file included from ../drivers/event/dlb2/pf/dlb2_pf.c:35:
> > ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: error: use of
> > unknown builtin '__builtin_ia32_movntdq'
> > [-Wimplicit-function-declaration]
> >         __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
>
> Not sure why this builds for me, but I do not see this error.

May be you can try with
[for-main][dpdk-next-eventdev] $ clang -v
clang version 10.0.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin


> According to information online, '__builtin_ia32_movntdq' should be available if -msse2 is set.
> According to the following snippet from config/x86/meson.build, it looks like msse4 is defined.
> <snippet>
> # we require SSE4.2 for DPDK
> if cc.get_define('__SSE4_2__', args: machine_args) == ''
>         message('SSE 4.2 not enabled by default, explicitly enabling')
>         machine_args += '-msse4'
> endif
> <end snippet>
>
> I realize this is from clang, not gcc, but why are they out of sync?
> such that __builtin_ia32_movntdq is not available in clang, but is available in gcc.
>
> Should I convert to _mm_stream_si128 (__m128i *__A, __m128i __B) ?
>
> Any guidance on how to get past this would be greatly appreciated.

Added @Richardson, Bruce  @Ananyev, Konstantin

I do see the same problem in DLB driver as well.


>
> Thanks,
> Tim
>
>
>

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[Van Haaren, Harry]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Friday, October 30, 2020 3:31 PM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> Gage <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> Jerin Jacob <jerinj@marvell.com>; Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD
> 
> + @Richardson, Bruce  @Ananyev, Konstantin
> 
> On Fri, Oct 30, 2020 at 8:55 PM McDaniel, Timothy
> <timothy.mcdaniel@intel.com> wrote:
> >

<snip backlog and compiler error output>

> > > -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
> > > -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
> > > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -MF
> > > drivers/lib
> > > tmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o.d -o
> > > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -c
> > > ../drivers/event/dlb2/pf/dlb2_pf.c
> > > In file included from ../drivers/event/dlb2/pf/dlb2_pf.c:35:
> > > ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: error: use of
> > > unknown builtin '__builtin_ia32_movntdq'
> > > [-Wimplicit-function-declaration]
> > >         __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
> >
> > Not sure why this builds for me, but I do not see this error.
> 
> May be you can try with
> [for-main][dpdk-next-eventdev] $ clang -v
> clang version 10.0.1
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> InstalledDir: /usr/bin
> 
> 
> > According to information online, '__builtin_ia32_movntdq' should be available if -
> msse2 is set.
> > According to the following snippet from config/x86/meson.build, it looks like
> msse4 is defined.
> > <snippet>
> > # we require SSE4.2 for DPDK
> > if cc.get_define('__SSE4_2__', args: machine_args) == ''
> >         message('SSE 4.2 not enabled by default, explicitly enabling')
> >         machine_args += '-msse4'
> > endif
> > <end snippet>
> >
> > I realize this is from clang, not gcc, but why are they out of sync?
> > such that __builtin_ia32_movntdq is not available in clang, but is available in gcc.
> >
> > Should I convert to _mm_stream_si128 (__m128i *__A, __m128i __B) ?
> >
> > Any guidance on how to get past this would be greatly appreciated.

Confirm that __builtin_ia32_* works on GCC and not on Clang, reproducing error above.
Using the intrinsic _mm_stream or _mm_store versions work with both Clang and GCC, and will fix this issue.

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[McDaniel, Timothy]

> -----Original Message-----
> From: Van Haaren, Harry <harry.van.haaren@intel.com>
> Sent: Friday, October 30, 2020 11:09 AM
> To: Jerin Jacob <jerinjacobk@gmail.com>; McDaniel, Timothy
> <timothy.mcdaniel@intel.com>
> Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> Gage <gage.eads@intel.com>; Jerin Jacob <jerinj@marvell.com>; Thomas
> Monjalon <thomas@monjalon.net>
> Subject: RE: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD
> 
> > -----Original Message-----
> > From: Jerin Jacob <jerinjacobk@gmail.com>
> > Sent: Friday, October 30, 2020 3:31 PM
> > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>;
> Eads,
> > Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>;
> > Jerin Jacob <jerinj@marvell.com>; Thomas Monjalon
> <thomas@monjalon.net>
> > Subject: Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD
> >
> > + @Richardson, Bruce  @Ananyev, Konstantin
> >
> > On Fri, Oct 30, 2020 at 8:55 PM McDaniel, Timothy
> > <timothy.mcdaniel@intel.com> wrote:
> > >
> 
> <snip backlog and compiler error output>
> 
> > > > -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
> > > > -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
> > > > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -MF
> > > > drivers/lib
> > > > tmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o.d -o
> > > > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -c
> > > > ../drivers/event/dlb2/pf/dlb2_pf.c
> > > > In file included from ../drivers/event/dlb2/pf/dlb2_pf.c:35:
> > > > ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: error: use of
> > > > unknown builtin '__builtin_ia32_movntdq'
> > > > [-Wimplicit-function-declaration]
> > > >         __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
> > >
> > > Not sure why this builds for me, but I do not see this error.
> >
> > May be you can try with
> > [for-main][dpdk-next-eventdev] $ clang -v
> > clang version 10.0.1
> > Target: x86_64-pc-linux-gnu
> > Thread model: posix
> > InstalledDir: /usr/bin
> >
> >
> > > According to information online, '__builtin_ia32_movntdq' should be
> available if -
> > msse2 is set.
> > > According to the following snippet from config/x86/meson.build, it looks like
> > msse4 is defined.
> > > <snippet>
> > > # we require SSE4.2 for DPDK
> > > if cc.get_define('__SSE4_2__', args: machine_args) == ''
> > >         message('SSE 4.2 not enabled by default, explicitly enabling')
> > >         machine_args += '-msse4'
> > > endif
> > > <end snippet>
> > >
> > > I realize this is from clang, not gcc, but why are they out of sync?
> > > such that __builtin_ia32_movntdq is not available in clang, but is available in
> gcc.
> > >
> > > Should I convert to _mm_stream_si128 (__m128i *__A, __m128i __B) ?
> > >
> > > Any guidance on how to get past this would be greatly appreciated.
> 
> Confirm that __builtin_ia32_* works on GCC and not on Clang, reproducing
> error above.
> Using the intrinsic _mm_stream or _mm_store versions work with both Clang
> and GCC, and will fix this issue.

Thanks Harry.
I will convert to _mm_stream_si128.

Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD

[David Marchand]

On Fri, Oct 30, 2020 at 4:25 PM McDaniel, Timothy
<timothy.mcdaniel@intel.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Jerin Jacob <jerinjacobk@gmail.com>
> > Sent: Friday, October 30, 2020 9:22 AM
> > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> > Gage <gage.eads@intel.com>; Van Haaren, Harry
> > <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>; Thomas
> > Monjalon <thomas@monjalon.net>
> > Subject: Re: [dpdk-dev] [PATCH v5 00/23] Add DLB2 PMD
> >
> > On Fri, Oct 30, 2020 at 3:19 PM Timothy McDaniel
> > <timothy.mcdaniel@intel.com> wrote:
> >
> > >
> > > Timothy McDaniel (23):
> > >   event/dlb2: add documentation and meson build infrastructure
> > >   event/dlb2: add dynamic logging
> > >   event/dlb2: add private data structures and constants
> > >   event/dlb2: add definitions shared with LKM or shared code
> > >   event/dlb2: add inline functions
> > >   event/dlb2: add eventdev probe
> >
> > There is build error with clang  and static build here.
> > Please send the next version with fix.
> >
> > meson  -Dexamples=l3fwd --buildtype=debugoptimized --werror
> > --default-library=static /export/dpdk-next-eventdev/devtools/..
> > ./build-clang-static
> > The Meson build system
> > Version: 0.55.3
> > Source dir: /export/dpdk-next-eventdev
> > Build dir: /export/dpdk-next-eventdev/build-clang-static
> > Build type: native build
> > Program cat found: YES
> > Using 'PKG_CONFIG_PATH' from environment with value: ''
> > Using 'PKG_CONFIG_PATH' from environment with value: ''
> > Project name: DPDK
> > Project version: 20.11.0-rc1
> > Using 'CC' from environment with value: 'ccache clang'
> > Using 'CFLAGS' from environment with value: ''
> > Using 'LDFLAGS' from environment with value: ''
> > Using 'CPPFLAGS' from environment with value: ''
> > Using 'CC' from environment with value: 'ccache clang'
> > Using 'CFLAGS' from environment with value: ''
> > Using 'LDFLAGS' from environment with value: ''
> > Using 'CPPFLAGS' from environment with value: ''
> > C compiler for the host machine: ccache clang (clang 10.0.1 "clang
> > version 10.0.1 ")
> > C linker for the host machine: clang ld.bfd 2.35.1
> > Host machine cpu family: x86_64
> > Host machine cpu: x86_64
> > Program pkg-config found: YES
> > Program gen-pmdinfo-cfile.sh found: YES
> > Program list-dir-globs.py found: YES
> > Program check-symbols.sh found: YES
> > Program options-ibverbs-static.sh found: YES
> > Program binutils-avx512-check.sh found: YES
> > Program python3 found: YES (/usr/bin/python)
> >
> >
> >
> > ccache clang -Idrivers/libtmp_rte_event_dlb2.a.p -Idrivers
> > -I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
> > -Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
> > -I../config -Ilib/librte_eal/include -I../lib/librte_eal/i
> > nclude -Ilib/librte_eal/linux/include
> > -I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
> > -I../lib/librte_eal/x86/include -Ilib/librte_eal/common
> > -I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
> > -Ilib/librte_kvargs
> >  -I../lib/librte_kvargs -Ilib/librte_metrics -I../lib/librte_metrics
> > -Ilib/librte_telemetry -I../lib/librte_telemetry -Ilib/librte_ring
> > -I../lib/librte_ring -Ilib/librte_ethdev -I../lib/librte_ethdev
> > -Ilib/librte_net -I../lib/librte_net -Il
> > ib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
> > -I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
> > -Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_rcu
> > -I../lib/librte_rcu -Ilib/librte_timer -I../lib/librte_timer -
> > Ilib/librte_cryptodev -I../lib/librte_cryptodev -Ilib/librte_pci
> > -I../lib/librte_pci -Idrivers/bus/pci -I../drivers/bus/pci
> > -I../drivers/bus/pci/linux -Xclang -fcolor-diagnostics -pipe
> > -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2
> > -g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
> > -Wformat-nonliteral -Wformat-security -Wmissing-declarations
> > -Wmissing-prototypes -Wnested-externs -Wold-style-definition
> > -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -
> > Wwrite-strings -Wno-address-of-packed-member
> > -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
> > -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
> > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o -MF
> > drivers/l
> > ibtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o.d -o
> > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_main.c.o -c
> > ../drivers/event/dlb2/pf/dlb2_main.c
> > In file included from ../drivers/event/dlb2/pf/dlb2_main.c:22:
> > ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: error: use of
> > unknown builtin '__builtin_ia32_movntdq'
> > [-Wimplicit-function-declaration]
> >         __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
> >         ^
> > ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: note: did you mean
> > '__builtin_ia32_movntq'?
> > /usr/lib/clang/10.0.1/include/xmmintrin.h:2122:3: note:
> > '__builtin_ia32_movntq' declared here
> >   __builtin_ia32_movntq(__p, __a);
> >
> >   [2010/2491] Compiling C object
> > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o
> > FAILED: drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o
> > ccache clang -Idrivers/libtmp_rte_event_dlb2.a.p -Idrivers
> > -I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
> > -Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
> > -I../config -Ilib/librte_eal/include -I../lib/librte_eal/i
> > nclude -Ilib/librte_eal/linux/include
> > -I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
> > -I../lib/librte_eal/x86/include -Ilib/librte_eal/common
> > -I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
> > -Ilib/librte_kvargs
> >  -I../lib/librte_kvargs -Ilib/librte_metrics -I../lib/librte_metrics
> > -Ilib/librte_telemetry -I../lib/librte_telemetry -Ilib/librte_ring
> > -I../lib/librte_ring -Ilib/librte_ethdev -I../lib/librte_ethdev
> > -Ilib/librte_net -I../lib/librte_net -Il
> > ib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
> > -I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
> > -Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_rcu
> > -I../lib/librte_rcu -Ilib/librte_timer -I../lib/librte_timer -
> > Ilib/librte_cryptodev -I../lib/librte_cryptodev -Ilib/librte_pci
> > -I../lib/librte_pci -Idrivers/bus/pci -I../drivers/bus/pci
> > -I../drivers/bus/pci/linux -Xclang -fcolor-diagnostics -pipe
> > -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2
> > -g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
> > -Wformat-nonliteral -Wformat-security -Wmissing-declarations
> > -Wmissing-prototypes -Wnested-externs -Wold-style-definition
> > -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -
> > Wwrite-strings -Wno-address-of-packed-member
> > -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
> > -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
> > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -MF
> > drivers/lib
> > tmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o.d -o
> > drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_pf_dlb2_pf.c.o -c
> > ../drivers/event/dlb2/pf/dlb2_pf.c
> > In file included from ../drivers/event/dlb2/pf/dlb2_pf.c:35:
> > ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:41:2: error: use of
> > unknown builtin '__builtin_ia32_movntdq'
> > [-Wimplicit-function-declaration]
> >         __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
>
> Not sure why this builds for me, but I do not see this error.

Reproduced the same error on fc31.
$ clang --version
clang version 9.0.1 (Fedora 9.0.1-2.fc31)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

$ ./devtools/test-meson-builds.sh
...

FAILED: drivers/a715181@@tmp_rte_event_dlb2@sta/event_dlb2_dlb2.c.o
ccache clang -Idrivers/a715181@@tmp_rte_event_dlb2@sta -Idrivers
-I../../dpdk/drivers -Idrivers/event/dlb2
-I../../dpdk/drivers/event/dlb2 -Ilib/librte_eventdev
-I../../dpdk/lib/librte_eventdev -I. -I../../dpdk/ -Iconfig
-I../../dpdk/config -Ilib/librte_eal/include
-I../../dpdk/lib/librte_eal/include -Ilib/librte_eal/linux/include
-I../../dpdk/lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
-I../../dpdk/lib/librte_eal/x86/include -Ilib/librte_eal/common
-I../../dpdk/lib/librte_eal/common -Ilib/librte_eal
-I../../dpdk/lib/librte_eal -Ilib/librte_kvargs
-I../../dpdk/lib/librte_kvargs
-Ilib/librte_telemetry/../librte_metrics
-I../../dpdk/lib/librte_telemetry/../librte_metrics
-Ilib/librte_telemetry -I../../dpdk/lib/librte_telemetry
-Ilib/librte_ring -I../../dpdk/lib/librte_ring -Ilib/librte_ethdev
-I../../dpdk/lib/librte_ethdev -Ilib/librte_net
-I../../dpdk/lib/librte_net -Ilib/librte_mbuf
-I../../dpdk/lib/librte_mbuf -Ilib/librte_mempool
-I../../dpdk/lib/librte_mempool -Ilib/librte_meter
-I../../dpdk/lib/librte_meter -Ilib/librte_hash
-I../../dpdk/lib/librte_hash -Ilib/librte_rcu
-I../../dpdk/lib/librte_rcu -Ilib/librte_timer
-I../../dpdk/lib/librte_timer -Ilib/librte_cryptodev
-I../../dpdk/lib/librte_cryptodev -Ilib/librte_pci
-I../../dpdk/lib/librte_pci -Idrivers/bus/pci
-I../../dpdk/drivers/bus/pci -I../../dpdk/drivers/bus/pci/linux
-I/home/dmarchan/intel-ipsec-mb/install/include -Xclang
-fcolor-diagnostics -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
-Werror -O2 -g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
-Wformat-nonliteral -Wformat-security -Wmissing-declarations
-Wmissing-prototypes -Wnested-externs -Wold-style-definition
-Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef
-Wwrite-strings -Wno-address-of-packed-member
-Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
-DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
'drivers/a715181@@tmp_rte_event_dlb2@sta/event_dlb2_dlb2.c.o' -MF
'drivers/a715181@@tmp_rte_event_dlb2@sta/event_dlb2_dlb2.c.o.d' -o
'drivers/a715181@@tmp_rte_event_dlb2@sta/event_dlb2_dlb2.c.o' -c
../../dpdk/drivers/event/dlb2/dlb2.c
In file included from ../../dpdk/drivers/event/dlb2/dlb2.c:35:
../../dpdk/drivers/event/dlb2/dlb2_inline_fns.h:41:2: error: use of
unknown builtin '__builtin_ia32_movntdq'
[-Wimplicit-function-declaration]
        __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
        ^
../../dpdk/drivers/event/dlb2/dlb2_inline_fns.h:41:2: note: did you
mean '__builtin_ia32_movntq'?
/usr/lib64/clang/9.0.1/include/xmmintrin.h:2122:3: note:
'__builtin_ia32_movntq' declared here
  __builtin_ia32_movntq(__p, __a);
  ^
1 error generated.


FAILED: drivers/a715181@@tmp_rte_event_dlb2@sta/event_dlb2_dlb2_selftest.c.o
ccache clang -Idrivers/a715181@@tmp_rte_event_dlb2@sta -Idrivers
-I../../dpdk/drivers -Idrivers/event/dlb2
-I../../dpdk/drivers/event/dlb2 -Ilib/librte_eventdev
-I../../dpdk/lib/librte_eventdev -I. -I../../dpdk/ -Iconfig
-I../../dpdk/config -Ilib/librte_eal/include
-I../../dpdk/lib/librte_eal/include -Ilib/librte_eal/linux/include
-I../../dpdk/lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
-I../../dpdk/lib/librte_eal/x86/include -Ilib/librte_eal/common
-I../../dpdk/lib/librte_eal/common -Ilib/librte_eal
-I../../dpdk/lib/librte_eal -Ilib/librte_kvargs
-I../../dpdk/lib/librte_kvargs
-Ilib/librte_telemetry/../librte_metrics
-I../../dpdk/lib/librte_telemetry/../librte_metrics
-Ilib/librte_telemetry -I../../dpdk/lib/librte_telemetry
-Ilib/librte_ring -I../../dpdk/lib/librte_ring -Ilib/librte_ethdev
-I../../dpdk/lib/librte_ethdev -Ilib/librte_net
-I../../dpdk/lib/librte_net -Ilib/librte_mbuf
-I../../dpdk/lib/librte_mbuf -Ilib/librte_mempool
-I../../dpdk/lib/librte_mempool -Ilib/librte_meter
-I../../dpdk/lib/librte_meter -Ilib/librte_hash
-I../../dpdk/lib/librte_hash -Ilib/librte_rcu
-I../../dpdk/lib/librte_rcu -Ilib/librte_timer
-I../../dpdk/lib/librte_timer -Ilib/librte_cryptodev
-I../../dpdk/lib/librte_cryptodev -Ilib/librte_pci
-I../../dpdk/lib/librte_pci -Idrivers/bus/pci
-I../../dpdk/drivers/bus/pci -I../../dpdk/drivers/bus/pci/linux
-I/home/dmarchan/intel-ipsec-mb/install/include -Xclang
-fcolor-diagnostics -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
-Werror -O2 -g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
-Wformat-nonliteral -Wformat-security -Wmissing-declarations
-Wmissing-prototypes -Wnested-externs -Wold-style-definition
-Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef
-Wwrite-strings -Wno-address-of-packed-member
-Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
-DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
'drivers/a715181@@tmp_rte_event_dlb2@sta/event_dlb2_dlb2_selftest.c.o'
-MF 'drivers/a715181@@tmp_rte_event_dlb2@sta/event_dlb2_dlb2_selftest.c.o.d'
-o 'drivers/a715181@@tmp_rte_event_dlb2@sta/event_dlb2_dlb2_selftest.c.o'
-c ../../dpdk/drivers/event/dlb2/dlb2_selftest.c
../../dpdk/drivers/event/dlb2/dlb2_selftest.c:139:1: error: unused
function 'create_ordered_qids' [-Werror,-Wunused-function]
create_ordered_qids(struct test *t, int num_qids)
^
../../dpdk/drivers/event/dlb2/dlb2_selftest.c:145:1: error: unused
function 'create_unordered_qids' [-Werror,-Wunused-function]
create_unordered_qids(struct test *t, int num_qids)
^
../../dpdk/drivers/event/dlb2/dlb2_selftest.c:151:1: error: unused
function 'create_directed_qids' [-Werror,-Wunused-function]
create_directed_qids(struct test *t, int num_qids, const uint8_t ports[])
^
3 errors generated.




I also see a different issue with RHEL 7 gcc:

Found ninja-1.7.2 at /usr/bin/ninja-build
[1908/2413] Compiling C object
'drivers/drivers@@tmp_rte_event_dlb2@sta/event_dlb2_dlb2.c.o'.
../drivers/event/dlb2/dlb2.c: In function ‘dlb2_hw_create_ldb_port’:
../drivers/event/dlb2/dlb2.c:1091:9: warning: missing braces around
initializer [-Wmissing-braces]
  struct dlb2_create_ldb_port_args cfg = {0};
         ^
../drivers/event/dlb2/dlb2.c:1091:9: warning: (near initialization for
‘cfg.response’) [-Wmissing-braces]
../drivers/event/dlb2/dlb2.c: In function ‘dlb2_hw_create_dir_port’:
../drivers/event/dlb2/dlb2.c:1263:9: warning: missing braces around
initializer [-Wmissing-braces]
  struct dlb2_create_dir_port_args cfg = {0};
         ^
../drivers/event/dlb2/dlb2.c:1263:9: warning: (near initialization for
‘cfg.response’) [-Wmissing-braces]


-- 
David Marchand

[dpdk-dev] [PATCH v6 00/23] Add DLB2 PMD

[Timothy McDaniel]

The following patch series adds support for a new eventdev PMD. The DLB2
PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
hardware.
The DLB2 is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB2 hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB2 ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a
directed queue. Additionally, even though LDB ports may link multiple
queues,
the number of queues that may be linked is limited by hardware.

While reviewing the code, please be aware that this PMD has full
control over the DLB2 hardware. Intel will be extending the DLB2 PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.
Note that the DLB2 hardware is a successor of the DLB hardware, and
as such is structured similarly, both in terms of code layout and
implementation.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Changes since V5:
================
- convert to use rte_power_monitor patches
- replace __builtin_ia32_movntdq() with _mm_stream_si128()
- remove unused functions in dlb_selftest.c
- workaround for RHEL 7 gcc brace bug

Changes since V4:
================
- moved introduction of dlb in relnotes_20_11 to first patch in series
- fixed underlines in dlb.rst that were too short
- note that the code still uses its private byte-encoded versions of
  umonitor/umwait, rather than the new functions in the power
  patch that are built on top of those intrinsics. This is intentional.

Changes since V3:
================
- updated MAINTAINERS file to alphabetically insert DLB
- don't create RTE_ symbols in pmd
- converted to use version.map scheme
- converted to use .._master_lcore instead of .._main_lcore
- this patch set is based on dpdk-next-eventdev

Changes since V2:
================
- fixed meson conditional build. Moved test into driver’s meson.build
  file instead of event/meson.build
- documentation is populated as associated code is introduced
- add log_register in add dynamic logging patch
- rename RTE_xxx symbol(s) as DLB2_xxx
- replaced function ptr enqueue_four with direct call to movdir64b
- remove unused port_pages
- broke up probe patch into 3 smaller patches for easier review
- changed param order of movdir64b/movntdq to match intrinsics
- added self to MAINTAINERS files
- squashed announcement of availability into last patch in series
- correct spelling errors and delete repeated words
- DPDK_21.0 -> DPDK 21 in map file
- add experimental banner to public structs and APIs

Changes since V1:
=================
- implement changes requested in code reviews by Gage Eads and Mike
  Chen
- fix a memzone leak
- convert to use eal rte-cpuflags patch from Liang Ma

Known Issues:
- the documentation contained in dlb2.rst is not complete, and
  should be updated to include command line parameters (class of service,
        etc ...).

Depends-on: patch-82202 ("eventdev: increase MAX QUEUES PER DEV to 255")
Depends-on: patch-79539 ("eal: add new x86 cpuid support for WAITPKG")

Timothy McDaniel (23):
  event/dlb2: add documentation and meson build infrastructure
  event/dlb2: add dynamic logging
  event/dlb2: add private data structures and constants
  event/dlb2: add definitions shared with LKM or shared code
  event/dlb2: add inline functions
  event/dlb2: add eventdev probe
  event/dlb2: add flexible interface
  event/dlb2: add probe-time hardware init
  event/dlb2: add xstats
  event/dlb2: add infos get and configure
  event/dlb2: add queue and port default conf
  event/dlb2: add queue setup
  event/dlb2: add port setup
  event/dlb2: add port link
  event/dlb2: add port unlink and port unlinks in progress
  event/dlb2: add eventdev start
  event/dlb2: add enqueue and its burst variants
  event/dlb2: add dequeue and its burst variants
  event/dlb2: add eventdev stop and close
  event/dlb2: add PMD's token pop public interface
  event/dlb2: add PMD self-tests
  event/dlb2: add queue and port release
  event/dlb2: add timeout ticks entry point

 MAINTAINERS                                    |    6 +-
 app/test/test_eventdev.c                       |    7 +
 config/rte_config.h                            |    7 +
 doc/api/doxy-api-index.md                      |    1 +
 doc/guides/eventdevs/dlb2.rst                  |  365 ++
 doc/guides/eventdevs/index.rst                 |    1 +
 doc/guides/rel_notes/release_20_11.rst         |    5 +
 drivers/event/dlb2/dlb2.c                      | 3959 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c                |   74 +
 drivers/event/dlb2/dlb2_iface.h                |   74 +
 drivers/event/dlb2/dlb2_inline_fns.h           |   61 +
 drivers/event/dlb2/dlb2_log.h                  |   25 +
 drivers/event/dlb2/dlb2_priv.h                 |  578 +++
 drivers/event/dlb2/dlb2_selftest.c             | 1524 ++++++
 drivers/event/dlb2/dlb2_user.h                 |  679 +++
 drivers/event/dlb2/dlb2_xstats.c               | 1240 +++++
 drivers/event/dlb2/meson.build                 |   22 +
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 +++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 +
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c     | 6027 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  673 +++
 drivers/event/dlb2/pf/dlb2_main.h              |   97 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  728 +++
 drivers/event/dlb2/rte_pmd_dlb2.c              |   39 +
 drivers/event/dlb2/rte_pmd_dlb2.h              |   72 +
 drivers/event/dlb2/version.map                 |    9 +
 drivers/event/meson.build                      |    3 +-
 33 files changed, 22509 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
 create mode 100644 drivers/event/dlb2/dlb2_log.h
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c
 create mode 100644 drivers/event/dlb2/dlb2_user.h
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
 create mode 100644 drivers/event/dlb2/version.map

-- 
2.6.4

[dpdk-dev] [PATCH v6 01/23] event/dlb2: add documentation and meson build infrastructure

[Timothy McDaniel]

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux 64 bit X86 platforms at this time.

Adds announcement of availabililty for the new driver
for Intel Dynamic Load Balancer 2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 MAINTAINERS                            |  6 ++++-
 config/rte_config.h                    |  7 ++++++
 doc/guides/eventdevs/dlb2.rst          | 40 ++++++++++++++++++++++++++++++++++
 doc/guides/eventdevs/index.rst         |  1 +
 doc/guides/rel_notes/release_20_11.rst |  5 +++++
 drivers/event/dlb2/meson.build         | 13 +++++++++++
 drivers/event/dlb2/version.map         |  3 +++
 drivers/event/meson.build              |  3 ++-
 8 files changed, 76 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index a3d1927..727451b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1162,6 +1162,11 @@ Cavium OCTEON TX timvf
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 F: drivers/event/octeontx/timvf_*
 
+Intel DLB2
+M: Timothy McDaniel <timothy.mcdaniel@intel.com>
+F: drivers/event/dlb2/
+F: doc/guides/eventdevs/dlb2.rst
+
 Marvell OCTEON TX2
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 M: Jerin Jacob <jerinj@marvell.com>
@@ -1198,7 +1203,6 @@ M: Peter Mccarthy <peter.mccarthy@intel.com>
 F: drivers/event/opdl/
 F: doc/guides/eventdevs/opdl.rst
 
-
 Rawdev Drivers
 --------------
 
diff --git a/config/rte_config.h b/config/rte_config.h
index b78c6aa..64aa333 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -135,4 +135,11 @@
 /* QEDE PMD defines */
 #define RTE_LIBRTE_QEDE_FW ""
 
+/* DLB2 defines */
+#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
+#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
+#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
+#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
+
 #endif /* _RTE_CONFIG_H_ */
diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
new file mode 100644
index 0000000..e3091c9
--- /dev/null
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -0,0 +1,40 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Driver for the Intel® Dynamic Load Balancer (DLB2)
+==================================================
+
+The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
+
+Prerequisites
+-------------
+
+Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
+the basic DPDK environment.
+
+Configuration
+-------------
+
+The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
+device access. To use this operation mode, the PCIe PF device must be bound
+to a DPDK-compatible VFIO driver, such as vfio-pci.
+
+Eventdev API Notes
+------------------
+
+The DLB2 provides the functions of a DPDK event device; specifically, it
+supports atomic, ordered, and parallel scheduling events from queues to ports.
+However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
+features are abstracted by the PMD such as directed ports.
+
+In general the dlb PMD is designed for ease-of-use and does not require a
+detailed understanding of the hardware, but these details are important when
+writing high-performance code. This section describes the places where the
+eventdev API and DLB2 misalign.
+
+Flow ID
+~~~~~~~
+
+The flow ID field is preserved in the event when it is scheduled in the
+DLB2.
+
diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
index bb66a5e..738788d 100644
--- a/doc/guides/eventdevs/index.rst
+++ b/doc/guides/eventdevs/index.rst
@@ -11,6 +11,7 @@ application through the eventdev API.
     :maxdepth: 2
     :numbered:
 
+    dlb2
     dpaa
     dpaa2
     dsw
diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
index d8ac359..dbce3b1 100644
--- a/doc/guides/rel_notes/release_20_11.rst
+++ b/doc/guides/rel_notes/release_20_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
+
+  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
+  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
+
 * **Added write combining store APIs.**
 
   Added ``rte_write32_wc`` and ``rte_write32_wc_relaxed`` APIs
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
new file mode 100644
index 0000000..5324043
--- /dev/null
+++ b/drivers/event/dlb2/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2020 Intel Corporation
+
+if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
+        build = false
+        reason = 'only supported on ARCH_X86_64 Linux'
+        subdir_done()
+endif
+
+sources = files(
+)
+
+deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
new file mode 100644
index 0000000..4a76d1d
--- /dev/null
+++ b/drivers/event/dlb2/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index a7dac99..c433c10 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -5,7 +5,8 @@ if is_windows
 	subdir_done()
 endif
 
-drivers = ['dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw', 'dsw']
+drivers = ['dlb2', 'dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw',
+	   'dsw']
 if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
-- 
2.6.4

[dpdk-dev] [PATCH v6 02/23] event/dlb2: add dynamic logging

[Timothy McDaniel]

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c      |  7 +++++++
 drivers/event/dlb2/dlb2_log.h  | 25 +++++++++++++++++++++++++
 drivers/event/dlb2/meson.build |  3 +--
 3 files changed, 33 insertions(+), 2 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_log.h

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
new file mode 100644
index 0000000..2082dd6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2.c
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <rte_log.h>
+
+RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
new file mode 100644
index 0000000..dc1481e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_log.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_EVDEV_LOG_H_
+#define _DLB2_EVDEV_LOG_H_
+
+extern int eventdev_dlb2_log_level;
+
+/* Dynamic logging */
+#define DLB2_LOG_IMPL(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
+		__func__, ##args)
+
+#define DLB2_LOG_INFO(fmt, args...) \
+	DLB2_LOG_IMPL(INFO, fmt, ## args)
+
+#define DLB2_LOG_ERR(fmt, args...) \
+	DLB2_LOG_IMPL(ERR, fmt, ## args)
+
+/* remove debug logs at compile time unless actually debugging */
+#define DLB2_LOG_DBG(fmt, args...) \
+	RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
+
+#endif /* _DLB2_EVDEV_LOG_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 5324043..435c1ee 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,7 +7,6 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files(
-)
+sources = files('dlb2.c')
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v6 03/23] event/dlb2: add private data structures and constants

[Timothy McDaniel]

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_priv.h | 575 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 575 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_priv.h

diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
new file mode 100644
index 0000000..61567a6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -0,0 +1,575 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_PRIV_H_
+#define _DLB2_PRIV_H_
+
+#include <emmintrin.h>
+#include <stdbool.h>
+
+#include <rte_eventdev.h>
+#include <rte_config.h>
+#include "dlb2_user.h"
+#include "dlb2_log.h"
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
+#else
+#define DLB2_INC_STAT(_stat, _incr_val)
+#endif
+
+#define EVDEV_DLB2_NAME_PMD dlb2_event
+
+/*  command line arg strings */
+#define NUMA_NODE_ARG "numa_node"
+#define DLB2_MAX_NUM_EVENTS "max_num_events"
+#define DLB2_NUM_DIR_CREDITS "num_dir_credits"
+#define DEV_ID_ARG "dev_id"
+#define DLB2_DEFER_SCHED_ARG "defer_sched"
+#define DLB2_QID_DEPTH_THRESH_ARG "qid_depth_thresh"
+#define DLB2_COS_ARG "cos"
+
+/* Begin HW related defines and structs */
+
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_VFS 16
+#define DLB2_MAX_NUM_LDB_QUEUES 32
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS 64
+#define DLB2_MAX_NUM_DIR_QUEUES 64
+#define DLB2_MAX_NUM_FLOWS (64 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDITS (8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS (2 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_DIR_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQOS_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_MAX_DEVICE_PATH 32
+#define DLB2_MIN_DEQUEUE_TIMEOUT_NS 1
+/* Note: "- 1" here to support the timeout range check in eventdev_autotest */
+#define DLB2_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1)
+#define DLB2_SW_CREDIT_BATCH_SZ 32
+#define DLB2_NUM_SN_GROUPS 2
+#define DLB2_MAX_LDB_SN_ALLOC 1024
+#define DLB2_MAX_QUEUE_DEPTH_THRESHOLD 8191
+
+/* 2048 total hist list entries and 64 total ldb ports, which
+ * makes for 2048/64 == 32 hist list entries per port. However, CQ
+ * depth must be a power of 2 and must also be >= HIST LIST entries.
+ * As a result we just limit the maximum dequeue depth to 32.
+ */
+#define DLB2_MIN_CQ_DEPTH 1
+#define DLB2_MAX_CQ_DEPTH 32
+#define DLB2_MIN_HARDWARE_CQ_DEPTH 8
+#define DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \
+	DLB2_MAX_CQ_DEPTH
+
+/*
+ * Static per queue/port provisioning values
+ */
+#define DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 64
+
+#define CQ_BASE(is_dir) ((is_dir) ? DLB2_DIR_CQ_BASE : DLB2_LDB_CQ_BASE)
+#define CQ_SIZE(is_dir) ((is_dir) ? DLB2_DIR_CQ_MAX_SIZE : \
+				    DLB2_LDB_CQ_MAX_SIZE)
+#define PP_BASE(is_dir) ((is_dir) ? DLB2_DIR_PP_BASE : DLB2_LDB_PP_BASE)
+
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+
+#define DLB2_NUM_QES_PER_CACHE_LINE 4
+
+#define DLB2_MAX_ENQUEUE_DEPTH 64
+#define DLB2_MIN_ENQUEUE_DEPTH 4
+
+#define DLB2_NAME_SIZE 64
+
+#define DLB2_1K 1024
+#define DLB2_2K (2 * DLB2_1K)
+#define DLB2_4K (4 * DLB2_1K)
+#define DLB2_16K (16 * DLB2_1K)
+#define DLB2_32K (32 * DLB2_1K)
+#define DLB2_1MB (DLB2_1K * DLB2_1K)
+#define DLB2_16MB (16 * DLB2_1MB)
+
+/* Use the upper 3 bits of the event priority to select the DLB2 priority */
+#define EV_TO_DLB2_PRIO(x) ((x) >> 5)
+#define DLB2_TO_EV_PRIO(x) ((x) << 5)
+
+enum dlb2_hw_port_types {
+	DLB2_LDB_PORT,
+	DLB2_DIR_PORT,
+	DLB2_NUM_PORT_TYPES /* Must be last */
+};
+
+enum dlb2_hw_queue_types {
+	DLB2_LDB_QUEUE,
+	DLB2_DIR_QUEUE,
+	DLB2_NUM_QUEUE_TYPES /* Must be last */
+};
+
+#define PORT_TYPE(p) ((p)->is_directed ? DLB2_DIR_PORT : DLB2_LDB_PORT)
+
+/* Do not change - must match hardware! */
+enum dlb2_hw_sched_type {
+	DLB2_SCHED_ATOMIC = 0,
+	DLB2_SCHED_UNORDERED,
+	DLB2_SCHED_ORDERED,
+	DLB2_SCHED_DIRECTED,
+	/* DLB2_NUM_HW_SCHED_TYPES must be last */
+	DLB2_NUM_HW_SCHED_TYPES
+};
+
+struct dlb2_hw_rsrcs {
+	int32_t nb_events_limit;
+	uint32_t num_queues;		/* Total queues (lb + dir) */
+	uint32_t num_ldb_queues;	/* Number of available ldb queues */
+	uint32_t num_ldb_ports;         /* Number of load balanced ports */
+	uint32_t num_dir_ports;         /* Number of directed ports */
+	uint32_t num_ldb_credits;       /* Number of load balanced credits */
+	uint32_t num_dir_credits;       /* Number of directed credits */
+	uint32_t reorder_window_size;   /* Size of reorder window */
+};
+
+struct dlb2_hw_resource_info {
+	/**> Max resources that can be provided */
+	struct dlb2_hw_rsrcs hw_rsrc_max;
+	int num_sched_domains;
+	uint32_t socket_id;
+};
+
+enum dlb2_enqueue_type {
+	/**>
+	 * New : Used to inject a new packet into the QM.
+	 */
+	DLB2_ENQ_NEW,
+	/**>
+	 * Forward : Enqueues a packet, and
+	 *  - if atomic: release any lock it holds in the QM
+	 *  - if ordered: release the packet for egress re-ordering
+	 */
+	DLB2_ENQ_FWD,
+	/**>
+	 * Enqueue Drop : Release an inflight packet. Must be called with
+	 * event == NULL. Used to drop a packet.
+	 *
+	 * Note that all packets dequeued from a load-balanced port must be
+	 * released, either with DLB2_ENQ_DROP or DLB2_ENQ_FWD.
+	 */
+	DLB2_ENQ_DROP,
+
+	/* marker for array sizing etc. */
+	_DLB2_NB_ENQ_TYPES
+};
+
+/* hw-specific format - do not change */
+
+struct dlb2_event_type {
+	uint8_t major:4;
+	uint8_t unused:4;
+	uint8_t sub;
+};
+
+union dlb2_opaque_data {
+	uint16_t opaque_data;
+	struct dlb2_event_type event_type;
+};
+
+struct dlb2_msg_info {
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+};
+
+#define DLB2_NEW_CMD_BYTE 0x08
+#define DLB2_FWD_CMD_BYTE 0x0A
+#define DLB2_COMP_CMD_BYTE 0x02
+#define DLB2_POP_CMD_BYTE 0x01
+#define DLB2_NOOP_CMD_BYTE 0x00
+
+/* hw-specific format - do not change */
+struct dlb2_enqueue_qe {
+	uint64_t data;
+	/* Word 3 */
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	/* Word 4 */
+	uint16_t lock_id;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_cq_pop_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t tokens:10;
+	uint16_t rsvd2:6;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_dequeue_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t flow_id:16; /* was pp_id in v1 */
+	uint8_t debug;
+	uint8_t cq_gen:1;
+	uint8_t qid_depth:2; /* 2 bits in v2 */
+	uint8_t rsvd1:2;
+	uint8_t error:1;
+	uint8_t rsvd2:2;
+};
+
+union dlb2_port_config {
+	struct dlb2_create_ldb_port_args ldb;
+	struct dlb2_create_dir_port_args dir;
+};
+
+enum dlb2_port_state {
+	PORT_CLOSED,
+	PORT_STARTED,
+	PORT_STOPPED
+};
+
+enum dlb2_configuration_state {
+	/* The resource has not been configured */
+	DLB2_NOT_CONFIGURED,
+	/* The resource was configured, but the device was stopped */
+	DLB2_PREV_CONFIGURED,
+	/* The resource is currently configured */
+	DLB2_CONFIGURED
+};
+
+struct dlb2_port {
+	uint32_t id;
+	bool is_directed;
+	bool gen_bit;
+	uint16_t dir_credits;
+	uint32_t dequeue_depth;
+	union dlb2_port_config cfg;
+	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
+	uint16_t cached_ldb_credits;
+	uint16_t ldb_credits;
+	uint16_t cached_dir_credits;
+	bool int_armed;
+	uint16_t owed_tokens;
+	int16_t issued_releases;
+	int cq_depth;
+	uint16_t cq_idx;
+	uint16_t cq_idx_unmasked;
+	uint16_t cq_depth_mask;
+	uint16_t gen_bit_shift;
+	enum dlb2_port_state state;
+	enum dlb2_configuration_state config_state;
+	int num_mapped_qids;
+	uint8_t *qid_mappings;
+	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
+	struct dlb2_enqueue_qe *int_arm_qe;
+	struct dlb2_cq_pop_qe *consume_qe;
+	struct dlb2_eventdev *dlb2; /* back ptr */
+	struct dlb2_eventdev_port *ev_port; /* back ptr */
+};
+
+/* Per-process per-port mmio and memory pointers */
+struct process_local_port_data {
+	uint64_t *pp_addr;
+	struct dlb2_dequeue_qe *cq_base;
+	const struct rte_memzone *mz;
+	bool mmaped;
+};
+
+struct dlb2_eventdev;
+
+struct dlb2_config {
+	int configured;
+	int reserved;
+	uint32_t num_ldb_credits;
+	uint32_t num_dir_credits;
+	struct dlb2_create_sched_domain_args resources;
+};
+
+enum dlb2_cos {
+	DLB2_COS_DEFAULT = -1,
+	DLB2_COS_0 = 0,
+	DLB2_COS_1,
+	DLB2_COS_2,
+	DLB2_COS_3
+};
+
+struct dlb2_hw_dev {
+	struct dlb2_config cfg;
+	struct dlb2_hw_resource_info info;
+	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
+	uint32_t domain_id;
+	enum dlb2_cos cos_id;
+	rte_spinlock_t resource_lock; /* for MP support */
+} __rte_cache_aligned;
+
+/* End HW related defines and structs */
+
+/* Begin DLB2 PMD Eventdev related defines and structs */
+
+#define DLB2_MAX_NUM_QUEUES \
+	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
+
+#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS + DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
+
+/** Structure to hold the queue to port link establishment attributes */
+
+struct dlb2_event_queue_link {
+	uint8_t queue_id;
+	uint8_t priority;
+	bool mapped;
+	bool valid;
+};
+
+struct dlb2_traffic_stats {
+	uint64_t rx_ok;
+	uint64_t rx_drop;
+	uint64_t rx_interrupt_wait;
+	uint64_t rx_umonitor_umwait;
+	uint64_t tx_ok;
+	uint64_t total_polls;
+	uint64_t zero_polls;
+	uint64_t tx_nospc_ldb_hw_credits;
+	uint64_t tx_nospc_dir_hw_credits;
+	uint64_t tx_nospc_inflight_max;
+	uint64_t tx_nospc_new_event_limit;
+	uint64_t tx_nospc_inflight_credits;
+};
+
+/* DLB2 HW sets the 2bit qid_depth in rx QEs based on the programmable depth
+ * threshold. The global default value in config/common_base (or rte_config.h)
+ * can be overridden on a per-qid basis using a vdev command line parameter.
+ * 3: depth > threshold
+ * 2: threshold >= depth > 3/4 threshold
+ * 1: 3/4 threshold >= depth > 1/2 threshold
+ * 0: depth <= 1/2 threshold.
+ */
+#define DLB2_QID_DEPTH_LE50 0
+#define DLB2_QID_DEPTH_GT50_LE75 1
+#define DLB2_QID_DEPTH_GT75_LE100 2
+#define DLB2_QID_DEPTH_GT100 3
+#define DLB2_NUM_QID_DEPTH_STAT_VALS 4 /* 2 bits */
+
+struct dlb2_queue_stats {
+	uint64_t enq_ok;
+	uint64_t qid_depth[DLB2_NUM_QID_DEPTH_STAT_VALS];
+};
+
+struct dlb2_port_stats {
+	struct dlb2_traffic_stats traffic;
+	uint64_t tx_op_cnt[4]; /* indexed by rte_event.op */
+	uint64_t tx_implicit_rel;
+	uint64_t tx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t tx_invalid;
+	uint64_t rx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t rx_sched_invalid;
+	struct dlb2_queue_stats queue[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_eventdev_port {
+	struct dlb2_port qm_port; /* hw specific data structure */
+	struct rte_event_port_conf conf; /* user-supplied configuration */
+	uint16_t inflight_credits; /* num credits this port has right now */
+	uint16_t credit_update_quanta;
+	struct dlb2_eventdev *dlb2; /* backlink optimization */
+	struct dlb2_port_stats stats __rte_cache_aligned;
+	struct dlb2_event_queue_link link[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	int num_links;
+	uint32_t id; /* port id */
+	/* num releases yet to be completed on this port.
+	 * Only applies to load-balanced ports.
+	 */
+	uint16_t outstanding_releases;
+	uint16_t inflight_max; /* app requested max inflights for this port */
+	/* setup_done is set when the event port is setup */
+	bool setup_done;
+	/* enq_configured is set when the qm port is created */
+	bool enq_configured;
+	uint8_t implicit_release; /* release events before dequeueing */
+}  __rte_cache_aligned;
+
+struct dlb2_queue {
+	uint32_t num_qid_inflights; /* User config */
+	uint32_t num_atm_inflights; /* User config */
+	enum dlb2_configuration_state config_state;
+	int sched_type; /* LB queue only */
+	uint32_t id;
+	bool is_directed;
+};
+
+struct dlb2_eventdev_queue {
+	struct dlb2_queue qm_queue;
+	struct rte_event_queue_conf conf; /* User config */
+	int depth_threshold; /* use default if 0 */
+	uint32_t id;
+	bool setup_done;
+	uint8_t num_links;
+};
+
+enum dlb2_run_state {
+	DLB2_RUN_STATE_STOPPED = 0,
+	DLB2_RUN_STATE_STOPPING,
+	DLB2_RUN_STATE_STARTING,
+	DLB2_RUN_STATE_STARTED
+};
+
+struct dlb2_eventdev {
+	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
+	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each queue */
+	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
+	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each port */
+	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
+	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
+	struct dlb2_get_num_resources_args hw_rsrc_query_results;
+	uint32_t xstats_count_mode_queue;
+	struct dlb2_hw_dev qm_instance; /* strictly hw related */
+	uint64_t global_dequeue_wait_ticks;
+	struct dlb2_xstats_entry *xstats;
+	struct rte_eventdev *event_dev; /* backlink to dev */
+	uint32_t xstats_count_mode_dev;
+	uint32_t xstats_count_mode_port;
+	uint32_t xstats_count;
+	uint32_t inflights; /* use __atomic builtins */
+	uint32_t new_event_limit;
+	int max_num_events_override;
+	int num_dir_credits_override;
+	volatile enum dlb2_run_state run_state;
+	uint16_t num_dir_queues; /* total num of evdev dir queues requested */
+	uint16_t num_dir_credits;
+	uint16_t num_ldb_credits;
+	uint16_t num_queues; /* total queues */
+	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested */
+	uint16_t num_ports; /* total num of evdev ports requested */
+	uint16_t num_ldb_ports; /* total num of ldb ports requested */
+	uint16_t num_dir_ports; /* total num of dir ports requested */
+	bool umwait_allowed;
+	bool global_dequeue_wait; /* Not using per dequeue wait if true */
+	bool defer_sched;
+	enum dlb2_cq_poll_modes poll_mode;
+	uint8_t revision;
+	bool configured;
+	uint16_t max_ldb_credits;
+	uint16_t max_dir_credits;
+
+	/* force hw credit pool counters into exclusive cache lines */
+
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t ldb_credit_pool __rte_cache_aligned;
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t dir_credit_pool __rte_cache_aligned;
+};
+
+/* used for collecting and passing around the dev args */
+struct dlb2_qid_depth_thresholds {
+	int val[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_devargs {
+	int socket_id;
+	int max_num_events;
+	int num_dir_credits_override;
+	int dev_id;
+	int defer_sched;
+	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
+	enum dlb2_cos cos_id;
+};
+
+/* End Eventdev related defines and structs */
+
+/* Forwards for non-inlined functions */
+
+void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
+
+int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
+
+void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
+
+int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n);
+
+int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstat_names,
+		unsigned int *ids, unsigned int size);
+
+uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+					  const char *name, unsigned int *id);
+
+int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode,
+		int16_t queue_port_id,
+		const uint32_t ids[],
+		uint32_t nb_ids);
+
+int test_dlb2_eventdev(void);
+
+int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+				const char *name,
+				struct dlb2_devargs *dlb2_args);
+
+int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+				  const char *name);
+
+uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *queue);
+
+int dlb2_parse_params(const char *params,
+		      const char *name,
+		      struct dlb2_devargs *dlb2_args);
+
+/* Extern globals */
+extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
+
+#endif	/* _DLB2_PRIV_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v6 04/23] event/dlb2: add definitions shared with LKM or shared code

[Timothy McDaniel]

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_user.h | 679 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 679 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_user.h

diff --git a/drivers/event/dlb2/dlb2_user.h b/drivers/event/dlb2/dlb2_user.h
new file mode 100644
index 0000000..f4bda78
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_user.h
@@ -0,0 +1,679 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#define DLB2_MAX_NAME_LEN 64
+
+#include <linux/types.h>
+
+enum dlb2_error {
+	DLB2_ST_SUCCESS = 0,
+	DLB2_ST_NAME_EXISTS,
+	DLB2_ST_DOMAIN_UNAVAILABLE,
+	DLB2_ST_LDB_PORTS_UNAVAILABLE,
+	DLB2_ST_DIR_PORTS_UNAVAILABLE,
+	DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB2_ST_INVALID_DOMAIN_ID,
+	DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB2_ST_INVALID_LDB_QUEUE_ID,
+	DLB2_ST_INVALID_CQ_DEPTH,
+	DLB2_ST_INVALID_CQ_VIRT_ADDR,
+	DLB2_ST_INVALID_PORT_ID,
+	DLB2_ST_INVALID_QID,
+	DLB2_ST_INVALID_PRIORITY,
+	DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+	DLB2_ST_INVALID_DIR_QUEUE_ID,
+	DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB2_ST_DOMAIN_NOT_CONFIGURED,
+	DLB2_ST_INTERNAL_ERROR,
+	DLB2_ST_DOMAIN_IN_USE,
+	DLB2_ST_DOMAIN_NOT_FOUND,
+	DLB2_ST_QUEUE_NOT_FOUND,
+	DLB2_ST_DOMAIN_STARTED,
+	DLB2_ST_DOMAIN_NOT_STARTED,
+	DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB2_ST_DOMAIN_RESET_FAILED,
+	DLB2_ST_MBOX_ERROR,
+	DLB2_ST_INVALID_HIST_LIST_DEPTH,
+	DLB2_ST_NO_MEMORY,
+	DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB2_ST_INVALID_COS_ID,
+};
+
+static const char dlb2_error_strings[][128] = {
+	"DLB2_ST_SUCCESS",
+	"DLB2_ST_NAME_EXISTS",
+	"DLB2_ST_DOMAIN_UNAVAILABLE",
+	"DLB2_ST_LDB_PORTS_UNAVAILABLE",
+	"DLB2_ST_DIR_PORTS_UNAVAILABLE",
+	"DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+	"DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+	"DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+	"DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+	"DLB2_ST_INVALID_DOMAIN_ID",
+	"DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+	"DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+	"DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+	"DLB2_ST_INVALID_LDB_QUEUE_ID",
+	"DLB2_ST_INVALID_CQ_DEPTH",
+	"DLB2_ST_INVALID_CQ_VIRT_ADDR",
+	"DLB2_ST_INVALID_PORT_ID",
+	"DLB2_ST_INVALID_QID",
+	"DLB2_ST_INVALID_PRIORITY",
+	"DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+	"DLB2_ST_INVALID_DIR_QUEUE_ID",
+	"DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+	"DLB2_ST_DOMAIN_NOT_CONFIGURED",
+	"DLB2_ST_INTERNAL_ERROR",
+	"DLB2_ST_DOMAIN_IN_USE",
+	"DLB2_ST_DOMAIN_NOT_FOUND",
+	"DLB2_ST_QUEUE_NOT_FOUND",
+	"DLB2_ST_DOMAIN_STARTED",
+	"DLB2_ST_DOMAIN_NOT_STARTED",
+	"DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+	"DLB2_ST_DOMAIN_RESET_FAILED",
+	"DLB2_ST_MBOX_ERROR",
+	"DLB2_ST_INVALID_HIST_LIST_DEPTH",
+	"DLB2_ST_NO_MEMORY",
+	"DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+	"DLB2_ST_INVALID_COS_ID",
+};
+
+struct dlb2_cmd_response {
+	__u32 status; /* Interpret using enum dlb2_error */
+	__u32 id;
+};
+
+/*******************/
+/* 'dlb2' commands */
+/*******************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+	DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	This ioctl interface is the same in all driver versions and is always
+ *	the first ioctl.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+	/* Output parameters */
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ *	load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ *		percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ *	load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ *   ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+enum dlb2_cq_poll_modes {
+	DLB2_CQ_POLL_MODE_STD,
+	DLB2_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB2_CQ_POLL_MODE must be last */
+	NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/********************************/
+/* 'scheduling domain' commands */
+/********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_start_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
+#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
+
+
+#endif /* __DLB2_USER_H */
-- 
2.6.4

[dpdk-dev] [PATCH v6 05/23] event/dlb2: add inline functions

[Timothy McDaniel]

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_inline_fns.h | 61 ++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h

diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
new file mode 100644
index 0000000..10bd062
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_inline_fns.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_INLINE_FNS_H_
+#define _DLB2_INLINE_FNS_H_
+
+/* Inline functions required in more than one source file. */
+
+static inline struct dlb2_eventdev *
+dlb2_pmd_priv(const struct rte_eventdev *eventdev)
+{
+	return eventdev->data->dev_private;
+}
+
+static inline void
+dlb2_umonitor(volatile void *addr)
+{
+	asm volatile(".byte 0xf3, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (addr));
+}
+
+static inline void
+dlb2_umwait(int state, uint64_t timeout)
+{
+	uint32_t eax = timeout & UINT32_MAX;
+	uint32_t edx = timeout >> 32;
+
+	asm volatile(".byte 0xf2, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (state),  "a" (eax), "d" (edx));
+}
+
+static inline void
+dlb2_movntdq_single(void *pp_addr, void *qe4)
+{
+	long long *_qe  = (long long *)qe4;
+	__v2di src_data0 = (__v2di){_qe[0], _qe[1]};
+
+	_mm_stream_si128(pp_addr, src_data0);
+}
+
+static inline void
+dlb2_cldemote(void *addr)
+{
+	/* Load addr into RSI, then demote the cache line of the address
+	 * contained in that register.
+	 */
+	asm volatile(".byte 0x0f, 0x1c, 0x06" :: "S" (addr));
+}
+
+static inline void
+dlb2_movdir64b(void *pp_addr, void *qe4)
+{
+	asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
+		     :
+		     : "a" (pp_addr), "d" (qe4));
+}
+
+#endif /* _DLB2_INLINE_FNS_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v6 06/23] event/dlb2: add eventdev probe

[Timothy McDaniel]

Add the eventdev portion of probe, and parse command line
options, but do not initialize hardware.

Changes since v2 patch-set probe:

Primary and secondary probe-time init has been removed, and
will be introduced in subsequent patches contained in
the v3 patch-set.

Hardware init has been moved to a subsequent patch in order to
minimize the patch size. The files dlb2/pf/dlb2_pf.c and
dlb2/pf/dlb_main.c include stubs of hardware init functions,
and these will be replaced with the real versions when the
hardware layer is committed.

Initialization of the flexible interface layer has been moved to
a subsequent patch in order to minimize patch size.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                      |  343 ++++
 drivers/event/dlb2/meson.build                 |    5 +-
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +++
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  621 ++++++
 drivers/event/dlb2/pf/dlb2_main.h              |   98 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  196 ++
 13 files changed, 7497 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2082dd6..1b11979 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2,6 +2,349 @@
  * Copyright(c) 2016-2020 Intel Corporation
  */
 
+#include <assert.h>
+#include <errno.h>
+#include <nmmintrin.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_cycles.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_io.h>
+#include <rte_kvargs.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_ring.h>
+#include <rte_string_fns.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+/*
+ * Resources exposed to eventdev. Some values overridden at runtime using
+ * values returned by the DLB kernel driver.
+ */
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
+#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
+#endif
+
+struct process_local_port_data
+dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
+
+#define DLB2_BASE_10 10
+
+static int
+dlb2_string_to_int(int *result, const char *str)
+{
+	long ret;
+	char *endptr;
+
+	if (str == NULL || result == NULL)
+		return -EINVAL;
+
+	errno = 0;
+	ret = strtol(str, &endptr, DLB2_BASE_10);
+	if (errno)
+		return -errno;
+
+	/* long int and int may be different width for some architectures */
+	if (ret < INT_MIN || ret > INT_MAX || endptr == str)
+		return -EINVAL;
+
+	*result = ret;
+	return 0;
+}
+
+static int
+set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
+{
+	int *socket_id = opaque;
+	int ret;
+
+	ret = dlb2_string_to_int(socket_id, value);
+	if (ret < 0)
+		return ret;
+
+	if (*socket_id > RTE_MAX_NUMA_NODES)
+		return -EINVAL;
+	return 0;
+}
+
+static int
+set_max_num_events(const char *key __rte_unused,
+		   const char *value,
+		   void *opaque)
+{
+	int *max_num_events = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(max_num_events, value);
+	if (ret < 0)
+		return ret;
+
+	if (*max_num_events < 0 || *max_num_events >
+			DLB2_MAX_NUM_LDB_CREDITS) {
+		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
+			     DLB2_MAX_NUM_LDB_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_num_dir_credits(const char *key __rte_unused,
+		    const char *value,
+		    void *opaque)
+{
+	int *num_dir_credits = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(num_dir_credits, value);
+	if (ret < 0)
+		return ret;
+
+	if (*num_dir_credits < 0 ||
+	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
+		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
+			     DLB2_MAX_NUM_DIR_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_dev_id(const char *key __rte_unused,
+	   const char *value,
+	   void *opaque)
+{
+	int *dev_id = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(dev_id, value);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+set_cos(const char *key __rte_unused,
+	const char *value,
+	void *opaque)
+{
+	enum dlb2_cos *cos_id = opaque;
+	int x = 0;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(&x, value);
+	if (ret < 0)
+		return ret;
+
+	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
+		DLB2_LOG_ERR(
+			"COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
+			x);
+		return -EINVAL;
+	}
+
+	*cos_id = x;
+
+	return 0;
+}
+
+
+static int
+set_qid_depth_thresh(const char *key __rte_unused,
+		     const char *value,
+		     void *opaque)
+{
+	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
+	int first, last, thresh, i;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	/* command line override may take one of the following 3 forms:
+	 * qid_depth_thresh=all:<threshold_value> ... all queues
+	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
+	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
+	 */
+	if (sscanf(value, "all:%d", &thresh) == 1) {
+		first = 0;
+		last = DLB2_MAX_NUM_QUEUES - 1;
+	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
+		/* we have everything we need */
+	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
+		last = first;
+	} else {
+		DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val, qid-qid:val, or qid:val\n");
+		return -EINVAL;
+	}
+
+	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
+		return -EINVAL;
+	}
+
+	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
+			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
+		return -EINVAL;
+	}
+
+	for (i = first; i <= last; i++)
+		qid_thresh->val[i] = thresh; /* indexed by qid */
+
+	return 0;
+}
+
+int
+dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+			    const char *name,
+			    struct dlb2_devargs *dlb2_args)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+	RTE_SET_USED(dlb2_args);
+
+	return 0;
+}
+
+int
+dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+			      const char *name)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+int
+dlb2_parse_params(const char *params,
+		  const char *name,
+		  struct dlb2_devargs *dlb2_args)
+{
+	int ret = 0;
+	static const char * const args[] = { NUMA_NODE_ARG,
+					     DLB2_MAX_NUM_EVENTS,
+					     DLB2_NUM_DIR_CREDITS,
+					     DEV_ID_ARG,
+					     DLB2_QID_DEPTH_THRESH_ARG,
+					     DLB2_COS_ARG,
+					     NULL };
+
+	if (params != NULL && params[0] != '\0') {
+		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
+
+		if (kvlist == NULL) {
+			RTE_LOG(INFO, PMD,
+				"Ignoring unsupported parameters when creating device '%s'\n",
+				name);
+		} else {
+			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
+						     set_numa_node,
+						     &dlb2_args->socket_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing numa node parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
+						 set_max_num_events,
+						 &dlb2_args->max_num_events);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist,
+					DLB2_NUM_DIR_CREDITS,
+					set_num_dir_credits,
+					&dlb2_args->num_dir_credits_override);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
+						 set_dev_id,
+						 &dlb2_args->dev_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(
+					kvlist,
+					DLB2_QID_DEPTH_THRESH_ARG,
+					set_qid_depth_thresh,
+					&dlb2_args->qid_depth_thresholds);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
+						 set_cos,
+						 &dlb2_args->cos_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing cos parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
 
+			rte_kvargs_free(kvlist);
+		}
+	}
+	return ret;
+}
 RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 435c1ee..1eab9b9 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,6 +7,9 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files('dlb2.c')
+sources = files('dlb2.c',
+		'pf/dlb2_main.c',
+		'pf/dlb2_pf.c'
+)
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
new file mode 100644
index 0000000..1d99f1e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_osdep_list.h"
+#include "dlb2_osdep_types.h"
+
+#define DLB2_MAX_NUM_VDEVS			16
+#define DLB2_MAX_NUM_DOMAINS			32
+#define DLB2_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS			64
+#define DLB2_MAX_NUM_DIR_PORTS			64
+#define DLB2_MAX_NUM_LDB_CREDITS		(8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS		(2 * 1024)
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB2_MAX_NUM_AQED_ENTRIES		2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB2_QID_PRIORITIES			8
+#define DLB2_NUM_ARB_WEIGHTS			8
+#define DLB2_MAX_WEIGHT				255
+#define DLB2_NUM_COS_DOMAINS			4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#ifdef FPGA
+#define DLB2_HZ					2000000
+#else
+#define DLB2_HZ					800000000
+#endif
+
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
+
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+	DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+	DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID		3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID		4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW		5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ	1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
+#define DLB2_VF_BASE_CQ_VECTOR_ID	     0
+#define DLB2_VF_LAST_CQ_VECTOR_ID	     30
+#define DLB2_VF_MBOX_VECTOR_ID		     31
+#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
+
+#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
+					     DLB2_MAX_NUM_DIR_PORTS + 1)
+
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE   0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
+				DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE   0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
+				DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE       0x2100000
+#define DLB2_LDB_PP_STRIDE     0x1000
+#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
+				DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE       0x2000000
+#define DLB2_DIR_PP_STRIDE     0x1000
+#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
+				DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
+struct dlb2_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb2_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb2_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
+struct dlb2_ldb_queue {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB2_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB2_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB2_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB2_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+	enum dlb2_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb2_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+	u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+	u32 bound[6] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & (1 << i))) {
+			group->slot_use_bitmap |= 1 << i;
+			return i;
+		}
+	}
+
+	return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+	int i, cnt = 0;
+
+	for (i = 0; i < 32; i++)
+		cnt += !!(group->slot_use_bitmap & (1 << i));
+
+	return cnt;
+}
+
+struct dlb2_hw_domain {
+	struct dlb2_function_resources *parent_func;
+	struct dlb2_list_entry func_list;
+	struct dlb2_list_head used_ldb_queues;
+	struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head used_dir_pq_pairs;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb2_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb2_bitmap;
+
+struct dlb2_function_resources {
+	struct dlb2_list_head avail_domains;
+	struct dlb2_list_head used_domains;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	struct dlb2_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+	struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+	struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+	struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
+struct dlb2_mbox {
+	u32 *mbox;
+	u32 *isr_in_progress;
+};
+
+struct dlb2_sw_mbox {
+	struct dlb2_mbox vdev_to_pf;
+	struct dlb2_mbox pf_to_vdev;
+	void (*pf_to_vdev_inject)(void *arg);
+	void *pf_to_vdev_inject_arg;
+};
+
+struct dlb2_hw {
+	/* BAR 0 address */
+	void  *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void  *func_kva;
+	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb2_hw_resources rsrcs;
+	struct dlb2_function_resources pf;
+	struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+	struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+	/* Virtualization */
+	int virt_mode;
+	struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
+	unsigned int pasid[DLB2_MAX_NUM_VDEVS];
+};
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
new file mode 100644
index 0000000..ce462c0
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
@@ -0,0 +1,596 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_BASE_DLB2_MBOX_H
+#define __DLB2_BASE_DLB2_MBOX_H
+
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+
+#define DLB2_MBOX_INTERFACE_VERSION 1
+
+/*
+ * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
+ * to send requests of its own (e.g. notifying a VF of an impending FLR).
+ * To avoid communication race conditions, e.g. the PF sends a response and then
+ * sends a request before the VF reads the response, the PF->VF mailbox is
+ * divided into two sections:
+ * - Bytes 0-47: PF responses
+ * - Bytes 48-63: PF requests
+ *
+ * Partitioning the PF->VF mailbox allows responses and requests to occupy the
+ * mailbox simultaneously.
+ */
+#define DLB2_PF2VF_RESP_BYTES	  48
+#define DLB2_PF2VF_RESP_BASE	  0
+#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
+
+#define DLB2_PF2VF_REQ_BYTES	  16
+#define DLB2_PF2VF_REQ_BASE	  (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
+#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
+
+/*
+ * Similarly, the VF->PF mailbox is divided into two sections:
+ * - Bytes 0-239: VF requests
+ * -- (Bytes 0-3 are unused due to a hardware errata)
+ * - Bytes 240-255: VF responses
+ */
+#define DLB2_VF2PF_REQ_BYTES	 236
+#define DLB2_VF2PF_REQ_BASE	 4
+#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
+
+#define DLB2_VF2PF_RESP_BYTES	  16
+#define DLB2_VF2PF_RESP_BASE	  (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
+#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
+
+/* VF-initiated commands */
+enum dlb2_mbox_cmd_type {
+	DLB2_MBOX_CMD_REGISTER,
+	DLB2_MBOX_CMD_UNREGISTER,
+	DLB2_MBOX_CMD_GET_NUM_RESOURCES,
+	DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
+	DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
+	DLB2_MBOX_CMD_CREATE_LDB_PORT,
+	DLB2_MBOX_CMD_CREATE_DIR_PORT,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT,
+	DLB2_MBOX_CMD_DISABLE_LDB_PORT,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT,
+	DLB2_MBOX_CMD_DISABLE_DIR_PORT,
+	DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_MAP_QID,
+	DLB2_MBOX_CMD_UNMAP_QID,
+	DLB2_MBOX_CMD_START_DOMAIN,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
+	DLB2_MBOX_CMD_ARM_CQ_INTR,
+	DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
+	DLB2_MBOX_CMD_GET_SN_ALLOCATION,
+	DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
+	DLB2_MBOX_CMD_GET_COS_BW,
+	DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
+	DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_QE_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_CMD_TYPES,
+};
+
+static const char dlb2_mbox_cmd_type_strings[][128] = {
+	"DLB2_MBOX_CMD_REGISTER",
+	"DLB2_MBOX_CMD_UNREGISTER",
+	"DLB2_MBOX_CMD_GET_NUM_RESOURCES",
+	"DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_LDB_PORT",
+	"DLB2_MBOX_CMD_CREATE_DIR_PORT",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_DISABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_DISABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_MAP_QID",
+	"DLB2_MBOX_CMD_UNMAP_QID",
+	"DLB2_MBOX_CMD_START_DOMAIN",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
+	"DLB2_MBOX_CMD_ARM_CQ_INTR",
+	"DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
+	"DLB2_MBOX_CMD_GET_SN_ALLOCATION",
+	"DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
+	"DLB2_MBOX_CMD_GET_COS_BW",
+	"DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
+	"DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
+};
+
+/* PF-initiated commands */
+enum dlb2_mbox_vf_cmd_type {
+	DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
+	DLB2_MBOX_VF_CMD_NOTIFICATION,
+	DLB2_MBOX_VF_CMD_IN_USE,
+
+	/* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_VF_CMD_TYPES,
+};
+
+static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
+	"DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
+	"DLB2_MBOX_VF_CMD_NOTIFICATION",
+	"DLB2_MBOX_VF_CMD_IN_USE",
+};
+
+#define DLB2_MBOX_CMD_TYPE(hdr) \
+	(((struct dlb2_mbox_req_hdr *)hdr)->type)
+#define DLB2_MBOX_CMD_STRING(hdr) \
+	dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
+
+enum dlb2_mbox_status_type {
+	DLB2_MBOX_ST_SUCCESS,
+	DLB2_MBOX_ST_INVALID_CMD_TYPE,
+	DLB2_MBOX_ST_VERSION_MISMATCH,
+	DLB2_MBOX_ST_INVALID_OWNER_VF,
+};
+
+static const char dlb2_mbox_status_type_strings[][128] = {
+	"DLB2_MBOX_ST_SUCCESS",
+	"DLB2_MBOX_ST_INVALID_CMD_TYPE",
+	"DLB2_MBOX_ST_VERSION_MISMATCH",
+	"DLB2_MBOX_ST_INVALID_OWNER_VF",
+};
+
+#define DLB2_MBOX_ST_TYPE(hdr) \
+	(((struct dlb2_mbox_resp_hdr *)hdr)->status)
+#define DLB2_MBOX_ST_STRING(hdr) \
+	dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
+
+/* This structure is always the first field in a request structure */
+struct dlb2_mbox_req_hdr {
+	u32 type;
+};
+
+/* This structure is always the first field in a response structure */
+struct dlb2_mbox_resp_hdr {
+	u32 status;
+};
+
+struct dlb2_mbox_register_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 min_interface_version;
+	u16 max_interface_version;
+};
+
+struct dlb2_mbox_register_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 interface_version;
+	u8 pf_id;
+	u8 vf_id;
+	u8 is_auxiliary_vf;
+	u8 primary_vf_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u16 num_sched_domains;
+	u16 num_ldb_queues;
+	u16 num_ldb_ports;
+	u16 num_cos_ldb_ports[4];
+	u16 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 max_contiguous_hist_list_entries;
+	u16 num_ldb_credits;
+	u16 num_dir_credits;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 num_ldb_queues;
+	u32 num_ldb_ports;
+	u32 num_cos_ldb_ports[4];
+	u32 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u8 cos_strict;
+	u8 padding0[3];
+	u32 padding1;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 num_sequence_numbers;
+	u32 num_qid_inflights;
+	u32 num_atomic_inflights;
+	u32 lock_id_comp_level;
+	u32 depth_threshold;
+	u32 padding;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 depth_threshold;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u16 cq_depth;
+	u16 cq_history_list_size;
+	u8 cos_id;
+	u8 cos_strict;
+	u16 padding1;
+	u64 cq_base_address;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u64 cq_base_address;
+	u16 cq_depth;
+	u16 padding0;
+	s32 queue_id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_map_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+	u32 priority;
+	u32 padding0;
+};
+
+struct dlb2_mbox_map_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_start_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+};
+
+struct dlb2_mbox_start_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 is_ldb;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding0;
+};
+
+/*
+ * The alert_id and aux_alert_data follows the format of the alerts defined in
+ * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
+ * the aux_alert_data value varies depending on the alert.
+ */
+struct dlb2_mbox_vf_alert_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 alert_id;
+	u32 aux_alert_data;
+};
+
+enum dlb2_mbox_vf_notification_type {
+	DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
+	DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
+
+	/* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
+	NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
+};
+
+struct dlb2_mbox_vf_notification_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 notification;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 in_use;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 num;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 cos_id;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 mode;
+};
+
+#endif /* __DLB2_BASE_DLB2_MBOX_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
new file mode 100644
index 0000000..c4c34eb
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
@@ -0,0 +1,230 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_H
+#define __DLB2_OSDEP_H
+
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <rte_string_fns.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include "../dlb2_main.h"
+#include "dlb2_resource.h"
+#include "../../dlb2_log.h"
+#include "../../dlb2_user.h"
+
+
+#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
+#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
+#define DLB2_CSR_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
+#define DLB2_FUNC_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
+/* Map to PMDs logging interface */
+#define DLB2_ERR(dev, fmt, args...) \
+	DLB2_LOG_ERR(fmt, ## args)
+
+#define DLB2_INFO(dev, fmt, args...) \
+	DLB2_LOG_INFO(fmt, ## args)
+
+#define DLB2_DEBUG(dev, fmt, args...) \
+	DLB2_LOG_DBG(fmt, ## args)
+
+/**
+ * os_udelay() - busy-wait for a number of microseconds
+ * @usecs: delay duration.
+ */
+static inline void os_udelay(int usecs)
+{
+	rte_delay_us(usecs);
+}
+
+/**
+ * os_msleep() - sleep for a number of milliseconds
+ * @usecs: delay duration.
+ */
+static inline void os_msleep(int msecs)
+{
+	rte_delay_ms(msecs);
+}
+
+#define DLB2_PP_BASE(__is_ldb) \
+	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
+
+/**
+ * os_map_producer_port() - map a producer port into the caller's address space
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ *
+ * This function maps the requested producer port memory into the caller's
+ * address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void *os_map_producer_port(struct dlb2_hw *hw,
+					 u8 port_id,
+					 bool is_ldb)
+{
+	uint64_t addr;
+	uint64_t pp_dma_base;
+
+	pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
+	addr = (pp_dma_base + (PAGE_SIZE * port_id));
+
+	return (void *)(uintptr_t)addr;
+}
+
+/**
+ * os_unmap_producer_port() - unmap a producer port
+ * @addr: mapped producer port address
+ *
+ * This function undoes os_map_producer_port() by unmapping the producer port
+ * memory from the caller's address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(addr);
+}
+
+/**
+ * os_fence_hcw() - fence an HCW to ensure it arrives at the device
+ * @hw: dlb2_hw handle for a particular device.
+ * @pp_addr: producer port address
+ */
+static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
+{
+	RTE_SET_USED(hw);
+
+	/* To ensure outstanding HCWs reach the device, read the PP address. IA
+	 * memory ordering prevents reads from passing older writes, and the
+	 * mfence also ensures this.
+	 */
+	rte_mb();
+
+	*(volatile u64 *)pp_addr;
+}
+
+/**
+ * DLB2_HW_ERR() - log an error message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_ERR(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_ERR(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/**
+ * DLB2_HW_DBG() - log an info message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_DBG(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_DEBUG(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/* The callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void *dlb2_complete_queue_map_unmap(void *__args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
+	int ret;
+
+	while (1) {
+		rte_spinlock_lock(&dlb2_dev->resource_mutex);
+
+		ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+		ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+		if (ret != 0) {
+			rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+			/* Relinquish the CPU so the application can process
+			 * its CQs, so this function doesn't deadlock.
+			 */
+			sched_yield();
+		} else {
+			break;
+		}
+	}
+
+	dlb2_dev->worker_launched = false;
+
+	rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+
+	return NULL;
+}
+
+
+/**
+ * os_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static inline void os_schedule_work(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+	pthread_t complete_queue_map_unmap_thread;
+	int ret;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
+				     "dlb_queue_unmap_waiter",
+				     NULL,
+				     dlb2_complete_queue_map_unmap,
+				     dlb2_dev);
+	if (ret)
+		DLB2_ERR(dlb2_dev,
+			 "Could not create queue complete map/unmap thread, err=%d\n",
+			 ret);
+	else
+		dlb2_dev->worker_launched = true;
+}
+
+/**
+ * os_worker_active() - query whether the map/unmap worker thread is active
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether a thread (launched by
+ * os_schedule_work()) is active. This function is used to determine
+ * whether or not to launch a worker thread.
+ */
+static inline bool os_worker_active(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	return dlb2_dev->worker_launched;
+}
+
+#endif /*  __DLB2_OSDEP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
new file mode 100644
index 0000000..423233b
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <rte_bitmap.h>
+#include <rte_string_fns.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include "../dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+	struct rte_bitmap *map;
+	unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+				    unsigned int len)
+{
+	struct dlb2_bitmap *bm;
+	void *mem;
+	uint32_t alloc_size;
+	uint32_t nbits = (uint32_t)len;
+
+	if (bitmap == NULL || nbits == 0)
+		return -EINVAL;
+
+	/* Allocate DLB2 bitmap control struct */
+	bm = rte_malloc("DLB2_PF",
+			sizeof(struct dlb2_bitmap),
+			RTE_CACHE_LINE_SIZE);
+
+	if (bm == NULL)
+		return -ENOMEM;
+
+	/* Allocate bitmap memory */
+	alloc_size = rte_bitmap_get_memory_footprint(nbits);
+	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
+	if (mem == NULL) {
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->map = rte_bitmap_init(len, mem, alloc_size);
+	if (bm->map == NULL) {
+		rte_free(mem);
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap == NULL)
+		return;
+
+	rte_free(bitmap->map);
+	rte_free(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++)
+		rte_bitmap_set(bitmap->map, i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	rte_bitmap_reset(bitmap->map);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set() - set a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
+				  unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_set(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+					unsigned int bit,
+					unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_set(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear() - clear a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
+				    unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_clear(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+					  unsigned int bit,
+					  unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_clear(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+						 unsigned int len)
+{
+	unsigned int i, j = 0;
+
+	if (bitmap  == NULL || bitmap->map == NULL || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			if (++j == len)
+				return i - j + 1;
+		} else {
+			j = 0;
+		}
+	}
+
+	/* No set bit range of length len? */
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			return i;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+	int weight = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			weight++;
+	}
+	return weight;
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				      bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+	int max_len = 0, len = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			len++;
+		} else {
+			if (len > max_len)
+				max_len = len;
+			len = 0;
+		}
+	}
+
+	if (len > max_len)
+		max_len = len;
+
+	return max_len;
+}
+
+/**
+ * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
+ * @dest: pointer to dlb2_bitmap structure, which will contain the results of
+ *	  the 'or' of src1 and src2.
+ * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
+ * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
+ *
+ * This function 'or's two bitmaps together and stores the result in a third
+ * bitmap. The source and destination bitmaps can be the same.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - One of the bitmaps is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
+				 struct dlb2_bitmap *src1,
+				 struct dlb2_bitmap *src2)
+{
+	unsigned int i, min;
+	int numset = 0;
+
+	if (dest  == NULL || dest->map == NULL ||
+	    src1  == NULL || src1->map == NULL ||
+	    src2  == NULL || src2->map == NULL)
+		return -EINVAL;
+
+	min = dest->len;
+	min = (min > src1->len) ? src1->len : min;
+	min = (min > src2->len) ? src2->len : min;
+
+	for (i = 0; i != min; i++) {
+		if  (rte_bitmap_get(src1->map, i) ||
+		     rte_bitmap_get(src2->map, i)) {
+			rte_bitmap_set(dest->map, i);
+			numset++;
+		} else {
+			rte_bitmap_clear(dest->map, i);
+		}
+	}
+
+	return numset;
+}
+
+#endif /*  __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
new file mode 100644
index 0000000..5531739
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_LIST_H
+#define __DLB2_OSDEP_LIST_H
+
+#include <rte_tailq.h>
+
+struct dlb2_list_entry {
+	TAILQ_ENTRY(dlb2_list_entry) node;
+};
+
+/* Dummy - just a struct definition */
+TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
+
+/* =================
+ * TAILQ Supplements
+ * =================
+ */
+
+#ifndef TAILQ_FOREACH_ENTRY
+#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)		\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter)						\
+		&& (ptr = container_of(iter, typeof(*(ptr)), name)); \
+	    (iter) = TAILQ_NEXT((iter), node))
+#endif
+
+#ifndef TAILQ_FOREACH_ENTRY_SAFE
+#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)	\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter) &&						\
+		(ptr = container_of(iter, typeof(*(ptr)), name)) &&\
+		((tvar) = TAILQ_NEXT((iter), node), 1);	\
+	    (iter) = (tvar))
+#endif
+
+/***********************/
+/*** List operations ***/
+/***********************/
+
+/**
+ * dlb2_list_init_head() - initialize the head of a list
+ * @head: list head
+ */
+static inline void dlb2_list_init_head(struct dlb2_list_head *head)
+{
+	TAILQ_INIT(head);
+}
+
+/**
+ * dlb2_list_add() - add an entry to a list
+ * @head: list head
+ * @entry: new list entry
+ */
+static inline void
+dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
+{
+	TAILQ_INSERT_TAIL(head, entry, node);
+}
+
+/**
+ * dlb2_list_del() - delete an entry from a list
+ * @entry: list entry
+ * @head: list head
+ */
+static inline void dlb2_list_del(struct dlb2_list_head *head,
+				 struct dlb2_list_entry *entry)
+{
+	TAILQ_REMOVE(head, entry, node);
+}
+
+/**
+ * dlb2_list_empty() - check if a list is empty
+ * @head: list head
+ *
+ * Return:
+ * Returns 1 if empty, 0 if not.
+ */
+static inline int dlb2_list_empty(struct dlb2_list_head *head)
+{
+	return TAILQ_EMPTY(head);
+}
+
+/**
+ * dlb2_list_splice() - splice a list
+ * @src_head: list to be added
+ * @ head: where src_head will be inserted
+ */
+static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
+				    struct dlb2_list_head *head)
+{
+	TAILQ_CONCAT(head, src_head, node);
+}
+
+/**
+ * DLB2_LIST_HEAD() - retrieve the head of the list
+ * @head: list head
+ * @type: type of the list variable
+ * @name: name of the list field within the containing struct
+ */
+#define DLB2_LIST_HEAD(head, type, name)                       \
+	(TAILQ_FIRST(&head) ?					\
+		container_of(TAILQ_FIRST(&head), type, name) :	\
+		NULL)
+
+/**
+ * DLB2_LIST_FOR_EACH() - iterate over a list
+ * @head: list head
+ * @ptr: pointer to struct containing a struct list
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ */
+#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
+	TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
+
+/**
+ * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
+ * an element is removed from the list while processing it.
+ * @ptr: pointer to struct containing a struct list
+ * @ptr_tmp: pointer to struct containing a struct list (temporary)
+ * @head: list head
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ * @iter_tmp: iterator variable (temporary)
+ */
+#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
+	TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
+
+#endif /*  __DLB2_OSDEP_LIST_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
new file mode 100644
index 0000000..0a48f7e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_TYPES_H
+#define __DLB2_OSDEP_TYPES_H
+
+#include <linux/types.h>
+
+#include <inttypes.h>
+#include <ctype.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Types for user mode PF PMD */
+typedef uint8_t         u8;
+typedef int8_t          s8;
+typedef uint16_t        u16;
+typedef int16_t         s16;
+typedef uint32_t        u32;
+typedef int32_t         s32;
+typedef uint64_t        u64;
+
+#define __iomem
+
+/* END types for user mode PF PMD */
+
+#endif /* __DLB2_OSDEP_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
new file mode 100644
index 0000000..43ecad4
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
@@ -0,0 +1,2527 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "dlb2_osdep_types.h"
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+	struct {
+		u32 isr_pend : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+	struct {
+		u32 vf0_reset_in_progress : 1;
+		u32 vf1_reset_in_progress : 1;
+		u32 vf2_reset_in_progress : 1;
+		u32 vf3_reset_in_progress : 1;
+		u32 vf4_reset_in_progress : 1;
+		u32 vf5_reset_in_progress : 1;
+		u32 vf6_reset_in_progress : 1;
+		u32 vf7_reset_in_progress : 1;
+		u32 vf8_reset_in_progress : 1;
+		u32 vf9_reset_in_progress : 1;
+		u32 vf10_reset_in_progress : 1;
+		u32 vf11_reset_in_progress : 1;
+		u32 vf12_reset_in_progress : 1;
+		u32 vf13_reset_in_progress : 1;
+		u32 vf14_reset_in_progress : 1;
+		u32 vf15_reset_in_progress : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+	struct {
+		u32 vec_mask : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+	struct {
+		u32 func_vf_bar_dis : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+	struct {
+		u32 total_vas : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+	struct {
+		u32 total_dir_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+	struct {
+		u32 total_ldb_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+	struct {
+		u32 total_dir_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+	struct {
+		u32 total_ldb_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+	struct {
+		u32 total_dir_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+	struct {
+		u32 total_ldb_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 meas : 1;
+		u32 debug : 7;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 cq_int_rearm : 1;
+		u32 dsi_error : 1;
+		u32 rsvd0 : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 rsvd0 : 3;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+	struct {
+		u32 qid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+	struct {
+		u32 vqid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+	struct {
+		u32 qid : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 debug : 8;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 isz : 1;
+		u32 dsi_error : 1;
+		u32 dlbrsvd : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 vf_synd0_parity : 1;
+		u32 vf_synd1_parity : 1;
+		u32 vf_synd2_parity : 1;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+	struct {
+		u32 sn_cfg_v : 1;
+		u32 fid_cfg_v : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS  0
+#define DLB2_CQ_ISR_MODE_MSI  1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI  3
+union dlb2_sys_ldb_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+	struct {
+		u32 keep_pf_ppid : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+	struct {
+		u32 illegal_hcw : 1;
+		u32 illegal_pp : 1;
+		u32 illegal_pasid : 1;
+		u32 illegal_qid : 1;
+		u32 disabled_qid : 1;
+		u32 illegal_ldb_qid_cfg : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+	struct {
+		u32 msix_0_ack : 1;
+		u32 msix_1_ack : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+	struct {
+		u32 msix_0_passthru : 1;
+		u32 msix_1_passthru : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED     0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+	struct {
+		u32 mode : 1;
+		u32 poll_mode : 1;
+		u32 poll_mask : 1;
+		u32 poll_lock : 1;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+	struct {
+		u32 cq : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 alarm : 1;
+		u32 cwd : 1;
+		u32 vf_pf_mb : 1;
+		u32 rsvd0 : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+	struct {
+		u32 qid_fid_limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+	struct {
+		u32 compress_code : 3;
+		u32 rsvd0 : 29;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+	(DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+	struct {
+		u32 count : 15;
+		u32 rsvd0 : 17;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+	struct {
+		u32 sn : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+	struct {
+		u32 mode : 3;
+		u32 slot : 4;
+		u32 rsvz0 : 1;
+		u32 grp : 1;
+		u32 rsvz1 : 1;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+	struct {
+		u32 en : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+	struct {
+		u32 write_pointer : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+	struct {
+		u32 base : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+	struct {
+		u32 limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+	struct {
+		u32 pop_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+	struct {
+		u32 push_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+	struct {
+		u32 depth : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+	struct {
+		u32 write_pointer : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 dlb_cor_alarm_enable : 1;
+		u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+		u32 cfg_64bytes_qe_dir_cq_mode : 1;
+		u32 pad_write_ldb : 1;
+		u32 pad_write_dir : 1;
+		u32 pad_first_write_ldb : 1;
+		u32 pad_first_write_dir : 1;
+		u32 rsvz0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+	struct {
+		u32 egress_credit_status_empty : 1;
+		u32 egress_credit_status_afull : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+		u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+		u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+		u32 chp_rop_pipe_credit_status_empty : 1;
+		u32 chp_rop_pipe_credit_status_afull : 1;
+		u32 qed_to_cq_pipe_credit_status_empty : 1;
+		u32 qed_to_cq_pipe_credit_status_afull : 1;
+		u32 egress_lsp_token_credit_status_empty : 1;
+		u32 egress_lsp_token_credit_status_afull : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 rsvz0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+	struct {
+		u32 sn_mode_0 : 3;
+		u32 rszv0 : 5;
+		u32 sn_mode_1 : 3;
+		u32 rszv1 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+	struct {
+		u32 unit_single_step_mode : 1;
+		u32 rr_en : 1;
+		u32 rszv0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+	struct {
+		u32 prio : 24;
+		u32 v : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+	struct {
+		u32 qid_p0 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p1 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p2 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p3 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+	struct {
+		u32 qid_p4 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p5 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p6 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p7 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+	struct {
+		u32 token_depth_select : 4;
+		u32 disable_wb_opt : 1;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+	struct {
+		u32 token_count : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+	struct {
+		u32 thresh : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+	(DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+	(DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+	struct {
+		u32 depth : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+	struct {
+		u32 cq : 8;
+		u32 qidix : 3;
+		u32 value : 1;
+		u32 nalb_haswork_v : 1;
+		u32 rlist_haswork_v : 1;
+		u32 slist_haswork_v : 1;
+		u32 inflight_ok_v : 1;
+		u32 aqed_nfull_v : 1;
+		u32 rsvz0 : 15;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+	struct {
+		u32 transfer : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+	struct {
+		u32 bw_range : 9;
+		u32 rsvz0 : 22;
+		u32 no_extra_credit : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+	struct {
+		u32 disab_atq_empty_arb : 1;
+		u32 inc_tok_unit_idle : 1;
+		u32 disab_rlist_pri : 1;
+		u32 inc_cmp_unit_idle : 1;
+		u32 rsvz0 : 2;
+		u32 dir_single_op : 1;
+		u32 dir_half_bw : 1;
+		u32 dir_single_out : 1;
+		u32 dir_disab_multi : 1;
+		u32 atq_single_op : 1;
+		u32 atq_half_bw : 1;
+		u32 atq_single_out : 1;
+		u32 atq_disab_multi : 1;
+		u32 dirrpl_single_op : 1;
+		u32 dirrpl_half_bw : 1;
+		u32 dirrpl_single_out : 1;
+		u32 lbrpl_single_op : 1;
+		u32 lbrpl_half_bw : 1;
+		u32 lbrpl_single_out : 1;
+		u32 ldb_single_op : 1;
+		u32 ldb_half_bw : 1;
+		u32 ldb_disab_multi : 1;
+		u32 atm_single_sch : 1;
+		u32 atm_single_cmp : 1;
+		u32 ldb_ce_tog_arb : 1;
+		u32 rsvz1 : 1;
+		u32 smon0_valid_sel : 2;
+		u32 smon0_value_sel : 1;
+		u32 smon0_compare_sel : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+	struct {
+		u32 chp_pf_reset_done : 1;
+		u32 rop_pf_reset_done : 1;
+		u32 lsp_pf_reset_done : 1;
+		u32 nalb_pf_reset_done : 1;
+		u32 ap_pf_reset_done : 1;
+		u32 dp_pf_reset_done : 1;
+		u32 qed_pf_reset_done : 1;
+		u32 dqed_pf_reset_done : 1;
+		u32 aqed_pf_reset_done : 1;
+		u32 sys_pf_reset_done : 1;
+		u32 pf_reset_active : 1;
+		u32 flrsm_state : 7;
+		u32 rsvd0 : 13;
+		u32 dlb_proc_reset_done : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+	struct {
+		u32 chp_pipeidle : 1;
+		u32 rop_pipeidle : 1;
+		u32 lsp_pipeidle : 1;
+		u32 nalb_pipeidle : 1;
+		u32 ap_pipeidle : 1;
+		u32 dp_pipeidle : 1;
+		u32 qed_pipeidle : 1;
+		u32 dqed_pipeidle : 1;
+		u32 aqed_pipeidle : 1;
+		u32 sys_pipeidle : 1;
+		u32 chp_unit_idle : 1;
+		u32 rop_unit_idle : 1;
+		u32 lsp_unit_idle : 1;
+		u32 nalb_unit_idle : 1;
+		u32 ap_unit_idle : 1;
+		u32 dp_unit_idle : 1;
+		u32 qed_unit_idle : 1;
+		u32 dqed_unit_idle : 1;
+		u32 aqed_unit_idle : 1;
+		u32 sys_unit_idle : 1;
+		u32 rsvd1 : 4;
+		u32 mstr_cfg_ring_idle : 1;
+		u32 mstr_cfg_mstr_idle : 1;
+		u32 mstr_flr_clkreq_b : 1;
+		u32 mstr_proc_idle : 1;
+		u32 mstr_proc_idle_masked : 1;
+		u32 rsvd0 : 2;
+		u32 dlb_func_idle : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+	struct {
+		u32 prochot : 1;
+		u32 pgcb_dlb_idle : 1;
+		u32 pgcb_dlb_pg_rdy_ack_b : 1;
+		u32 pmsm_pgcb_req_b : 1;
+		u32 pgbc_pmc_pg_req_b : 1;
+		u32 pmc_pgcb_pg_ack_b : 1;
+		u32 pmc_pgcb_fet_en_b : 1;
+		u32 pgcb_fet_en_b : 1;
+		u32 rsvz0 : 1;
+		u32 rsvz1 : 1;
+		u32 fuse_force_on : 1;
+		u32 fuse_proc_disable : 1;
+		u32 rsvz2 : 1;
+		u32 rsvz3 : 1;
+		u32 pm_fsm_d0tod3_ok : 1;
+		u32 pm_fsm_d3tod0_ok : 1;
+		u32 dlb_in_d3 : 1;
+		u32 rsvz4 : 7;
+		u32 pmsm : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+	struct {
+		u32 disable : 1;
+		u32 rsvz0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+	struct {
+		u32 isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+	struct {
+		u32 pf_isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+	struct {
+		u32 isr_pend : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+	struct {
+		u32 reset_in_progress : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+	struct {
+		u32 vf_msi_isr : 32;
+	} field;
+	u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
new file mode 100644
index 0000000..503fdf3
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
@@ -0,0 +1,1913 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_osdep_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_start_domain() - start a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: start domain arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int dlb2_hw_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp,
+			 bool vdev_request,
+			 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_request,
+		    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding unmap procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_finish_map_qid_procedures() - finish any pending map procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding map procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ *					interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+			  int port_id,
+			  bool is_ldb,
+			  bool vdev_request,
+			  unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+					 u32 *ldb_interrupts,
+					 u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+				 u32 *ldb_interrupts,
+				 u32 *dir_interrupts);
+
+/**
+ * dlb2_read_vf_intr_status() - read the VF interrupt status register
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function can be called from a VF's interrupt handler to determine
+ * which interrupts have fired. The first 31 bits correspond to CQ interrupt
+ * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
+ *
+ * Return:
+ * Returns a bit vector indicating which interrupt vectors are active.
+ */
+u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_intr_status() - ack VF interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function ACKs a VF's interrupts. Its interrupts argument should be the
+ * value returned by dlb2_read_vf_intr_status().
+ */
+void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function clears the VF's MSI interrupt pending register. Its interrupts
+ * argument should be contain the MSI vectors to ACK. For example, if MSI MME
+ * is in mode 0, then one bit 0 should ever be set.
+ */
+void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When done processing the PF mailbox request, this function unsets
+ * the PF's mailbox ISR register.
+ */
+void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
+
+/**
+ * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
+ *					vdevs
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the vdev->PF ISR fires, this function can be called to determine which
+ * vdev(s) are requesting service. This bitvector must be passed to
+ * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
+ * vdevs.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested service.
+ */
+u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ *
+ * When done processing all VF mailbox requests, this function unsets the VF's
+ * mailbox ISR register.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
+ *				    FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the VF FLR ISR fires, this function can be called to determine which
+ * VF(s) are requesting FLRs. This bitvector must passed to
+ * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
+ */
+u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF FLR requests, this function unsets the VF's FLR
+ * ISR register.
+ */
+void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF requests, this function communicates to the
+ * hardware that processing is complete.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
+			     u32 mbox_bitvec,
+			     u32 flr_bitvec);
+
+/**
+ * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the watchdog timer interrupt cause registers to
+ * determine which port(s) had a watchdog timeout, and notifies the
+ * application(s) that own the port(s).
+ */
+void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads and logs the alarm syndrome, then acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_request,
+			      unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of resources in use by the PF or a VF. It
+ * fills in the fields that args points to, except the following:
+ * - max_contiguous_atomic_inflights
+ * - max_contiguous_hist_list_entries
+ * - max_contiguous_ldb_credits
+ * - max_contiguous_dir_credits
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
+				   struct dlb2_get_num_resources_args *arg,
+				   bool vdev_request,
+				   unsigned int vdev_id);
+
+/**
+ * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
+ *				       the PF
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function sends a VF->PF mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the PF has processed
+ * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
+ * when the PF has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
+ *				 request
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * processing a VF->PF mailbox request. It should only be called after sending
+ * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vf_flr_complete() - check the status of a VF FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * executing the VF FLR. It should only be called after setting the VF's FLR
+ * bit.
+ */
+bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
+ *					vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function sends a PF->vdev mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the vdev has
+ * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
+ * determine when the vdev has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
+ *			       request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function returns a boolean indicating whether the vdev has finished
+ * processing a PF->vdev mailbox request. It should only be called after
+ * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
+			     unsigned int vf_id,
+			     void *data,
+			     int len);
+
+/**
+ * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
+			       unsigned int vf_id,
+			       void *data,
+			       int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_reset_vdev() - reset the hardware owned by a virtual device
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function resets the hardware owned by a vdev, by resetting the vdev's
+ * domains one by one.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function returns whether or not the vdev's resource assignments are
+ * locked. If locked, no resources can be added to or subtracted from the
+ * group.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_lock_vdev() - lock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sets a flag indicating that the vdev is using its resources.
+ * When the vdev is locked, its resource assignment cannot be changed.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_unlock_vdev() - unlock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function unlocks the vdev's resource assignment, allowing it to be
+ * modified.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of scheduling domains to assign to this vdev
+ *
+ * This function assigns num scheduling domains to the specified vdev. If the
+ * vdev already has domains assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB queues to assign to this vdev
+ *
+ * This function assigns num LDB queues to the specified vdev. If the vdev
+ * already has LDB queues assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports to the specified vdev. If the vdev
+ * already has LDB ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @cos: class-of-service ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports from class-of-service cos to the
+ * specified vdev. If the vdev already has LDB ports from this class-of-service
+ * assigned, this existing assignment is adjusted accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
+				   u32 id,
+				   u32 cos,
+				   u32 num);
+
+/**
+ * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR ports to assign to this vdev
+ *
+ * This function assigns num DIR ports to the specified vdev. If the vdev
+ * already has DIR ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB credit credits to assign to this vdev
+ *
+ * This function assigns num LDB credit to the specified vdev. If the vdev
+ * already has LDB credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR credits to assign to this vdev
+ *
+ * This function assigns num DIR credit to the specified vdev. If the vdev
+ * already has DIR credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of history list entries to assign to this vdev
+ *
+ * This function assigns num history list entries to the specified vdev. If the
+ * vdev already has history list entries assigned, this existing assignment is
+ * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
+ * this function may fail if a sufficiently large contiguous chunk is not
+ * available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of atomic inflights to assign to this vdev
+ *
+ * This function assigns num atomic inflights to the specified vdev. If the vdev
+ * already has atomic inflights assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of entries, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function takes any resources currently assigned to the vdev and
+ * reassigns them to the PF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_notify_vf() - send an alarm to a VF
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID
+ * @notification: notification
+ *
+ * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
+ *
+ * Return:
+ * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
+ */
+int dlb2_notify_vf(struct dlb2_hw *hw,
+		   unsigned int vf_id,
+		   u32 notification);
+
+/**
+ * dlb2_vdev_in_use() - query whether a virtual device is in use
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sends a mailbox request to the vdev to query whether the vdev
+ * is in use.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
+ * an internal error occurs.
+ */
+int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+enum dlb2_virt_mode {
+	DLB2_VIRT_NONE,
+	DLB2_VIRT_SRIOV,
+	DLB2_VIRT_SIOV,
+
+	/* NUM_DLB2_VIRT_MODES must be last */
+	NUM_DLB2_VIRT_MODES,
+};
+
+/**
+ * dlb2_hw_set_virt_mode() - set the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: either none, SR-IOV, or Scalable IOV.
+ *
+ * This function sets the virtualization mode of the device. This controls
+ * whether the device uses a software or hardware mailbox.
+ *
+ * This should be called by the PF driver when either SR-IOV or Scalable IOV is
+ * selected as the virtualization mechanism, and by the VF/VDEV driver during
+ * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
+ *
+ * Errors:
+ * EINVAL - Invalid mode.
+ */
+int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
+
+/**
+ * dlb2_hw_get_virt_mode() - get the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function gets the virtualization mode of the device.
+ */
+enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_register_sw_mbox() - register a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
+ * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
+ * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
+ * @inject_arg: user argument for pf2vdev_inject callback.
+ *
+ * When Scalable IOV is enabled, the VDCM must register a software mailbox for
+ * every virtual device during vdev creation.
+ *
+ * This function notifies the driver to use a software mailbox using the
+ * provided pointers, instead of the device's hardware mailbox. When the driver
+ * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
+ * go to the software mailbox instead of the hardware one. This is used in
+ * Scalable IOV virtualization.
+ */
+void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
+			      unsigned int vdev_id,
+			      u32 *vdev2pf_mbox,
+			      u32 *pf2vdev_mbox,
+			      void (*pf2vdev_inject)(void *),
+			      void *inject_arg);
+
+/**
+ * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function notifies the driver to stop using a previously registered
+ * software mailbox.
+ */
+void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ * @addr_lo: least-significant 32 bits of address.
+ * @data: 32 data bits.
+ *
+ * This sets up the CQ's IMS entry with the provided address and data values.
+ * This function should only be called if the device is configured for Scalable
+ * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
+ * not needed.
+ */
+void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb,
+				u32 addr_lo,
+				u32 data);
+
+/**
+ * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ *
+ * This clears the CQ's IMS entry, reverting it to its reset state.
+ */
+void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb);
+
+/**
+ * dlb2_hw_register_pasid() - register a vdev's PASID
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @pasid: the vdev's PASID.
+ *
+ * This function stores the user-supplied PASID, and uses it when configuring
+ * the vdev's CQs.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+int dlb2_hw_register_pasid(struct dlb2_hw *hw,
+			   unsigned int vdev_id,
+			   unsigned int pasid);
+
+/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vf_request,
+				unsigned int vf_id);
+
+/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ *	to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ *	port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ *	    cause the total bandwidth across all classes of service to exceed
+ *	    100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+enum dlb2_wd_tmo {
+	/* 40s watchdog timeout */
+	DLB2_WD_TMO_40S,
+	/* 10s watchdog timeout */
+	DLB2_WD_TMO_10S,
+	/* 1s watchdog timeout */
+	DLB2_WD_TMO_1S,
+
+	/* Must be last */
+	NUM_DLB2_WD_TMOS,
+};
+
+/**
+ * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
+ *	caller-specified timeout.
+ * @hw: dlb2_hw handle for a particular device.
+ * @tmo: watchdog timeout.
+ *
+ * This function should be called during device initialization and after reset.
+ * The watchdog timer interrupt must also be enabled per-CQ, using either
+ * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid timeout.
+ */
+int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
+
+/**
+ * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid directed port ID.
+ */
+int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid load-balanced port ID.
+ */
+int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *	ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * load-balanced CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_ldb_cq_interrupt().
+ */
+void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical directed port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * directed CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_dir_cq_interrupt().
+ */
+void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+#endif /* __DLB2_RESOURCE_H */
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
new file mode 100644
index 0000000..3c51cab
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -0,0 +1,621 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "base/dlb2_resource.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_regs.h"
+#include "dlb2_main.h"
+#include "../dlb2_user.h"
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+
+#define PF_ID_ZERO 0	/* PF ONLY! */
+#define NO_OWNER_VF 0	/* PF ONLY! */
+#define NOT_VF_REQ false /* PF ONLY! */
+
+#define DLB2_PCI_CFG_SPACE_SIZE 256
+#define DLB2_PCI_CAP_POINTER 0x34
+#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
+#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
+#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
+#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
+#define DLB2_PCI_EXT_CAP_ID_ERR 1
+#define DLB2_PCI_ERR_UNCOR_MASK 8
+#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
+
+#define DLB2_PCI_EXP_DEVCTL 8
+#define DLB2_PCI_LNKCTL 16
+#define DLB2_PCI_SLTCTL 24
+#define DLB2_PCI_RTCTL 28
+#define DLB2_PCI_EXP_DEVCTL2 40
+#define DLB2_PCI_LNKCTL2 48
+#define DLB2_PCI_SLTCTL2 56
+#define DLB2_PCI_CMD 4
+#define DLB2_PCI_X_CMD 2
+#define DLB2_PCI_EXP_DEVSTA 10
+#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
+#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
+
+#define DLB2_PCI_CAP_ID_EXP       0x10
+#define DLB2_PCI_CAP_ID_MSIX      0x11
+#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
+#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
+#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
+
+#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
+#define DLB2_PCI_PRI_ALLOC_REQ           0xC
+#define DLB2_PCI_PRI_CTRL                0x4
+#define DLB2_PCI_MSIX_FLAGS              0x2
+#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
+#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
+#define DLB2_PCI_ERR_ROOT_STATUS         0x30
+#define DLB2_PCI_ERR_COR_STATUS          0x10
+#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
+#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
+#define DLB2_PCI_ACS_CAP                 0x4
+#define DLB2_PCI_ACS_CTRL                0x6
+#define DLB2_PCI_ACS_SV                  0x1
+#define DLB2_PCI_ACS_RR                  0x4
+#define DLB2_PCI_ACS_CR                  0x8
+#define DLB2_PCI_ACS_UF                  0x10
+#define DLB2_PCI_ACS_EC                  0x20
+
+/* Stubs: Allow building partial probe patch */
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	int ret = 0;
+	RTE_SET_USED(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+/* End stubs */
+
+static int
+dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint32_t hdr;
+	size_t sz;
+	int pos;
+
+	pos = DLB2_PCI_CFG_SPACE_SIZE;
+	sz = sizeof(hdr);
+
+	while (pos > 0xFF) {
+		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
+			return -1;
+
+		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
+			return pos;
+
+		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint8_t pos;
+	int ret;
+	uint16_t hdr;
+
+	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
+	pos &= 0xFC;
+
+	if (ret != 1)
+		return -1;
+
+	while (pos > 0x3F) {
+		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
+		if (ret != 2)
+			return -1;
+
+		if (DLB2_PCI_CAP_ID(hdr) == id)
+			return pos;
+
+		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
+			return -1;
+
+		pos = DLB2_PCI_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+	rte_spinlock_init(&dlb2_dev->resource_mutex);
+	rte_spinlock_init(&dlb2_dev->measurement_lock);
+
+	return 0;
+}
+
+static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
+{
+	u32 retries = 0;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
+		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+		union dlb2_cfg_mstr_cfg_pm_status pm_st;
+		u32 addr;
+
+		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+			break;
+
+		rte_delay_ms(1);
+	};
+
+	if (retries == DLB2_READY_RETRY_LIMIT) {
+		printf("[%s()] wait for device ready timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+struct dlb2_dev *
+dlb2_probe(struct rte_pci_device *pdev)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret = 0;
+
+	DLB2_INFO(dlb2_dev, "probe\n");
+
+	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
+			      RTE_CACHE_LINE_SIZE);
+
+	if (dlb2_dev == NULL) {
+		ret = -ENOMEM;
+		goto dlb2_dev_malloc_fail;
+	}
+
+	/* PCI Bus driver has already mapped bar space into process.
+	 * Save off our IO register and FUNC addresses.
+	 */
+
+	/* BAR 0 */
+	if (pdev->mem_resource[0].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
+	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.func_kva,
+		  (void *)dlb2_dev->hw.func_phys_addr,
+		  (void *)(pdev->mem_resource[0].len));
+
+	/* BAR 2 */
+	if (pdev->mem_resource[2].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
+	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.csr_kva,
+		  (void *)dlb2_dev->hw.csr_phys_addr,
+		  (void *)(pdev->mem_resource[2].len));
+
+	dlb2_dev->pdev = pdev;
+
+	/* PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb2_pf_enable_pm(dlb2_dev);
+
+	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb2_pf_reset(dlb2_dev);
+	if (ret)
+		goto dlb2_reset_fail;
+
+	ret = dlb2_pf_init_driver_state(dlb2_dev);
+	if (ret)
+		goto init_driver_state_fail;
+
+	ret = dlb2_resource_init(&dlb2_dev->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	return dlb2_dev;
+
+resource_init_fail:
+	dlb2_resource_free(&dlb2_dev->hw);
+init_driver_state_fail:
+dlb2_reset_fail:
+pci_mmap_bad_addr:
+wait_for_device_ready_fail:
+	rte_free(dlb2_dev);
+dlb2_dev_malloc_fail:
+	rte_errno = ret;
+	return NULL;
+}
+
+int
+dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
+{
+	int ret = 0;
+	int i = 0;
+	uint32_t dword[16];
+	uint16_t cmd;
+	off_t off;
+
+	uint16_t dev_ctl_word;
+	uint16_t dev_ctl2_word;
+	uint16_t lnk_word;
+	uint16_t lnk_word2;
+	uint16_t slt_word;
+	uint16_t slt_word2;
+	uint16_t rt_ctl_word;
+	uint32_t pri_reqs_dword;
+	uint16_t pri_ctrl_word;
+
+	int pcie_cap_offset;
+	int pri_cap_offset;
+	int msix_cap_offset;
+	int err_cap_offset;
+	int acs_cap_offset;
+	int wait_count;
+
+	uint16_t devsta_busy_word;
+	uint16_t devctl_word;
+
+	struct rte_pci_device *pdev = dlb2_dev->pdev;
+
+	/* Save PCI config state */
+
+	for (i = 0; i < 16; i++) {
+		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
+			return ret;
+	}
+
+	pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
+
+	if (pcie_cap_offset < 0) {
+		printf("[%s()] failed to find the pcie capability\n",
+		       __func__);
+		return pcie_cap_offset;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
+		dev_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
+		lnk_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
+		slt_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_RTCTL;
+	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
+		rt_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
+		dev_ctl2_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
+		lnk_word2 = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
+		slt_word2 = 0;
+
+	off = DLB2_PCI_EXT_CAP_ID_PRI;
+	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (pri_cap_offset >= 0) {
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
+			pri_reqs_dword = 0;
+	}
+
+	/* clear the PCI command register before issuing the FLR */
+
+	off = DLB2_PCI_CMD;
+	cmd = 0;
+	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+		printf("[%s()] failed to write the pci command\n",
+		       __func__);
+		return ret;
+	}
+
+	/* issue the FLR */
+	for (wait_count = 0; wait_count < 4; wait_count++) {
+		int sleep_time;
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
+		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to read the pci device status\n",
+			       __func__);
+			return ret;
+		}
+
+		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
+			break;
+
+		sleep_time = (1 << (wait_count)) * 100;
+		rte_delay_ms(sleep_time);
+	}
+
+	if (wait_count == 4) {
+		printf("[%s()] wait for pci pending transactions timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to read the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
+
+	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to write the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	rte_delay_ms(100);
+
+	/* Restore PCI config state */
+
+	if (pcie_cap_offset >= 0) {
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie device control at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_RTCTL;
+		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	if (pri_cap_offset >= 0) {
+		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
+
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
+		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ERR;
+	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (err_cap_offset >= 0) {
+		uint32_t tmp;
+
+		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	for (i = 16; i > 0; i--) {
+		off = (i - 1) * 4;
+		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_CMD;
+	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
+		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+			printf("[%s()] failed to write the pci command\n",
+			       __func__);
+			return ret;
+		}
+	}
+
+	msix_cap_offset = dlb2_pci_find_capability(pdev,
+						   DLB2_PCI_CAP_ID_MSIX);
+	if (msix_cap_offset >= 0) {
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
+			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ACS;
+	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (acs_cap_offset >= 0) {
+		uint16_t acs_cap, acs_ctrl, acs_mask;
+		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
+		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
+			acs_cap = 0;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
+		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
+		acs_ctrl |= (acs_cap & acs_mask);
+
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
+		acs_mask |= DLB2_PCI_ACS_EC;
+		acs_ctrl &= ~acs_mask;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
new file mode 100644
index 0000000..f082a94
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -0,0 +1,98 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+#endif
+
+#include "base/dlb2_hw_types.h"
+#include "../dlb2_user.h"
+
+#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
+
+struct dlb2_dev;
+
+struct dlb2_port_memory {
+	struct dlb2_list_head list;
+	void *cq_base;
+	bool valid;
+};
+
+struct dlb2_dev {
+	struct rte_pci_device *pdev;
+	struct dlb2_hw hw;
+	/* struct list_head list; */
+	struct device *dlb2_device;
+	bool domain_reset_failed;
+	/* The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	rte_spinlock_t resource_mutex;
+	rte_spinlock_t measurement_lock;
+	bool worker_launched;
+	u8 revision;
+};
+
+struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
+int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
+int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+
+#endif /* __DLB2_MAIN_H */
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
new file mode 100644
index 0000000..ae4d785
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -0,0 +1,196 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_devargs.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_pci.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+#include "dlb2_main.h"
+#include "base/dlb2_hw_types.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_resource.h"
+
+static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
+
+/* Stubs: Allow building partial probe patch */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(arg);
+	RTE_SET_USED(vdev_req);
+	RTE_SET_USED(vdev_id);
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+/* End stubs */
+
+static void
+dlb2_pf_iface_fn_ptrs_init(void)
+{
+/* flexible iface fn ptr assignments will go here */
+}
+
+/* PCI DEV HOOKS */
+static int
+dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
+{
+	int ret = 0;
+	struct rte_pci_device *pci_dev;
+	struct dlb2_devargs dlb2_args = {
+		.socket_id = rte_socket_id(),
+		.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+		.num_dir_credits_override = -1,
+		.qid_depth_thresholds = { {0} },
+		.cos_id = DLB2_COS_DEFAULT
+	};
+	struct dlb2_eventdev *dlb2;
+
+	DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
+		     eventdev->data->dev_id, eventdev->data->socket_id);
+
+	dlb2_pf_iface_fn_ptrs_init();
+
+	pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
+
+		/* Probe the DLB2 PF layer */
+		dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
+
+		if (dlb2->qm_instance.pf_dev == NULL) {
+			DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
+				     rte_errno);
+			ret = -rte_errno;
+			goto dlb2_probe_failed;
+		}
+
+		/* Were we invoked with runtime parameters? */
+		if (pci_dev->device.devargs) {
+			ret = dlb2_parse_params(pci_dev->device.devargs->args,
+						pci_dev->device.devargs->name,
+						&dlb2_args);
+			if (ret) {
+				DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
+					     ret, rte_errno);
+				goto dlb2_probe_failed;
+			}
+		}
+
+		ret = dlb2_primary_eventdev_probe(eventdev,
+						  event_dlb2_pf_name,
+						  &dlb2_args);
+	} else {
+		ret = dlb2_secondary_eventdev_probe(eventdev,
+						    event_dlb2_pf_name);
+	}
+	if (ret)
+		goto dlb2_probe_failed;
+
+	DLB2_LOG_INFO("DLB2 PF Probe success\n");
+
+	return 0;
+
+dlb2_probe_failed:
+
+	DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
+
+	return ret;
+}
+
+#define EVENTDEV_INTEL_VENDOR_ID 0x8086
+
+static const struct rte_pci_id pci_id_dlb2_map[] = {
+	{
+		RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
+			       PCI_DEVICE_ID_INTEL_DLB2_PF)
+	},
+	{
+		.vendor_id = 0,
+	},
+};
+
+static int
+event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
+		     struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
+					     sizeof(struct dlb2_eventdev),
+					     dlb2_eventdev_pci_init,
+					     event_dlb2_pf_name);
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+}
+
+static int
+event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_remove(pci_dev, NULL);
+
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+
+}
+
+static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
+	.id_table = pci_id_dlb2_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe = event_dlb2_pci_probe,
+	.remove = event_dlb2_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
-- 
2.6.4

Re: [dpdk-dev] [PATCH v6 06/23] event/dlb2: add eventdev probe

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, October 30, 2020 1:29 PM
> To: Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com; thomas@monjalon.net
> Subject: [PATCH v6 06/23] event/dlb2: add eventdev probe
> 
> Add the eventdev portion of probe, and parse command line
> options, but do not initialize hardware.
> 
> Changes since v2 patch-set probe:
> 
> Primary and secondary probe-time init has been removed, and
> will be introduced in subsequent patches contained in
> the v3 patch-set.
> 
> Hardware init has been moved to a subsequent patch in order to
> minimize the patch size. The files dlb2/pf/dlb2_pf.c and
> dlb2/pf/dlb_main.c include stubs of hardware init functions,
> and these will be replaced with the real versions when the
> hardware layer is committed.
> 
> Initialization of the flexible interface layer has been moved to
> a subsequent patch in order to minimize patch size.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

If there's a next version, please put the 'Changes since' info below the '---'
line, so it's not included in the commit message. (Otherwise this can get fixed
during merge.)

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v6 07/23] event/dlb2: add flexible interface

[Timothy McDaniel]

This commit introduces the flexible interface. This
interface allows the core code to operate in PF mode (direct
hardware access) or bifurcated mode (hardware configured via
kernel driver). This driver currently only supports PF mode
but bifurcated mode will be added in a future DPDK patch-set.
Note that the flexible interface is not used for data path
operations, and thus there are no performance concerns
related to the use of function pointers.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_iface.c | 28 ++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.h | 29 +++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build  |  1 +
 3 files changed, 58 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h

diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
new file mode 100644
index 0000000..0fc9991
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+
+#include "dlb2_priv.h"
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+void (*dlb2_iface_low_level_io_init)(void);
+
+int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+				     uint8_t *revision);
+
+void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+				   enum dlb2_cq_poll_modes *mode);
+
+int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
new file mode 100644
index 0000000..4fb416e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_IFACE_H_
+#define _DLB2_IFACE_H_
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+extern void (*dlb2_iface_low_level_io_init)(void);
+
+extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+					    uint8_t *revision);
+
+extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+					  enum dlb2_cq_poll_modes *mode);
+
+extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
+
+#endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 1eab9b9..491e76d 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -8,6 +8,7 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
 endif
 
 sources = files('dlb2.c',
+		'dlb2_iface.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c'
 )
-- 
2.6.4

Re: [dpdk-dev] [PATCH v6 07/23] event/dlb2: add flexible interface

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, October 30, 2020 1:29 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com; thomas@monjalon.net
> Subject: [PATCH v6 07/23] event/dlb2: add flexible interface
> 
> This commit introduces the flexible interface. This
> interface allows the core code to operate in PF mode (direct
> hardware access) or bifurcated mode (hardware configured via
> kernel driver). This driver currently only supports PF mode
> but bifurcated mode will be added in a future DPDK patch-set.
> Note that the flexible interface is not used for data path
> operations, and thus there are no performance concerns
> related to the use of function pointers.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v6 08/23] event/dlb2: add probe-time hardware init

[Timothy McDaniel]

This commit adds probe-time low level hardware
initialization.  It also adds probe-time init for both
primary and secondary DPDK processes.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 188 +++++++++++++++++++-
 drivers/event/dlb2/meson.build             |   3 +-
 drivers/event/dlb2/pf/base/dlb2_resource.c | 274 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  23 +--
 drivers/event/dlb2/pf/dlb2_main.h          |   1 -
 drivers/event/dlb2/pf/dlb2_pf.c            |  78 ++++++--
 6 files changed, 523 insertions(+), 44 deletions(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 1b11979..16653db 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -31,6 +31,7 @@
 #include <rte_string_fns.h>
 
 #include "dlb2_priv.h"
+#include "dlb2_iface.h"
 #include "dlb2_inline_fns.h"
 
 /*
@@ -40,10 +41,108 @@
 #if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
 #error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
 #endif
+static struct rte_event_dev_info evdev_dlb2_default_info = {
+	.driver_name = "", /* probe will set */
+	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
+	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
+	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
+#else
+	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
+#endif
+	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
+	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
+	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
+	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
+	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
+	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+	.max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
+	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
+			  RTE_EVENT_DEV_CAP_EVENT_QOS |
+			  RTE_EVENT_DEV_CAP_BURST_MODE |
+			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
+			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
+			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
+};
 
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/* override defaults with value(s) provided on command line */
+static void
+dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
+				 int *qid_depth_thresholds)
+{
+	int q;
+
+	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
+		if (qid_depth_thresholds[q] != 0)
+			dlb2->ev_queues[q].depth_threshold =
+				qid_depth_thresholds[q];
+	}
+}
+
+static int
+dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
+	int ret;
+
+	/* Query driver resources provisioned for this device */
+
+	ret = dlb2_iface_get_num_resources(handle,
+					   &dlb2->hw_rsrc_query_results);
+	if (ret) {
+		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
+		return ret;
+	}
+
+	/* Complete filling in device resource info returned to evdev app,
+	 * overriding any default values.
+	 * The capabilities (CAPs) were set at compile time.
+	 */
+
+	evdev_dlb2_default_info.max_event_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	evdev_dlb2_default_info.max_event_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	evdev_dlb2_default_info.max_num_events =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	/* Save off values used when creating the scheduling domain. */
+
+	handle->info.num_sched_domains =
+		dlb2->hw_rsrc_query_results.num_sched_domains;
+
+	handle->info.hw_rsrc_max.nb_events_limit =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	handle->info.hw_rsrc_max.num_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues +
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.num_ldb_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	handle->info.hw_rsrc_max.num_ldb_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	handle->info.hw_rsrc_max.num_dir_ports =
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.reorder_window_size =
+		dlb2->hw_rsrc_query_results.num_hist_list_entries;
+
+	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
+
+	return 0;
+}
+
 #define DLB2_BASE_10 10
 
 static int
@@ -235,14 +334,71 @@ set_qid_depth_thresh(const char *key __rte_unused,
 	return 0;
 }
 
+static void
+dlb2_entry_points_init(struct rte_eventdev *dev)
+{
+	RTE_SET_USED(dev);
+
+	/* Eventdev PMD entry points */
+}
+
 int
 dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    const char *name,
 			    struct dlb2_devargs *dlb2_args)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
-	RTE_SET_USED(dlb2_args);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	dlb2->event_dev = dev; /* backlink */
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	dlb2->max_num_events_override = dlb2_args->max_num_events;
+	dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
+	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
+					    &dlb2->revision);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_hardware_init(&dlb2->qm_instance);
+
+	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
+			     err);
+		return err;
+	}
+
+	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	dlb2_init_queue_depth_thresholds(dlb2,
+					 dlb2_args->qid_depth_thresholds.val);
 
 	return 0;
 }
@@ -251,8 +407,30 @@ int
 dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
 			      const char *name)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 491e76d..28fef3f 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -10,7 +10,8 @@ endif
 sources = files('dlb2.c',
 		'dlb2_iface.c',
 		'pf/dlb2_main.c',
-		'pf/dlb2_pf.c'
+		'pf/dlb2_pf.c',
+		'pf/base/dlb2_resource.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
new file mode 100644
index 0000000..6de8b95
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_mbox.h"
+#include "dlb2_osdep.h"
+#include "dlb2_osdep_bitmap.h"
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+	int i;
+
+	dlb2_list_init_head(&domain->used_ldb_queues);
+	dlb2_list_init_head(&domain->used_dir_pq_pairs);
+	dlb2_list_init_head(&domain->avail_ldb_queues);
+	dlb2_list_init_head(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+	int i;
+
+	dlb2_list_init_head(&rsrc->avail_domains);
+	dlb2_list_init_head(&rsrc->used_domains);
+	dlb2_list_init_head(&rsrc->avail_ldb_queues);
+	dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
+	arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
+	arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
+	arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb2_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	struct dlb2_list_entry *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. This is application
+	 * dependent, but the driver interleaves port IDs as much as possible
+	 * to reduce the likelihood of this. This initial allocation maximizes
+	 * the average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	/* Zero-out resource tracking data structures */
+	memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+	memset(&hw->pf, 0, sizeof(hw->pf));
+
+	dlb2_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+		dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+		dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_domains, list);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_ldb_queues, list);
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+		struct dlb2_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
+			      &port->func_list);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+				DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+	if (ret)
+		goto unwind;
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+					DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+		if (ret)
+			goto unwind;
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb2_resource_free(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+	r0.field.disable = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 3c51cab..210b3186 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -19,6 +19,7 @@
 #include "dlb2_main.h"
 #include "../dlb2_user.h"
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 
 #define PF_ID_ZERO 0	/* PF ONLY! */
@@ -72,27 +73,6 @@
 #define DLB2_PCI_ACS_UF                  0x10
 #define DLB2_PCI_ACS_EC                  0x20
 
-/* Stubs: Allow building partial probe patch */
-void dlb2_resource_free(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-int dlb2_resource_init(struct dlb2_hw *hw)
-{
-	int ret = 0;
-	RTE_SET_USED(hw);
-
-	return ret;
-}
-
-void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-/* End stubs */
-
 static int
 dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
 {
@@ -149,7 +129,6 @@ static int
 dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
 {
 	rte_spinlock_init(&dlb2_dev->resource_mutex);
-	rte_spinlock_init(&dlb2_dev->measurement_lock);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index f082a94..f3bee71 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -38,7 +38,6 @@ struct dlb2_dev {
 	 * hardware registers.
 	 */
 	rte_spinlock_t resource_mutex;
-	rte_spinlock_t measurement_lock;
 	bool worker_launched;
 	u8 revision;
 };
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index ae4d785..7d64309 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -32,6 +32,7 @@
 #include <rte_string_fns.h>
 
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 #include "dlb2_main.h"
 #include "base/dlb2_hw_types.h"
@@ -40,35 +41,82 @@
 
 static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
 
-/* Stubs: Allow building partial probe patch */
-int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
-			      struct dlb2_get_num_resources_args *arg,
-			      bool vdev_req,
-			      unsigned int vdev_id)
+static void
+dlb2_pf_low_level_io_init(void)
+{
+	int i;
+	/* Addresses will be initialized at port create */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		/* First directed ports */
+		dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
+
+		/* Now load balanced ports */
+		dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
+	}
+}
+
+static int
+dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
+{
+	RTE_SET_USED(handle);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+static int
+dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
+			   uint8_t *revision)
 {
-	RTE_SET_USED(hw);
-	RTE_SET_USED(arg);
-	RTE_SET_USED(vdev_req);
-	RTE_SET_USED(vdev_id);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	*revision = dlb2_dev->revision;
 
 	return 0;
 }
 
-void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+static void
+dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+	dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_num_resources_args *rsrcs)
 {
-	RTE_SET_USED(hw);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
 }
 
-void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+static int
+dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
+			 enum dlb2_cq_poll_modes *mode)
 {
-	RTE_SET_USED(hw);
+	RTE_SET_USED(handle);
+
+	*mode = DLB2_CQ_POLL_MODE_SPARSE;
+
+	return 0;
 }
-/* End stubs */
 
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
-/* flexible iface fn ptr assignments will go here */
+	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
+	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
+	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
+	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
+	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

Re: [dpdk-dev] [PATCH v6 08/23] event/dlb2: add probe-time hardware init

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, October 30, 2020 1:29 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com; thomas@monjalon.net
> Subject: [PATCH v6 08/23] event/dlb2: add probe-time hardware init
> 
> This commit adds probe-time low level hardware
> initialization.  It also adds probe-time init for both
> primary and secondary DPDK processes.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v6 09/23] event/dlb2: add xstats

[Timothy McDaniel]

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
Reviewed-by: Chen, Mike Ximing <mike.ximing.chen@intel.com>
---
 drivers/event/dlb2/dlb2.c        |   35 +-
 drivers/event/dlb2/dlb2_xstats.c | 1240 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build   |    1 +
 3 files changed, 1273 insertions(+), 3 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 16653db..925824e 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,6 +70,21 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/*
+ * DUMMY - added so that xstats path will compile/link.
+ * Will be replaced by real version in a subsequent
+ * patch.
+ */
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	RTE_SET_USED(dlb2);
+	RTE_SET_USED(queue);
+
+	return 0;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -337,9 +352,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
-	RTE_SET_USED(dev);
-
-	/* Eventdev PMD entry points */
+	/* Expose PMD's eventdev interface */
+	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dump             = dlb2_eventdev_dump,
+		.xstats_get       = dlb2_eventdev_xstats_get,
+		.xstats_get_names = dlb2_eventdev_xstats_get_names,
+		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
+		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+	};
+
+	dev->dev_ops = &dlb2_eventdev_entry_ops;
 }
 
 int
@@ -391,6 +413,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Complete xtstats runtime initialization */
+	err = dlb2_xstats_init(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
+		return err;
+	}
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
new file mode 100644
index 0000000..21391f7
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_xstats.c
@@ -0,0 +1,1240 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eventdev.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+enum dlb2_xstats_type {
+	/* common to device and port */
+	rx_ok,				/**< Receive an event */
+	rx_drop,                        /**< Error bit set in received QE */
+	rx_interrupt_wait,		/**< Wait on an interrupt */
+	rx_umonitor_umwait,		/**< Block using umwait */
+	tx_ok,				/**< Transmit an event */
+	total_polls,			/**< Call dequeue_burst */
+	zero_polls,			/**< Call dequeue burst and return 0 */
+	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
+	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
+	tx_nospc_inflight_max,		/**< Reach the new_event_threshold */
+	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
+	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
+	/* device specific */
+	nb_events_limit,
+	inflight_events,
+	ldb_pool_size,
+	dir_pool_size,
+	/* port specific */
+	tx_new,				/**< Send an OP_NEW event */
+	tx_fwd,				/**< Send an OP_FORWARD event */
+	tx_rel,				/**< Send an OP_RELEASE event */
+	tx_implicit_rel,		/**< Issue an implicit event release */
+	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED event */
+	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL event */
+	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event */
+	tx_sched_directed,		/**< Send a directed event */
+	tx_invalid,                     /**< Send an event with an invalid op */
+	outstanding_releases,		/**< # of releases a port owes */
+	max_outstanding_releases,	/**< max # of releases a port can owe */
+	rx_sched_ordered,		/**< Dequeue an ordered event */
+	rx_sched_unordered,		/**< Dequeue an unordered event */
+	rx_sched_atomic,		/**< Dequeue an atomic event */
+	rx_sched_directed,		/**< Dequeue an directed event */
+	rx_sched_invalid,               /**< Dequeue event sched type invalid */
+	/* common to port and queue */
+	is_configured,			/**< Port is configured */
+	is_load_balanced,		/**< Port is LDB */
+	hw_id,				/**< Hardware ID */
+	/* queue specific */
+	num_links,			/**< Number of ports linked */
+	sched_type,			/**< Queue sched type */
+	enq_ok,				/**< # events enqueued to the queue */
+	current_depth,			/**< Current queue depth */
+	depth_threshold,		/**< Programmed depth threshold */
+	depth_le50_threshold,
+	/**< Depth LE to 50% of the configured hardware threshold */
+	depth_gt50_le75_threshold,
+	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold */
+	depth_gt75_le100_threshold,
+	/**< Depth GT 75%. but LE to the configured hardware threshold */
+	depth_gt100_threshold
+	/**< Depth GT 100% of the configured hw threshold */
+};
+
+typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
+		uint16_t obj_idx, /* port or queue id */
+		enum dlb2_xstats_type stat, int extra_arg);
+
+enum dlb2_xstats_fn_type {
+	DLB2_XSTATS_FN_DEV,
+	DLB2_XSTATS_FN_PORT,
+	DLB2_XSTATS_FN_QUEUE
+};
+
+struct dlb2_xstats_entry {
+	struct rte_event_dev_xstats_name name;
+	uint64_t reset_value; /* an offset to be taken away to emulate resets */
+	enum dlb2_xstats_fn_type fn_id;
+	enum dlb2_xstats_type stat;
+	enum rte_event_dev_xstats_mode mode;
+	int extra_arg;
+	uint16_t obj_idx;
+	uint8_t reset_allowed; /* when set, this value can be reset */
+};
+
+/* Some device stats are simply a summation of the corresponding port values */
+static uint64_t
+dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
+			     int which_stat)
+{
+	int i;
+	uint64_t val = 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
+
+		if (!port->setup_done)
+			continue;
+
+		switch (which_stat) {
+		case rx_ok:
+			val += port->stats.traffic.rx_ok;
+			break;
+		case rx_drop:
+			val += port->stats.traffic.rx_drop;
+			break;
+		case rx_interrupt_wait:
+			val += port->stats.traffic.rx_interrupt_wait;
+			break;
+		case rx_umonitor_umwait:
+			val += port->stats.traffic.rx_umonitor_umwait;
+			break;
+		case tx_ok:
+			val += port->stats.traffic.tx_ok;
+			break;
+		case total_polls:
+			val += port->stats.traffic.total_polls;
+			break;
+		case zero_polls:
+			val += port->stats.traffic.zero_polls;
+			break;
+		case tx_nospc_ldb_hw_credits:
+			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
+			break;
+		case tx_nospc_dir_hw_credits:
+			val += port->stats.traffic.tx_nospc_dir_hw_credits;
+			break;
+		case tx_nospc_inflight_max:
+			val += port->stats.traffic.tx_nospc_inflight_max;
+			break;
+		case tx_nospc_new_event_limit:
+			val += port->stats.traffic.tx_nospc_new_event_limit;
+			break;
+		case tx_nospc_inflight_credits:
+			val += port->stats.traffic.tx_nospc_inflight_credits;
+			break;
+		default:
+			return -1;
+		}
+	}
+	return val;
+}
+
+static uint64_t
+get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
+	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	switch (type) {
+	case rx_ok:
+	case rx_drop:
+	case rx_interrupt_wait:
+	case rx_umonitor_umwait:
+	case tx_ok:
+	case total_polls:
+	case zero_polls:
+	case tx_nospc_ldb_hw_credits:
+	case tx_nospc_dir_hw_credits:
+	case tx_nospc_inflight_max:
+	case tx_nospc_new_event_limit:
+	case tx_nospc_inflight_credits:
+		return dlb2_device_traffic_stat_get(dlb2, type);
+	case nb_events_limit:
+		return dlb2->new_event_limit;
+	case inflight_events:
+		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
+	case ldb_pool_size:
+		return dlb2->num_ldb_credits;
+	case dir_pool_size:
+		return dlb2->num_dir_credits;
+	default: return -1;
+	}
+}
+
+static uint64_t
+get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
+
+	switch (type) {
+	case rx_ok: return ev_port->stats.traffic.rx_ok;
+
+	case rx_drop: return ev_port->stats.traffic.rx_drop;
+
+	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
+
+	case rx_umonitor_umwait:
+		return ev_port->stats.traffic.rx_umonitor_umwait;
+
+	case tx_ok: return ev_port->stats.traffic.tx_ok;
+
+	case total_polls: return ev_port->stats.traffic.total_polls;
+
+	case zero_polls: return ev_port->stats.traffic.zero_polls;
+
+	case tx_nospc_ldb_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
+
+	case tx_nospc_dir_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
+
+	case tx_nospc_inflight_max:
+		return ev_port->stats.traffic.tx_nospc_inflight_max;
+
+	case tx_nospc_new_event_limit:
+		return ev_port->stats.traffic.tx_nospc_new_event_limit;
+
+	case tx_nospc_inflight_credits:
+		return ev_port->stats.traffic.tx_nospc_inflight_credits;
+
+	case is_configured: return ev_port->setup_done;
+
+	case is_load_balanced: return !ev_port->qm_port.is_directed;
+
+	case hw_id: return ev_port->qm_port.id;
+
+	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
+
+	case tx_fwd: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
+
+	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
+
+	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
+
+	case tx_sched_ordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case tx_sched_unordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case tx_sched_atomic:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case tx_sched_directed:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case tx_invalid: return ev_port->stats.tx_invalid;
+
+	case outstanding_releases: return ev_port->outstanding_releases;
+
+	case max_outstanding_releases:
+		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	case rx_sched_ordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case rx_sched_unordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case rx_sched_atomic:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case rx_sched_directed:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
+
+	default: return -1;
+	}
+}
+
+static uint64_t
+dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
+{
+	int port = 0;
+	uint64_t tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
+
+	return tally;
+}
+
+static uint64_t
+dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
+{
+	int port = 0;
+	uint64_t enq_ok_tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
+
+	return enq_ok_tally;
+}
+
+static uint64_t
+get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_queue *ev_queue =
+		&dlb2->ev_queues[obj_idx];
+
+	switch (type) {
+	case is_configured: return ev_queue->setup_done;
+
+	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
+
+	case hw_id: return ev_queue->qm_queue.id;
+
+	case num_links: return ev_queue->num_links;
+
+	case sched_type: return ev_queue->qm_queue.sched_type;
+
+	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
+
+	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
+
+	case depth_threshold: return ev_queue->depth_threshold;
+
+	case depth_le50_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_LE50);
+
+	case depth_gt50_le75_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT50_LE75);
+
+	case depth_gt75_le100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT75_LE100);
+
+	case depth_gt100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT100);
+
+	default: return -1;
+	}
+}
+
+int
+dlb2_xstats_init(struct dlb2_eventdev *dlb2)
+{
+	/*
+	 * define the stats names and types. Used to build up the device
+	 * xstats array
+	 * There are multiple set of stats:
+	 *   - device-level,
+	 *   - per-port,
+	 *   - per-qid,
+	 *
+	 * For each of these sets, we have three parallel arrays, one for the
+	 * names, the other for the stat type parameter to be passed in the fn
+	 * call to get that stat. The third array allows resetting or not.
+	 * All these arrays must be kept in sync
+	 */
+	static const char * const dev_stats[] = {
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"nb_events_limit",
+		"inflight_events",
+		"ldb_pool_size",
+		"dir_pool_size",
+	};
+	static const enum dlb2_xstats_type dev_types[] = {
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		nb_events_limit,
+		inflight_events,
+		ldb_pool_size,
+		dir_pool_size,
+	};
+	/* Note: generated device stats are not allowed to be reset. */
+	static const uint8_t dev_reset_allowed[] = {
+		0, /* rx_ok */
+		0, /* rx_drop */
+		0, /* rx_interrupt_wait */
+		0, /* rx_umonitor_umwait */
+		0, /* tx_ok */
+		0, /* total_polls */
+		0, /* zero_polls */
+		0, /* tx_nospc_ldb_hw_credits */
+		0, /* tx_nospc_dir_hw_credits */
+		0, /* tx_nospc_inflight_max */
+		0, /* tx_nospc_new_event_limit */
+		0, /* tx_nospc_inflight_credits */
+		0, /* nb_events_limit */
+		0, /* inflight_events */
+		0, /* ldb_pool_size */
+		0, /* dir_pool_size */
+	};
+	static const char * const port_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"tx_new",
+		"tx_fwd",
+		"tx_rel",
+		"tx_implicit_rel",
+		"tx_sched_ordered",
+		"tx_sched_unordered",
+		"tx_sched_atomic",
+		"tx_sched_directed",
+		"tx_invalid",
+		"outstanding_releases",
+		"max_outstanding_releases",
+		"rx_sched_ordered",
+		"rx_sched_unordered",
+		"rx_sched_atomic",
+		"rx_sched_directed",
+		"rx_sched_invalid"
+	};
+	static const enum dlb2_xstats_type port_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		tx_new,
+		tx_fwd,
+		tx_rel,
+		tx_implicit_rel,
+		tx_sched_ordered,
+		tx_sched_unordered,
+		tx_sched_atomic,
+		tx_sched_directed,
+		tx_invalid,
+		outstanding_releases,
+		max_outstanding_releases,
+		rx_sched_ordered,
+		rx_sched_unordered,
+		rx_sched_atomic,
+		rx_sched_directed,
+		rx_sched_invalid
+	};
+	static const uint8_t port_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		1, /* rx_ok */
+		1, /* rx_drop */
+		1, /* rx_interrupt_wait */
+		1, /* rx_umonitor_umwait */
+		1, /* tx_ok */
+		1, /* total_polls */
+		1, /* zero_polls */
+		1, /* tx_nospc_ldb_hw_credits */
+		1, /* tx_nospc_dir_hw_credits */
+		1, /* tx_nospc_inflight_max */
+		1, /* tx_nospc_new_event_limit */
+		1, /* tx_nospc_inflight_credits */
+		1, /* tx_new */
+		1, /* tx_fwd */
+		1, /* tx_rel */
+		1, /* tx_implicit_rel */
+		1, /* tx_sched_ordered */
+		1, /* tx_sched_unordered */
+		1, /* tx_sched_atomic */
+		1, /* tx_sched_directed */
+		1, /* tx_invalid */
+		0, /* outstanding_releases */
+		0, /* max_outstanding_releases */
+		1, /* rx_sched_ordered */
+		1, /* rx_sched_unordered */
+		1, /* rx_sched_atomic */
+		1, /* rx_sched_directed */
+		1  /* rx_sched_invalid */
+	};
+
+	/* QID specific stats */
+	static const char * const qid_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"num_links",
+		"sched_type",
+		"enq_ok",
+		"current_depth",
+		"depth_threshold",
+		"depth_le50_threshold",
+		"depth_gt50_le75_threshold",
+		"depth_gt75_le100_threshold",
+		"depth_gt100_threshold",
+	};
+	static const enum dlb2_xstats_type qid_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		num_links,
+		sched_type,
+		enq_ok,
+		current_depth,
+		depth_threshold,
+		depth_le50_threshold,
+		depth_gt50_le75_threshold,
+		depth_gt75_le100_threshold,
+		depth_gt100_threshold,
+	};
+	static const uint8_t qid_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		0, /* num_links */
+		0, /* sched_type */
+		1, /* enq_ok */
+		0, /* current_depth */
+		0, /* depth_threshold */
+		1, /* depth_le50_threshold */
+		1, /* depth_gt50_le75_threshold */
+		1, /* depth_gt75_le100_threshold */
+		1, /* depth_gt100_threshold */
+	};
+
+	/* ---- end of stat definitions ---- */
+
+	/* check sizes, since a missed comma can lead to strings being
+	 * joined by the compiler.
+	 */
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
+
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_reset_allowed));
+
+	/* other vars */
+	const unsigned int count = RTE_DIM(dev_stats) +
+			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
+			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
+	unsigned int i, port, qid, stat_id = 0;
+
+	dlb2->xstats = rte_zmalloc_socket(NULL,
+			sizeof(dlb2->xstats[0]) * count, 0,
+			dlb2->qm_instance.info.socket_id);
+	if (dlb2->xstats == NULL)
+		return -ENOMEM;
+
+#define sname dlb2->xstats[stat_id].name.name
+	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
+		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
+			.fn_id = DLB2_XSTATS_FN_DEV,
+			.stat = dev_types[i],
+			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
+			.reset_allowed = dev_reset_allowed[i],
+		};
+		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
+	}
+	dlb2->xstats_count_mode_dev = stat_id;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
+		dlb2->xstats_offset_for_port[port] = stat_id;
+
+		uint32_t count_offset = stat_id;
+
+		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_PORT,
+				.obj_idx = port,
+				.stat = port_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_PORT,
+				.reset_allowed = port_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "port_%u_%s",
+				 port, port_stats[i]);
+		}
+
+		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_port = stat_id - dlb2->xstats_count_mode_dev;
+
+	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
+		uint32_t count_offset = stat_id;
+
+		dlb2->xstats_offset_for_qid[qid] = stat_id;
+
+		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_QUEUE,
+				.obj_idx = qid,
+				.stat = qid_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
+				.reset_allowed = qid_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "qid_%u_%s",
+				 qid, qid_stats[i]);
+		}
+
+		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_queue = stat_id -
+		(dlb2->xstats_count_mode_dev + dlb2->xstats_count_mode_port);
+#undef sname
+
+	dlb2->xstats_count = stat_id;
+
+	return 0;
+}
+
+void
+dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
+{
+	rte_free(dlb2->xstats);
+	dlb2->xstats_count = 0;
+}
+
+int
+dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstats_names,
+		unsigned int *ids, unsigned int size)
+{
+	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+	uint32_t start_offset = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			break;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			break;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
+		break;
+	default:
+		return -EINVAL;
+	};
+
+	if (xstats_mode_count > size || ids == NULL || xstats_names == NULL)
+		return xstats_mode_count;
+
+	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
+		if (dlb2->xstats[i].mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != dlb2->xstats[i].obj_idx)
+			continue;
+
+		xstats_names[xidx] = dlb2->xstats[i].name;
+		if (ids)
+			ids[xidx] = start_offset + xidx;
+		xidx++;
+	}
+	return xidx;
+}
+
+static int
+dlb2_xstats_update(struct dlb2_eventdev *dlb2,
+		enum rte_event_dev_xstats_mode mode,
+		uint8_t queue_port_id, const unsigned int ids[],
+		uint64_t values[], unsigned int n, const uint32_t reset)
+{
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			goto invalid_value;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			goto invalid_value;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		break;
+	default:
+		goto invalid_value;
+	};
+
+	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
+		dlb2_xstats_fn fn;
+
+		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != xs->obj_idx)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			goto invalid_value;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+
+		if (values)
+			values[xidx] = val;
+
+		if (xs->reset_allowed && reset)
+			xs->reset_value += val;
+
+		xidx++;
+	}
+
+	return xidx;
+
+invalid_value:
+	return -EINVAL;
+}
+
+int
+dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	const uint32_t reset = 0;
+
+	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
+				  reset);
+}
+
+uint64_t
+dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+				 const char *name, unsigned int *id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	dlb2_xstats_fn fn;
+
+	for (i = 0; i < dlb2->xstats_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (strncmp(xs->name.name, name,
+			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
+			if (id != NULL)
+				*id = i;
+
+			switch (xs->fn_id) {
+			case DLB2_XSTATS_FN_DEV:
+				fn = get_dev_stat;
+				break;
+			case DLB2_XSTATS_FN_PORT:
+				fn = get_port_stat;
+				break;
+			case DLB2_XSTATS_FN_QUEUE:
+				fn = get_queue_stat;
+				break;
+			default:
+				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
+					  xs->fn_id);
+				return (uint64_t)-1;
+			}
+
+			return fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+		}
+	}
+	if (id != NULL)
+		*id = (uint32_t)-1;
+	return (uint64_t)-1;
+}
+
+static void
+dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
+			uint32_t num)
+{
+	uint32_t i;
+	dlb2_xstats_fn fn;
+
+	for (i = start; i < start + num; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (!xs->reset_allowed)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			return;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
+		xs->reset_value = val;
+	}
+}
+
+static int
+dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
+			const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_QUEUE,
+					queue_id, ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	if (ids == NULL)
+		dlb2_xstats_reset_range(dlb2,
+			dlb2->xstats_offset_for_qid[queue_id],
+			dlb2->xstats_count_per_qid[queue_id]);
+
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
+		       const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+	int offset = dlb2->xstats_offset_for_port[port_id];
+	int nb_stat = dlb2->xstats_count_per_port[port_id];
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_PORT, port_id,
+					ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
+		      uint32_t nb_ids)
+{
+	uint32_t i;
+
+	if (ids) {
+		for (i = 0; i < nb_ids; i++) {
+			uint32_t id = ids[i];
+
+			if (id >= dlb2->xstats_count_mode_dev)
+				return -EINVAL;
+			dlb2_xstats_reset_range(dlb2, id, 1);
+		}
+	} else {
+		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
+			dlb2_xstats_reset_range(dlb2, i, 1);
+	}
+
+	return 0;
+}
+
+int
+dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+			   enum rte_event_dev_xstats_mode mode,
+			   int16_t queue_port_id,
+			   const uint32_t ids[],
+			   uint32_t nb_ids)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	uint32_t i;
+
+	/* handle -1 for queue_port_id here, looping over all ports/queues */
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
+			return -EINVAL;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+				if (dlb2_xstats_reset_port(dlb2, i,
+							   ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
+			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
+						   ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
+				if (dlb2_xstats_reset_queue(dlb2, i,
+							    ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
+			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
+						    ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	};
+
+	return 0;
+}
+
+void
+dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	int i;
+
+	if (f == NULL) {
+		printf("Invalid file pointer\n");
+		return;
+	}
+
+	if (dev == NULL) {
+		fprintf(f, "Invalid event device\n");
+		return;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (dlb2 == NULL) {
+		fprintf(f, "DLB2 Event device cannot be dumped!\n");
+		return;
+	}
+
+	if (!dlb2->configured)
+		fprintf(f, "DLB2 Event device is not configured\n");
+
+	handle = &dlb2->qm_instance;
+
+	fprintf(f, "================\n");
+	fprintf(f, "DLB2 Device Dump\n");
+	fprintf(f, "================\n");
+
+	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
+		dlb2->umwait_allowed ? "yes" : "no");
+
+	/* Generic top level device information */
+
+	fprintf(f, "device is configured and run state =");
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		fprintf(f, "STOPPED\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
+		fprintf(f, "STOPPING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
+		fprintf(f, "STARTING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
+		fprintf(f, "STARTED\n");
+	else
+		fprintf(f, "UNEXPECTED\n");
+
+	fprintf(f, "domain ID=%u, socket_id=%u, evdev=%p\n",
+		handle->domain_id, handle->info.socket_id, dlb2->event_dev);
+
+	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
+		dlb2->num_dir_ports, dlb2->num_dir_queues);
+
+	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
+		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
+
+	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
+		handle->cfg.resources.num_atomic_inflights,
+		handle->cfg.resources.num_hist_list_entries);
+
+	fprintf(f, "results from most recent hw resource query:\n");
+
+	fprintf(f, "\tnum_sched_domains = %u\n",
+		dlb2->hw_rsrc_query_results.num_sched_domains);
+
+	fprintf(f, "\tnum_ldb_queues = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_queues);
+
+	fprintf(f, "\tnum_ldb_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_ports);
+
+	fprintf(f, "\tnum_dir_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_ports);
+
+	fprintf(f, "\tnum_atomic_inflights = %u\n",
+		dlb2->hw_rsrc_query_results.num_atomic_inflights);
+
+	fprintf(f, "\tnum_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.num_hist_list_entries);
+
+	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
+
+	fprintf(f, "\tnum_ldb_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_credits);
+
+	fprintf(f, "\tnum_dir_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_credits);
+
+	/* Port level information */
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
+		int j;
+
+		if (!p->enq_configured)
+			fprintf(f, "Port_%d is not configured\n", i);
+
+		fprintf(f, "Port_%d\n", i);
+		fprintf(f, "=======\n");
+
+		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
+			p->id, p->setup_done);
+
+		fprintf(f, "\tconfig state=%d, port state=%d\n",
+			p->qm_port.config_state, p->qm_port.state);
+
+		fprintf(f, "\tport is %s\n",
+			p->qm_port.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\toutstanding releases=%u\n",
+			p->outstanding_releases);
+
+		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
+			p->inflight_max, p->inflight_credits);
+
+		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
+			p->credit_update_quanta, p->implicit_release);
+
+		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (p->link[j].valid)
+				fprintf(f, "id=%u prio=%u ",
+					p->link[j].queue_id,
+					p->link[j].priority);
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
+
+		fprintf(f, "\tcached_ldb_credits=%u\n",
+			p->qm_port.cached_ldb_credits);
+
+		fprintf(f, "\tldb_credits = %u\n",
+			p->qm_port.ldb_credits);
+
+		fprintf(f, "\tcached_dir_credits = %u\n",
+			p->qm_port.cached_dir_credits);
+
+		fprintf(f, "\tdir_credits = %u\n",
+			p->qm_port.dir_credits);
+
+		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
+			p->qm_port.gen_bit,
+			p->qm_port.cq_idx,
+			p->qm_port.cq_depth);
+
+		fprintf(f, "\tinterrupt armed=%d\n",
+			p->qm_port.int_armed);
+
+		fprintf(f, "\tPort statistics\n");
+
+		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
+			p->stats.traffic.rx_ok);
+
+		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
+			p->stats.traffic.rx_drop);
+
+		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
+			p->stats.traffic.rx_interrupt_wait);
+
+		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
+			p->stats.traffic.rx_umonitor_umwait);
+
+		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
+			p->stats.traffic.tx_ok);
+
+		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
+			p->stats.traffic.total_polls);
+
+		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
+			p->stats.traffic.zero_polls);
+
+		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_ldb_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_dir_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_max);
+
+		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_new_event_limit);
+
+		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_credits);
+
+		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
+
+		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
+
+		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
+
+		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
+			p->stats.tx_implicit_rel);
+
+		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
+			p->stats.tx_invalid);
+
+		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
+			p->stats.rx_sched_invalid);
+	}
+
+	/* Queue level information */
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
+		int j, k;
+
+		if (!q->setup_done)
+			fprintf(f, "Queue_%d is not configured\n", i);
+
+		fprintf(f, "Queue_%d\n", i);
+		fprintf(f, "========\n");
+
+		fprintf(f, "\tevqueue_%u is set up\n", q->id);
+
+		fprintf(f, "\tqueue is %s\n",
+			q->qm_queue.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
+
+		for (j = 0; j < dlb2->num_ports; j++) {
+			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
+
+			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++) {
+				if (p->link[k].valid &&
+				    p->link[k].queue_id == q->id)
+					fprintf(f, "id=%u prio=%u ",
+						p->id, p->link[k].priority);
+			}
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\tcurrent depth: %u events\n",
+			dlb2_get_queue_depth(dlb2, q));
+
+		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
+			q->qm_queue.num_qid_inflights, q->qm_queue.sched_type);
+	}
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 28fef3f..dc16c96 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files('dlb2.c',
 		'dlb2_iface.c',
+		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c'
-- 
2.6.4

[dpdk-dev] [PATCH v6 10/23] event/dlb2: add infos get and configure

[Timothy McDaniel]

Add support for configuring the DLB2 hardware.
In particular, this patch configures the DLB2
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  116 +
 drivers/event/dlb2/dlb2.c                  |  313 +++
 drivers/event/dlb2/dlb2_iface.c            |    5 +
 drivers/event/dlb2/dlb2_iface.h            |    4 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 3237 ++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |   15 +
 drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
 7 files changed, 3734 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index e3091c9..5f6c486 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -32,9 +32,125 @@ detailed understanding of the hardware, but these details are important when
 writing high-performance code. This section describes the places where the
 eventdev API and DLB2 misalign.
 
+Scheduling Domain Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are 32 scheduling domainis the DLB2.
+When one is configured, it allocates load-balanced and
+directed queues, ports, credits, and other hardware resources. Some
+resource allocations are user-controlled -- the number of queues, for example
+-- and others, like credit pools (one directed and one load-balanced pool per
+scheduling domain), are not.
+
+The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
+setup argument and the per-port ``new_event_threshold`` argument apply as
+defined in the eventdev header file. The limit is applied to all enqueues,
+regardless of whether it will consume a directed or load-balanced credit.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Reconfiguration
+~~~~~~~~~~~~~~~
+
+The Eventdev API allows one to reconfigure a device, its ports, and its queues
+by first stopping the device, calling the configuration function(s), then
+restarting the device. The DLB2 does not support configuring an individual queue
+or port without first reconfiguring the entire device, however, so there are
+certain reconfiguration sequences that are valid in the eventdev API but not
+supported by the PMD.
+
+Specifically, the PMD supports the following configuration sequence:
+1. Configure and start the device
+2. Stop the device
+3. (Optional) Reconfigure the device
+4. (Optional) If step 3 is run:
+
+   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
+   b. The reconfigured port(s) lose their previous queue links.
+
+5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
+6. Restart the device. If the device is reconfigured in step 3 but one or more
+   of its ports or queues are not, the PMD will apply their previous
+   configuration (including port->queue links) at this time.
+
+The PMD does not support the following configuration sequences:
+1. Configure and start the device
+2. Stop the device
+3. Setup queue or setup port
+4. Start the device
+
+This sequence is not supported because the event device must be reconfigured
+before its ports or queues can be.
+
+Atomic Inflights Allocation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
+inflight event in a temporary buffer that is divided among load-balanced
+queues. If a queue's atomic buffer storage fills up, this can result in
+head-of-line-blocking. For example:
+
+- An LDB queue allocated N atomic buffer entries
+- All N entries are filled with events from flow X, which is pinned to CQ 0.
+
+Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
+scheduled. The likelihood of this case depends on the eventdev configuration,
+traffic behavior, event processing latency, potential for a worker to be
+interrupted or otherwise delayed, etc.
+
+By default, the PMD allocates 16 buffer entries for each load-balanced queue,
+which provides an even division across all 128 queues but potentially wastes
+buffer space (e.g. if not all queues are used, or aren't used for atomic
+scheduling).
+
+The PMD provides a dev arg to override the default per-queue allocation. To
+increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,atm_inflights=64
+
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 925824e..6798e66 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -85,6 +85,25 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
 	return 0;
 }
 
+static void
+dlb2_free_qe_mem(struct dlb2_port *qm_port)
+{
+	if (qm_port == NULL)
+		return;
+
+	rte_free(qm_port->qe4);
+	qm_port->qe4 = NULL;
+
+	rte_free(qm_port->int_arm_qe);
+	qm_port->int_arm_qe = NULL;
+
+	rte_free(qm_port->consume_qe);
+	qm_port->consume_qe = NULL;
+
+	rte_memzone_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz);
+	dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz = NULL;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -350,10 +369,304 @@ set_qid_depth_thresh(const char *key __rte_unused,
 }
 
 static void
+dlb2_eventdev_info_get(struct rte_eventdev *dev,
+		       struct rte_event_dev_info *dev_info)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret;
+
+	ret = dlb2_hw_query_resources(dlb2);
+	if (ret) {
+		const struct rte_eventdev_data *data = dev->data;
+
+		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+			     ret, data->dev_id);
+		/* fn is void, so fall through and return values set up in
+		 * probe
+		 */
+	}
+
+	/* Add num resources currently owned by this domain.
+	 * These would become available if the scheduling domain were reset due
+	 * to the application recalling eventdev_configure to *reconfigure* the
+	 * domain.
+	 */
+	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
+	evdev_dlb2_default_info.max_event_queues += dlb2->num_ldb_queues;
+	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
+
+	evdev_dlb2_default_info.max_event_queues =
+		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
+			RTE_EVENT_MAX_QUEUES_PER_DEV);
+
+	evdev_dlb2_default_info.max_num_events =
+		RTE_MIN(evdev_dlb2_default_info.max_num_events,
+			dlb2->max_num_events_override);
+
+	*dev_info = evdev_dlb2_default_info;
+}
+
+static int
+dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
+			    const struct dlb2_hw_rsrcs *resources_asked)
+{
+	int ret = 0;
+	struct dlb2_create_sched_domain_args *cfg;
+
+	if (resources_asked == NULL) {
+		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
+		ret = EINVAL;
+		goto error_exit;
+	}
+
+	/* Map generic qm resources to dlb2 resources */
+	cfg = &handle->cfg.resources;
+
+	/* DIR ports and queues */
+
+	cfg->num_dir_ports = resources_asked->num_dir_ports;
+
+	cfg->num_dir_credits = resources_asked->num_dir_credits;
+
+	/* LDB queues */
+
+	cfg->num_ldb_queues = resources_asked->num_ldb_queues;
+
+	/* LDB ports */
+
+	cfg->cos_strict = 0; /* Best effort */
+	cfg->num_cos_ldb_ports[0] = 0;
+	cfg->num_cos_ldb_ports[1] = 0;
+	cfg->num_cos_ldb_ports[2] = 0;
+	cfg->num_cos_ldb_ports[3] = 0;
+
+	switch (handle->cos_id) {
+	case DLB2_COS_0:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[0] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_1:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[1] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_2:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[2] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_3:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[3] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_DEFAULT:
+		/* all ldb ports are don't care ports from a cos perspective */
+		cfg->num_ldb_ports =
+			resources_asked->num_ldb_ports;
+		break;
+	}
+
+	cfg->num_ldb_credits =
+		resources_asked->num_ldb_credits;
+
+	cfg->num_atomic_inflights =
+		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
+		cfg->num_ldb_queues;
+
+	cfg->num_hist_list_entries = resources_asked->num_ldb_ports *
+		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d\n",
+		     cfg->num_ldb_queues,
+		     resources_asked->num_ldb_ports,
+		     cfg->num_dir_ports,
+		     cfg->num_atomic_inflights,
+		     cfg->num_hist_list_entries,
+		     cfg->num_ldb_credits,
+		     cfg->num_dir_credits);
+
+	/* Configure the QM */
+
+	ret = dlb2_iface_sched_domain_create(handle, cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: domain create failed, ret = %d, extra status: %s\n",
+			     ret,
+			     dlb2_error_strings[cfg->response.status]);
+
+		goto error_exit;
+	}
+
+	handle->domain_id = cfg->response.id;
+	handle->cfg.configured = true;
+
+error_exit:
+
+	return ret;
+}
+
+static void
+dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	enum dlb2_configuration_state config_state;
+	int i, j;
+
+	dlb2_iface_domain_reset(dlb2);
+
+	/* Free all dynamically allocated port memory */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
+
+	/* If reconfiguring, mark the device's queues and ports as "previously
+	 * configured." If the user doesn't reconfigure them, the PMD will
+	 * reapply their previous configuration when the device is started.
+	 */
+	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
+		DLB2_NOT_CONFIGURED;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		dlb2->ev_ports[i].qm_port.config_state = config_state;
+		/* Reset setup_done so ports can be reconfigured */
+		dlb2->ev_ports[i].setup_done = false;
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			dlb2->ev_ports[i].link[j].mapped = false;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++)
+		dlb2->ev_queues[i].qm_queue.config_state = config_state;
+
+	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
+		dlb2->ev_queues[i].setup_done = false;
+
+	dlb2->num_ports = 0;
+	dlb2->num_ldb_ports = 0;
+	dlb2->num_dir_ports = 0;
+	dlb2->num_queues = 0;
+	dlb2->num_ldb_queues = 0;
+	dlb2->num_dir_queues = 0;
+	dlb2->configured = false;
+}
+
+/* Note: 1 QM instance per QM device, QM instance/device == event device */
+static int
+dlb2_eventdev_configure(const struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
+	const struct rte_eventdev_data *data = dev->data;
+	const struct rte_event_dev_config *config = &data->dev_conf;
+	int ret;
+
+	/* If this eventdev is already configured, we must release the current
+	 * scheduling domain before attempting to configure a new one.
+	 */
+	if (dlb2->configured) {
+		dlb2_hw_reset_sched_domain(dev, true);
+
+		ret = dlb2_hw_query_resources(dlb2);
+		if (ret) {
+			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+				     ret, data->dev_id);
+			return ret;
+		}
+	}
+
+	if (config->nb_event_queues > rsrcs->num_queues) {
+		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_queues,
+			     rsrcs->num_queues);
+		return -EINVAL;
+	}
+	if (config->nb_event_ports > (rsrcs->num_ldb_ports
+			+ rsrcs->num_dir_ports)) {
+		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_ports,
+			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
+		return -EINVAL;
+	}
+	if (config->nb_events_limit > rsrcs->nb_events_limit) {
+		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_events_limit,
+			     rsrcs->nb_events_limit);
+		return -EINVAL;
+	}
+
+	if (config->event_dev_cfg & RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT)
+		dlb2->global_dequeue_wait = false;
+	else {
+		uint32_t timeout32;
+
+		dlb2->global_dequeue_wait = true;
+
+		/* note size mismatch of timeout vals in eventdev lib. */
+		timeout32 = config->dequeue_timeout_ns;
+
+		dlb2->global_dequeue_wait_ticks =
+			timeout32 * (rte_get_timer_hz() / 1E9);
+	}
+
+	/* Does this platform support umonitor/umwait? */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
+		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
+			DLB2_LOG_ERR("invalid value (%d) for RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
+				     RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
+			return -EINVAL;
+		}
+		dlb2->umwait_allowed = true;
+	}
+
+	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
+	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs->num_dir_ports;
+	/* 1 dir queue per dir port */
+	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports;
+
+	/* Scale down nb_events_limit by 4 for directed credits, since there
+	 * are 4x as many load-balanced credits.
+	 */
+	rsrcs->num_ldb_credits = 0;
+	rsrcs->num_dir_credits = 0;
+
+	if (rsrcs->num_ldb_queues)
+		rsrcs->num_ldb_credits = config->nb_events_limit;
+	if (rsrcs->num_dir_ports)
+		rsrcs->num_dir_credits = config->nb_events_limit / 4;
+	if (dlb2->num_dir_credits_override != -1)
+		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
+
+	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
+		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
+		return -ENODEV;
+	}
+
+	dlb2->new_event_limit = config->nb_events_limit;
+	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
+
+	/* Save number of ports/queues for this event dev */
+	dlb2->num_ports = config->nb_event_ports;
+	dlb2->num_queues = config->nb_event_queues;
+	dlb2->num_dir_ports = rsrcs->num_dir_ports;
+	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
+	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
+	dlb2->num_dir_queues = dlb2->num_dir_ports;
+	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
+	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
+	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
+	dlb2->max_dir_credits = rsrcs->num_dir_credits;
+
+	dlb2->configured = true;
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dev_infos_get    = dlb2_eventdev_info_get,
+		.dev_configure    = dlb2_eventdev_configure,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 0fc9991..a829b9b 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -26,3 +26,8 @@ int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
+
+int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				struct dlb2_create_sched_domain_args *args);
+
+void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4fb416e..6663dab 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -26,4 +26,8 @@ extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
 
+extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				 struct dlb2_create_sched_domain_args *args);
+
+extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 6de8b95..63a68a6 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -12,6 +12,27 @@
 #include "dlb2_regs.h"
 #include "dlb2_resource.h"
 
+#include "../../dlb2_priv.h"
+#include "../../dlb2_inline_fns.h"
+
+#define DLB2_DOM_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, domain_list, iter)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, func_list, iter)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, domain_list, it, it_tmp)
+
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, func_list, it, it_tmp)
+
 static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
 {
 	int i;
@@ -272,3 +293,3219 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
 
 	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
 }
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+	union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+	r0.field.count = domain->num_ldb_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+	r1.field.count = domain->num_dir_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+		       struct dlb2_function_resources *rsrcs,
+		       u32 domain_id,
+		       u32 cos_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	RTE_SET_USED(iter);
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+				   typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				   struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_ports,
+				   u32 cos_id,
+				   struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_ldb_port *port;
+
+		port = dlb2_get_next_ldb_port(hw, rsrcs,
+					      domain->id.phys_id, cos_id);
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_ports[cos_id],
+			      &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_ports[cos_id],
+			      &port->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_create_sched_domain_args *args,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      num,
+						      i,
+						      resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+					ret = __dlb2_attach_ldb_ports(hw,
+								      rsrcs,
+								      domain,
+								      1,
+								      cos_id,
+								      resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret < 0)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      1,
+						      j,
+						      resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 u32 num_ports,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+					   typeof(*port));
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_dir_pq_pairs, &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_dir_pq_pairs, &port->domain_list);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+					struct dlb2_hw_domain *domain,
+					u32 num_atomic_inflights,
+					struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+				     struct dlb2_hw_domain *domain,
+				     u32 num_hist_list_entries,
+				     struct dlb2_cmd_response *resp)
+{
+	struct dlb2_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb2_bitmap_find_set_bit_range(bitmap,
+						      num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+				  struct dlb2_function_resources *rsrcs,
+				  struct dlb2_hw_domain *domain,
+				  u32 num_queues,
+				  struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb2_ldb_queue *queue;
+
+		queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+					    typeof(*queue));
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_queues, &queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_queues, &queue->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+			     struct dlb2_function_resources *rsrcs,
+			     struct dlb2_hw_domain *domain,
+			     struct dlb2_create_sched_domain_args *args,
+			     struct dlb2_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb2_attach_ldb_queues(hw,
+				     rsrcs,
+				     domain,
+				     args->num_ldb_queues,
+				     resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_ports(hw,
+				    rsrcs,
+				    domain,
+				    args,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_ports(hw,
+				    rsrcs,
+				    domain,
+				    args->num_dir_ports,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_credits(rsrcs,
+				      domain,
+				      args->num_ldb_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_credits(rsrcs,
+				      domain,
+				      args->num_dir_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+						   domain,
+						   args->num_hist_list_entries,
+						   resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_atomic_inflights(rsrcs,
+					   domain,
+					   args->num_atomic_inflights,
+					   resp);
+	if (ret < 0)
+		return ret;
+
+	dlb2_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+				  struct dlb2_create_sched_domain_args *args,
+				  struct dlb2_cmd_response *resp)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb2_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+				  struct dlb2_create_sched_domain_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		    args->num_ldb_queues);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		    args->num_ldb_ports);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		    args->num_cos_ldb_ports[0]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		    args->cos_strict);
+	DLB2_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		    args->num_dir_ports);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		    args->num_atomic_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		    args->num_hist_list_entries);
+	DLB2_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		    args->num_ldb_credits);
+	DLB2_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		    args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ *	domain and its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+	if (ret)
+		return ret;
+
+	domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available domains\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (domain->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_domains contains configured domains.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	dlb2_init_domain_rsrc_lists(domain);
+
+	ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to verify args.\n",
+			    __func__);
+
+		return ret;
+	}
+
+	dlb2_list_del(&rsrcs->avail_domains, &domain->func_list);
+
+	dlb2_list_add(&rsrcs->used_domains, &domain->func_list);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+	DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+			     struct dlb2_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb2_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	int ret;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_disable(hw, port);
+
+		ret = dlb2_drain_dir_cq(hw, port);
+		if (ret < 0)
+			return ret;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_dir_pq_pair *queue)
+{
+	union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *queue)
+{
+	return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+				      struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		dlb2_movdir64b(pp_addr, hcw);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int ret, i;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			if (toggle_port)
+				dlb2_ldb_port_cq_disable(hw, port);
+
+			ret = dlb2_drain_ldb_cq(hw, port);
+			if (ret < 0)
+				return ret;
+
+			if (toggle_port)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+
+	return 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_atm_active r1;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_ldb_queue *queue)
+{
+	return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb2_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+					   struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to unmap domain queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = true;
+
+			dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+			   u32 id,
+			   bool vdev_req,
+			   unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iter1) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter2)
+			if (queue->id.virt_id == id)
+				return queue;
+	}
+
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue, iter1)
+		if (queue->id.virt_id == id)
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+						      u32 id,
+						      bool vdev_req,
+						      unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iteration;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	RTE_SET_USED(iteration);
+
+	if (id >= DLB2_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iteration)
+		if (domain->id.virt_id == id)
+			return domain;
+
+	return NULL;
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot,
+					   enum dlb2_qid_map_state new_state)
+{
+	enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb2_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB2_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB2_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB2_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB2_HW_DBG(hw,
+		    "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB2_HW_ERR(hw,
+		    "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+				enum dlb2_qid_map_state state,
+				int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+				      enum dlb2_qid_map_state state,
+				      struct dlb2_ldb_queue *queue,
+				      int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+					      struct dlb2_hw_domain *domain,
+					      struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain,
+					     struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+						struct dlb2_ldb_port *port,
+						int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 0;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 1;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+					struct dlb2_ldb_port *p,
+					struct dlb2_ldb_queue *q,
+					u8 priority)
+{
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_lsp_cq2qid0 r1;
+	union dlb2_atm_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix_00 r3;
+	union dlb2_lsp_qid2cqidix2_00 r4;
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+	    !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+	r0.field.v |= 1 << i;
+	r0.field.prio |= (priority & 0x7) << i * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+	else
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		r1.field.qid_p0 = q->id.phys_id;
+	if (i == 1 || i == 5)
+		r1.field.qid_p1 = q->id.phys_id;
+	if (i == 2 || i == 6)
+		r1.field.qid_p2 = q->id.phys_id;
+	if (i == 3 || i == 7)
+		r1.field.qid_p3 = q->id.phys_id;
+
+	if (i < 4)
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+	else
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r4.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+						  p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		r2.field.cq_p0 |= 1 << i;
+		r3.field.cq_p0 |= 1 << i;
+		r4.field.cq_p0 |= 1 << i;
+		break;
+
+	case 1:
+		r2.field.cq_p1 |= 1 << i;
+		r3.field.cq_p1 |= 1 << i;
+		r4.field.cq_p1 |= 1 << i;
+		break;
+
+	case 2:
+		r2.field.cq_p2 |= 1 << i;
+		r3.field.cq_p2 |= 1 << i;
+		r4.field.cq_p2 |= 1 << i;
+		break;
+
+	case 3:
+		r2.field.cq_p3 |= 1 << i;
+		r3.field.cq_p3 |= 1 << i;
+		r4.field.cq_p3 |= 1 << i;
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r3.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		    r4.val);
+
+	dlb2_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB2_QUEUE_MAPPED;
+
+	return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	/* Set the atomic scheduling haswork bit */
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.rlist_haswork_v = r0.field.count > 0;
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      u8 slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.rlist_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.nalb_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+					      struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+	r0.field.limit = queue->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+						struct dlb2_ldb_queue *queue)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+						struct dlb2_hw_domain *domain,
+						struct dlb2_ldb_port *port,
+						struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_lsp_qid_ldb_infl_cnt r0;
+	enum dlb2_qid_map_state state;
+	int slot, ret, i;
+	u8 prio;
+	RTE_SET_USED(iter);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: non-zero QID inflight count\n",
+			    __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			state = DLB2_QUEUE_MAPPED;
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+					 struct dlb2_ldb_port *port,
+					 struct dlb2_ldb_queue *queue,
+					 u8 priority)
+{
+	union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	int domain_id, slot, ret;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+		DLB2_HW_ERR(hw,
+			    "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		if (port->enabled)
+			dlb2_ldb_port_cq_enable(hw, port);
+
+		dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain,
+					struct dlb2_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		union dlb2_lsp_qid_ldb_infl_cnt r0;
+		struct dlb2_ldb_queue *queue;
+		int qid;
+
+		if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: unable to find queue %d\n",
+				    __func__, qid);
+			continue;
+		}
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count)
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb2_ldb_port_cq_disable(hw, port);
+
+		dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count) {
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+
+			dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port,
+				   struct dlb2_ldb_queue *queue)
+{
+	enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_atm_qid2cqidix_00 r1;
+	union dlb2_lsp_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix2_00 r3;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB2_QUEUE_MAPPED;
+	in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: QID %d isn't mapped\n",
+			    __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+	r0.field.v &= ~(1 << i);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		r1.field.cq_p0 &= ~(1 << i);
+		r2.field.cq_p0 &= ~(1 << i);
+		r3.field.cq_p0 &= ~(1 << i);
+		break;
+
+	case 1:
+		r1.field.cq_p1 &= ~(1 << i);
+		r2.field.cq_p1 &= ~(1 << i);
+		r3.field.cq_p1 &= ~(1 << i);
+		break;
+
+	case 2:
+		r1.field.cq_p2 &= ~(1 << i);
+		r2.field.cq_p2 &= ~(1 << i);
+		r3.field.cq_p2 &= ~(1 << i);
+		break;
+
+	case 3:
+		r1.field.cq_p3 &= ~(1 << i);
+		r2.field.cq_p3 &= ~(1 << i);
+		r3.field.cq_p3 &= ~(1 << i);
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+		    r1.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+		    r3.val);
+
+	dlb2_flush_csr(hw);
+
+	unmapped = DLB2_QUEUE_UNMAPPED;
+
+	return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_ldb_port *port,
+				 struct dlb2_ldb_queue *queue,
+				 u8 prio)
+{
+	if (domain->started)
+		return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   int slot)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb2_ldb_port_qid_map *map;
+		struct dlb2_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain,
+					  struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (r0.field.count > 0)
+		return false;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = false;
+
+			dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_dir_vpp_v r1;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		unsigned int offs;
+		u32 virt_id;
+
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+	}
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_ldb_vpp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			unsigned int offs;
+			u32 virt_id;
+
+			if (hw->virt_mode == DLB2_VIRT_SRIOV)
+				virt_id = port->id.virt_id;
+			else
+				virt_id = port->id.phys_id;
+
+			offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+			DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+	union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+				    r0.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+				    r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+	union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+			    r0.val);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+			    r1.val);
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+		union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+		union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+				    r1.val);
+
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID_V(idx),
+				    r2.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID2QID(idx),
+				    r3.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID_V(idx),
+				    r1.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID2QID(idx),
+				    r2.val);
+		}
+	}
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+					       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_sn_chk_enbl r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.en = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			int i;
+
+			for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+				if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					    __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		port->enabled = false;
+
+		dlb2_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	union dlb2_sys_dir_pp_v r1;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+			    r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_ldb_pp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_ldb_queue *queue;
+	int i;
+	RTE_SET_USED(iter);
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty ldb queue %d\n",
+				    __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port, iter) {
+			if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb2_ldb_cq_token_count(hw, ldb_port)) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to empty ldb port %d\n",
+					    __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir queue %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb2_dir_cq_token_count(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir port %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						   struct dlb2_ldb_port *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP2PP(offs),
+			    DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP_V(offs),
+			    DLB2_SYS_VF_LDB_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    DLB2_SYS_LDB_PP_V_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_DSBL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_BASE_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID0(port->id.phys_id),
+		    DLB2_LSP_CQ2QID0_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID1(port->id.phys_id),
+		    DLB2_LSP_CQ2QID1_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+		    DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			__dlb2_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+				       struct dlb2_dir_pq_pair *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_DSBL_RST);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_FMT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP2PP(offs),
+			    DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP_V(offs),
+			    DLB2_SYS_VF_DIR_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		__dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+			    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_ITS(queue_id),
+			    DLB2_SYS_LDB_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN(queue_id),
+			    DLB2_CHP_ORD_QID_SN_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+			    DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_V(queue_id),
+			    DLB2_SYS_LDB_QID_V_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_CFG_V(queue_id),
+			    DLB2_SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB2_CSR_WR(hw,
+				    offs[queue->sn_group],
+				    DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX2(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX2_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_ATM_QID2CQIDIX(queue_id, i),
+				    DLB2_ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb2_domain_reset_dir_port_registers(hw, domain);
+
+	dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_dir_pq_pair *tmp_dir_port;
+	struct dlb2_ldb_queue *tmp_ldb_queue;
+	struct dlb2_ldb_port *tmp_ldb_port;
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_list_head *list;
+	int ret, i;
+	RTE_SET_USED(tmp_dir_port);
+	RTE_SET_USED(tmp_ldb_queue);
+	RTE_SET_USED(tmp_ldb_port);
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list = &domain->used_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb2_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		dlb2_list_del(&domain->used_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list = &domain->avail_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		ldb_queue->owned = false;
+
+		dlb2_list_del(&domain->avail_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		list = &domain->used_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB2_QUEUE_UNMAPPED;
+
+			dlb2_list_del(&domain->used_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list = &domain->avail_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			ldb_port->owned = false;
+
+			dlb2_list_del(&domain->avail_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list = &domain->used_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		dlb2_list_del(&domain->used_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list = &domain->avail_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				    domain->hist_list_entry_base,
+				    domain->total_hist_list_entries);
+	if (ret) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			    __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	dlb2_list_del(&rsrcs->used_domains, &domain->func_list);
+	dlb2_list_add(&rsrcs->avail_domains, &domain->func_list);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain,
+					    struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_ldb_port *port;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (!dlb2_list_empty(&domain->used_ldb_ports[i]))
+			break;
+	}
+
+	if (i == DLB2_NUM_COS_DOMAINS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: No configured LDB ports\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb2_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+	 * prior to this in dlb2_domain_drain_mapped_queues().
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings != 0 ||
+		    dlb2_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ *	hardware resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain  == NULL || !domain->configured)
+		return -EINVAL;
+
+	/* Disable VPPs */
+	if (vdev_req) {
+		dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+		dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+	}
+
+	/* Disable CQ interrupts */
+	dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+	dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+	dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb2_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_mapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	dlb2_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb2_domain_disable_dir_cqs(hw, domain);
+
+	/* Disable PPs */
+	dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+	ret = dlb2_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Reset the QID and port state. */
+	dlb2_domain_reset_registers(hw, domain);
+
+	/* Hardware reset complete. Reset the domain's software state */
+	ret = dlb2_domain_reset_software_state(hw, domain);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 210b3186..285ad2c 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -598,3 +598,18 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
 
 	return 0;
 }
+
+int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+			    struct dlb2_create_sched_domain_args *args,
+			    struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_sched_domain(hw, args, resp, NOT_VF_REQ,
+					   PF_ID_ZERO);
+}
+
+int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 7d64309..9310f72 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -108,15 +108,59 @@ dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
 	return 0;
 }
 
+static int
+dlb2_pf_sched_domain_create(struct dlb2_hw_dev *handle,
+			    struct dlb2_create_sched_domain_args *arg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (dlb2_dev->domain_reset_failed) {
+		response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = dlb2_pf_create_sched_domain(&dlb2_dev->hw, arg, &response);
+	if (ret)
+		goto done;
+
+done:
+
+	arg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static void
+dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret;
+
+	dlb2_dev = (struct dlb2_dev *)dlb2->qm_instance.pf_dev;
+	ret = dlb2_pf_reset_domain(&dlb2_dev->hw, dlb2->qm_instance.domain_id);
+	if (ret)
+		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
 	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
 	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_domain_reset = dlb2_pf_domain_reset;
 	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
 	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v6 11/23] event/dlb2: add queue and port default conf

[Timothy McDaniel]

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 6798e66..ef1c000 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -661,12 +661,42 @@ dlb2_eventdev_configure(const struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
+				    uint8_t port_id,
+				    struct rte_event_port_conf *port_conf)
+{
+	RTE_SET_USED(port_id);
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	port_conf->new_event_threshold = dlb2->new_event_limit;
+	port_conf->dequeue_depth = 32;
+	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
+	port_conf->event_port_cfg = 0;
+}
+
+static void
+dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
+				     uint8_t queue_id,
+				     struct rte_event_queue_conf *queue_conf)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(queue_id);
+
+	queue_conf->nb_atomic_flows = 1024;
+	queue_conf->nb_atomic_order_sequences = 64;
+	queue_conf->event_queue_cfg = 0;
+	queue_conf->priority = 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v6 12/23] event/dlb2: add queue setup

[Timothy McDaniel]

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  73 +++--
 drivers/event/dlb2/dlb2.c                  | 312 +++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |  12 +
 drivers/event/dlb2/dlb2_iface.h            |  12 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 464 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  82 +++++
 7 files changed, 926 insertions(+), 39 deletions(-)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 5f6c486..b154cac 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -47,45 +47,40 @@ setup argument and the per-port ``new_event_threshold`` argument apply as
 defined in the eventdev header file. The limit is applied to all enqueues,
 regardless of whether it will consume a directed or load-balanced credit.
 
-Load-balanced and Directed Ports
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
-does not have the same concept, but it has a similar one: ports and queues that
-are singly-linked (i.e. linked to a single queue or port, respectively).
-
-The ``rte_event_dev_info_get()`` function reports the number of available
-event ports and queues (among other things). For the DLB2 PMD, max_event_ports
-and max_event_queues report the number of available load-balanced ports and
-queues, and max_single_link_event_port_queue_pairs reports the number of
-available directed ports and queues.
-
-When a scheduling domain is created in ``rte_event_dev_configure()``, the user
-specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
-control the total number of ports (load-balanced and directed) and the number
-of directed ports. Hence, the number of requested load-balanced ports is
-``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
-specifies the total number of queues (load-balanced and directed). The number
-of directed queues comes from ``nb_single_link_event_port_queues``, since
-directed ports and queues come in pairs.
-
-When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
-whether it should be configured as a directed (the flag is set) or a
-load-balanced (the flag is unset) port. Similarly, the
-``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
-whether it is a directed or load-balanced queue.
-
-Load-balanced ports can only be linked to load-balanced queues, and directed
-ports can only be linked to directed queues. Furthermore, directed ports can
-only be linked to a single directed queue (and vice versa), and that link
-cannot change after the eventdev is started.
-
-The eventdev API does not have a directed scheduling type. To support directed
-traffic, the dlb PMD detects when an event is being sent to a directed queue
-and overrides its scheduling type. Note that the originally selected scheduling
-type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
-will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
-port.
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
 
 Flow ID
 ~~~~~~~
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index ef1c000..8c1e06d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -688,6 +688,317 @@ dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
 	queue_conf->priority = 0;
 }
 
+static int32_t
+dlb2_get_sn_allocation(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_allocation_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_set_sn_allocation(struct dlb2_eventdev *dlb2, int group, int num)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_set_sn_allocation_args cfg;
+	int ret;
+
+	cfg.num = num;
+	cfg.group = group;
+
+	ret = dlb2_iface_set_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: set_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int32_t
+dlb2_get_sn_occupancy(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_occupancy_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_occupancy(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_occupancy ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+/* Query the current sequence number allocations and, if they conflict with the
+ * requested LDB queue configuration, attempt to re-allocate sequence numbers.
+ * This is best-effort; if it fails, the PMD will attempt to configure the
+ * load-balanced queue and return an error.
+ */
+static void
+dlb2_program_sn_allocation(struct dlb2_eventdev *dlb2,
+			   const struct rte_event_queue_conf *queue_conf)
+{
+	int grp_occupancy[DLB2_NUM_SN_GROUPS];
+	int grp_alloc[DLB2_NUM_SN_GROUPS];
+	int i, sequence_numbers;
+
+	sequence_numbers = (int)queue_conf->nb_atomic_order_sequences;
+
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		int total_slots;
+
+		grp_alloc[i] = dlb2_get_sn_allocation(dlb2, i);
+		if (grp_alloc[i] < 0)
+			return;
+
+		total_slots = DLB2_MAX_LDB_SN_ALLOC / grp_alloc[i];
+
+		grp_occupancy[i] = dlb2_get_sn_occupancy(dlb2, i);
+		if (grp_occupancy[i] < 0)
+			return;
+
+		/* DLB has at least one available slot for the requested
+		 * sequence numbers, so no further configuration required.
+		 */
+		if (grp_alloc[i] == sequence_numbers &&
+		    grp_occupancy[i] < total_slots)
+			return;
+	}
+
+	/* None of the sequence number groups are configured for the requested
+	 * sequence numbers, so we have to reconfigure one of them. This is
+	 * only possible if a group is not in use.
+	 */
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		if (grp_occupancy[i] == 0)
+			break;
+	}
+
+	if (i == DLB2_NUM_SN_GROUPS) {
+		printf("[%s()] No groups with %d sequence_numbers are available or have free slots\n",
+		       __func__, sequence_numbers);
+		return;
+	}
+
+	/* Attempt to configure slot i with the requested number of sequence
+	 * numbers. Ignore the return value -- if this fails, the error will be
+	 * caught during subsequent queue configuration.
+	 */
+	dlb2_set_sn_allocation(dlb2, i, sequence_numbers);
+}
+
+static int32_t
+dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 const struct rte_event_queue_conf *evq_conf)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_queue *queue = &ev_queue->qm_queue;
+	struct dlb2_create_ldb_queue_args cfg;
+	int32_t ret;
+	uint32_t qm_qid;
+	int sched_type = -1;
+
+	if (evq_conf == NULL)
+		return -EINVAL;
+
+	if (evq_conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_ALL_TYPES) {
+		if (evq_conf->nb_atomic_order_sequences != 0)
+			sched_type = RTE_SCHED_TYPE_ORDERED;
+		else
+			sched_type = RTE_SCHED_TYPE_PARALLEL;
+	} else
+		sched_type = evq_conf->schedule_type;
+
+	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
+	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
+	cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences;
+
+	if (sched_type != RTE_SCHED_TYPE_ORDERED) {
+		cfg.num_sequence_numbers = 0;
+		cfg.num_qid_inflights = 2048;
+	}
+
+	/* App should set this to the number of hardware flows they want, not
+	 * the overall number of flows they're going to use. E.g. if app is
+	 * using 64 flows and sets compression to 64, best-case they'll get
+	 * 64 unique hashed flows in hardware.
+	 */
+	switch (evq_conf->nb_atomic_flows) {
+	/* Valid DLB2 compression levels */
+	case 64:
+	case 128:
+	case 256:
+	case 512:
+	case (1 * 1024): /* 1K */
+	case (2 * 1024): /* 2K */
+	case (4 * 1024): /* 4K */
+	case (64 * 1024): /* 64K */
+		cfg.lock_id_comp_level = evq_conf->nb_atomic_flows;
+		break;
+	default:
+		/* Invalid compression level */
+		cfg.lock_id_comp_level = 0; /* no compression */
+	}
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create LB event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	qm_qid = cfg.response.id;
+
+	/* Save off queue config for debug, resource lookups, and reconfig */
+	queue->num_qid_inflights = cfg.num_qid_inflights;
+	queue->num_atm_inflights = cfg.num_atomic_inflights;
+
+	queue->sched_type = sched_type;
+	queue->config_state = DLB2_CONFIGURED;
+
+	DLB2_LOG_DBG("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n",
+		     qm_qid,
+		     cfg.num_atomic_inflights,
+		     cfg.num_sequence_numbers,
+		     cfg.num_qid_inflights);
+
+	return qm_qid;
+}
+
+static int
+dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int32_t qm_qid;
+
+	if (queue_conf->nb_atomic_order_sequences)
+		dlb2_program_sn_allocation(dlb2, queue_conf);
+
+	qm_qid = dlb2_hw_create_ldb_queue(dlb2, ev_queue, queue_conf);
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the load-balanced queue\n");
+
+		return qm_qid;
+	}
+
+	dlb2->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int dlb2_num_dir_queues_setup(struct dlb2_eventdev *dlb2)
+{
+	int i, num = 0;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].setup_done &&
+		    dlb2->ev_queues[i].qm_queue.is_directed)
+			num++;
+	}
+
+	return num;
+}
+
+static void
+dlb2_queue_link_teardown(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue)
+{
+	struct dlb2_eventdev_port *ev_port;
+	int i, j;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (!ev_port->link[j].valid ||
+			    ev_port->link[j].queue_id != ev_queue->id)
+				continue;
+
+			ev_port->link[j].valid = false;
+			ev_port->num_links--;
+		}
+	}
+
+	ev_queue->num_links = 0;
+}
+
+static int
+dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
+			  uint8_t ev_qid,
+			  const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_queue *ev_queue;
+	int ret;
+
+	if (queue_conf == NULL)
+		return -EINVAL;
+
+	if (ev_qid >= dlb2->num_queues)
+		return -EINVAL;
+
+	ev_queue = &dlb2->ev_queues[ev_qid];
+
+	ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg &
+		RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+	ev_queue->id = ev_qid;
+	ev_queue->conf = *queue_conf;
+
+	if (!ev_queue->qm_queue.is_directed) {
+		ret = dlb2_eventdev_ldb_queue_setup(dev, ev_queue, queue_conf);
+	} else {
+		/* The directed queue isn't setup until link time, at which
+		 * point we know its directed port ID. Directed queue setup
+		 * will only fail if this queue is already setup or there are
+		 * no directed queues left to configure.
+		 */
+		ret = 0;
+
+		ev_queue->qm_queue.config_state = DLB2_NOT_CONFIGURED;
+
+		if (ev_queue->setup_done ||
+		    dlb2_num_dir_queues_setup(dlb2) == dlb2->num_dir_queues)
+			ret = -EINVAL;
+	}
+
+	/* Tear down pre-existing port->queue links */
+	if (!ret && dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_queue_link_teardown(dlb2, ev_queue);
+
+	if (!ret)
+		ev_queue->setup_done = true;
+
+	return ret;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -696,6 +1007,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a829b9b..4c07574 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -31,3 +31,15 @@ int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				struct dlb2_create_sched_domain_args *args);
 
 void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_ldb_queue_args *cfg);
+
+int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_sn_allocation_args *args);
+
+int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_set_sn_allocation_args *args);
+
+int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_occupancy_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 6663dab..4c88fe0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -30,4 +30,16 @@ extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				 struct dlb2_create_sched_domain_args *args);
 
 extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+extern int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_queue_args *cfg);
+
+extern int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_allocation_args *args);
+
+extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_set_sn_allocation_args *args);
+
+extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				  struct dlb2_get_sn_occupancy_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 63a68a6..e6ea0d7 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3509,3 +3509,467 @@ unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
 
 	return num;
 }
+
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_ldb_queue *queue,
+				     struct dlb2_create_ldb_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+	union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+	union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+	union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+	union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+	union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+	union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+	union dlb2_sys_ldb_qid_its r7 = { {0} };
+	union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+	union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+	union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+	union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+	union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+	union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+	struct dlb2_sn_group *sn_group;
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r3.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	r4.field.limit = args->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+	r5.field.limit = queue->aqed_limit;
+
+	if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+		r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+		    r5.val);
+
+	switch (args->lock_id_comp_level) {
+	case 64:
+		r6.field.compress_code = 1;
+		break;
+	case 128:
+		r6.field.compress_code = 2;
+		break;
+	case 256:
+		r6.field.compress_code = 3;
+		break;
+	case 512:
+		r6.field.compress_code = 4;
+		break;
+	case 1024:
+		r6.field.compress_code = 5;
+		break;
+	case 2048:
+		r6.field.compress_code = 6;
+		break;
+	case 4096:
+		r6.field.compress_code = 7;
+		break;
+	case 0:
+	case 65536:
+		r6.field.compress_code = 0;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+		    r6.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r7.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+		    r7.val);
+
+	r8.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+		    r8.val);
+
+	r9.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+		    r9.val);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	r10.field.qid_fid_limit = 512;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+		    r10.val);
+
+	/* Configure SNs */
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	r11.field.mode = sn_group->mode;
+	r11.field.slot = queue->sn_slot;
+	r11.field.grp  = sn_group->id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+	r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+	r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		r0.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+		r1.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+		r2.field.vqid = queue->id.virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+			    r2.val);
+	}
+
+	r13.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+				  struct dlb2_ldb_queue *queue,
+				  struct dlb2_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb2_sn_group_full(group)) {
+			slot = dlb2_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no sequence number slots available\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+				struct dlb2_hw_domain *domain,
+				struct dlb2_ldb_queue *queue,
+				struct dlb2_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_ldb_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (dlb2_list_empty(&domain->avail_ldb_queues)) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb2_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_ldb_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		    args->num_sequence_numbers);
+	DLB2_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		    args->num_qid_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		    args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+
+	dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			    __func__, __LINE__);
+		return ret;
+	}
+
+	dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_queues, &queue->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_queues, &queue->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+						unsigned int group_id,
+						unsigned long val)
+{
+	DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+	DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+	DLB2_HW_DBG(hw, "\tValue:    %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val)
+{
+	u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+	union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+	struct dlb2_sn_group *group;
+	int mode;
+
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	group = &hw->rsrcs.sn_groups[group_id];
+
+	/*
+	 * Once the first load-balanced queue using an SN group is configured,
+	 * the group cannot be changed.
+	 */
+	if (group->slot_use_bitmap != 0)
+		return -EPERM;
+
+	for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+		if (val == valid_allocations[mode])
+			break;
+
+	if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+		return -EINVAL;
+
+	group->mode = mode;
+	group->sequence_numbers_per_queue = val;
+
+	r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+	r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+	DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+	dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 285ad2c..14bb04b 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -613,3 +613,13 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
 {
 	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_ldb_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 9310f72..a6cd178 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -150,6 +150,84 @@ dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
 		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
 }
 
+static int
+dlb2_pf_ldb_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_ldb_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_ldb_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw_dev *handle,
+			 struct dlb2_get_sn_occupancy_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_number_occupancy(&dlb2_dev->hw,
+						       args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_numbers(&dlb2_dev->hw, args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_set_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_set_group_sequence_numbers(&dlb2_dev->hw, args->group,
+					      args->num);
+
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -161,6 +239,10 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
+	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
+	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
+	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v6 13/23] event/dlb2: add port setup

[Timothy McDaniel]

Configure the load balanded (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  75 +++
 drivers/event/dlb2/dlb2.c                  | 498 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   8 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 922 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  28 +
 drivers/event/dlb2/pf/dlb2_pf.c            | 180 ++++++
 7 files changed, 1719 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index b154cac..bbd6ac8 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -82,6 +82,81 @@ The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
 the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
 load-balanced queues can use the full 16-bit flow ID range.
 
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 8c1e06d..7898e42 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -999,6 +999,503 @@ dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
 	return ret;
 }
 
+static int
+dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_cq_pop_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_cq_pop_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL)	{
+		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->consume_qe = qe;
+
+	qe->qe_valid = 0;
+	qe->qe_frag = 0;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
+	 * and so on.
+	 */
+	qe->tokens = 0;	/* set at run time */
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_enqueue_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_enqueue_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL) {
+		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->int_arm_qe = qe;
+
+	/* V2 - INT ARM is CQ_TOKEN + FRAG */
+	qe->qe_valid = 0;
+	qe->qe_frag = 1;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
+{
+	int ret, sz;
+
+	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct dlb2_enqueue_qe);
+
+	qm_port->qe4 = rte_zmalloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
+
+	if (qm_port->qe4 == NULL) {
+		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	ret = dlb2_init_consume_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	return 0;
+
+error_exit:
+
+	dlb2_free_qe_mem(qm_port);
+
+	return ret;
+}
+
+static int
+dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_ldb_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_CQ_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	cfg.cq_history_list_size = DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	if (handle->cos_id == DLB2_COS_DEFAULT)
+		cfg.cos_id = 0;
+	else
+		cfg.cos_id = handle->cos_id;
+
+	cfg.cos_strict = 0;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* If there are no directed ports, the kernel driver will ignore this
+	 * port's directed credit settings. Don't use enqueue_depth if it would
+	 * require more directed credits than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2; /* back ptr */
+	/*
+	 * Allocate and init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_ldb_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(qm_port->cfg.ldb));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     qm_port->ldb_credits,
+		     qm_port->dir_credits);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
+
+	return ret;
+}
+
+static void
+dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port)
+{
+	struct dlb2_eventdev_queue *ev_queue;
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (!ev_port->link[i].valid)
+			continue;
+
+		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
+
+		ev_port->link[i].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+}
+
+static int
+dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (dlb2 == NULL || handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* Directed queues are configured at link time. */
+	cfg.queue_id = -1;
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* Don't use enqueue_depth if it would require more directed credits
+	 * than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2;  /* back ptr */
+
+	/*
+	 * Init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE to be aligned
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_dir_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = cfg.cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(qm_port->cfg.dir));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     dir_credit_high_watermark,
+		     ldb_credit_high_watermark);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_setup(struct rte_eventdev *dev,
+			 uint8_t ev_port_id,
+			 const struct rte_event_port_conf *port_conf)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_eventdev_port *ev_port;
+	int ret;
+
+	if (dev == NULL || port_conf == NULL) {
+		DLB2_LOG_ERR("Null parameter\n");
+		return -EINVAL;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
+		return -EINVAL;
+
+	if (port_conf->dequeue_depth >
+		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
+	    port_conf->enqueue_depth >
+		evdev_dlb2_default_info.max_event_port_enqueue_depth)
+		return -EINVAL;
+
+	ev_port = &dlb2->ev_ports[ev_port_id];
+	/* configured? */
+	if (ev_port->setup_done) {
+		DLB2_LOG_ERR("evport %d is already configured\n", ev_port_id);
+		return -EINVAL;
+	}
+
+	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
+		RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (!ev_port->qm_port.is_directed) {
+		ret = dlb2_hw_create_ldb_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the lB port ve portId=%d\n",
+				     ev_port_id);
+
+			return ret;
+		}
+	} else {
+		ret = dlb2_hw_create_dir_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the DIR port\n");
+			return ret;
+		}
+	}
+
+	/* Save off port config for reconfig */
+	ev_port->conf = *port_conf;
+
+	ev_port->id = ev_port_id;
+	ev_port->enq_configured = true;
+	ev_port->setup_done = true;
+	ev_port->inflight_max = port_conf->new_event_threshold;
+	ev_port->implicit_release = !(port_conf->event_port_cfg &
+		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
+	ev_port->outstanding_releases = 0;
+	ev_port->inflight_credits = 0;
+	ev_port->credit_update_quanta = RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
+	ev_port->dlb2 = dlb2; /* reverse link */
+
+	/* Tear down pre-existing port->queue links */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
+
+	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1009,6 +1506,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
+		.port_setup       = dlb2_eventdev_port_setup,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 4c07574..01818c7 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -43,3 +43,11 @@ int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				   struct dlb2_get_sn_occupancy_args *args);
+
+int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_dir_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4c88fe0..a6b923a 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -42,4 +42,12 @@ extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				  struct dlb2_get_sn_occupancy_args *args);
+
+extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_ldb_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_dir_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index e6ea0d7..a633d3e 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3973,3 +3973,925 @@ int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_ldb_port *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_pp2vas r0 = { {0} };
+	union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+		union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_ldb_port *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_ldb_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+	union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+	union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+	union dlb2_chp_hist_list_lim r5 = { {0} };
+	union dlb2_chp_hist_list_base r6 = { {0} };
+	union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+	union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+	union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+	union dlb2_sys_ldb_cq_at r10 = { {0} };
+	union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+	union dlb2_chp_ldb_cq2vas r12 = { {0} };
+	union dlb2_lsp_cq2priov r13 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r14.field.token_count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    r14.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	r5.field.limit = port->hist_list_entry_limit - 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+	r6.field.base = port->hist_list_entry_base;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	r7.field.limit = args->cq_history_list_size;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+	r8.field.push_ptr = r6.field.base;
+	r8.field.generation = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    r8.val);
+
+	r9.field.pop_ptr = r6.field.base;
+	r9.field.generation = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+	/* Disable the port's QID mappings */
+	r13.field.v = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+	return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_ldb_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb2_ldb_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+	if (ret < 0)
+		return ret;
+
+	dlb2_ldb_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_ldb_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		    args->cq_history_list_size);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+	DLB2_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		    args->cos_strict);
+}
+
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_ldb_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+		resp->status = DLB2_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		if (dlb2_list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	} else {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			if (!dlb2_list_empty(&domain->avail_ldb_ports[i]))
+				break;
+		}
+
+		if (i == DLB2_NUM_COS_DOMAINS) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ *	its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	int ret, cos_id, i;
+
+	dlb2_log_create_ldb_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->cos_strict) {
+		cos_id = args->cos_id;
+
+		port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+					  typeof(*port));
+	} else {
+		int idx;
+
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+			port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+						  typeof(*port));
+			if (port)
+				break;
+		}
+
+		cos_id = idx;
+	}
+
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (port->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_ldb_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_ports[cos_id], &port->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_ports[cos_id], &port->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_dir_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+}
+
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+			    bool vdev_req,
+			    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+
+	return NULL;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_dir_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the queue is already configured, validate
+	 * the queue ID, its domain, and whether the queue is configured.
+	 */
+	if (args->queue_id != -1) {
+		struct dlb2_dir_pq_pair *queue;
+
+		queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+						    vdev_req,
+						    domain);
+
+		if (queue == NULL || queue->domain_id.phys_id !=
+				domain->id.phys_id ||
+				!queue->queue_configured) {
+			resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the port's queue is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->queue_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_dir_pq_pair *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_dir_pp2vas r0 = { {0} };
+	union dlb2_sys_dir_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+		union dlb2_sys_dir_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_dir_pq_pair *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_dir_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+	union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+	union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+	union dlb2_sys_dir_cq_fmt r9 = { {0} };
+	union dlb2_sys_dir_cq_at r10 = { {0} };
+	union dlb2_sys_dir_cq_pasid r11 = { {0} };
+	union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r13.field.count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    r13.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.disable_wb_opt = 0;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	r9.field.keep_pf_ppid = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+	return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_dir_pq_pair *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_dir_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb2_dir_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+
+	if (ret < 0)
+		return ret;
+
+	dlb2_dir_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ *	and queue. The port/queue pair have the same ID and name.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *port;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (args->queue_id != -1)
+		port = dlb2_get_domain_used_dir_pq(args->queue_id,
+						   vdev_req,
+						   domain);
+	else
+		port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					  typeof(*port));
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_dir_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs, &port->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs, &port->domain_list);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 14bb04b..2a089f6 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -623,3 +623,31 @@ dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_ldb_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
+
+int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_dir_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a6cd178..2a82b8f 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -228,6 +228,184 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static void *
+dlb2_alloc_coherent_aligned(const struct rte_memzone **mz, uintptr_t *phys,
+			    size_t size, int align)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t core_id = rte_lcore_id();
+	unsigned int socket_id;
+
+	snprintf(mz_name, sizeof(mz_name) - 1, "event_dlb2_pf_%lx",
+		 (unsigned long)rte_get_timer_cycles());
+	if (core_id == (unsigned int)LCORE_ID_ANY)
+		core_id = rte_get_main_lcore();
+	socket_id = rte_lcore_to_socket_id(core_id);
+	*mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
+					 RTE_MEMZONE_IOVA_CONTIG, align);
+	if (*mz == NULL) {
+		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu bytes - %s\n",
+			     size, rte_strerror(rte_errno));
+		*phys = 0;
+		return NULL;
+	}
+	*phys = (*mz)->iova;
+	return (*mz)->addr;
+}
+
+static int
+dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_ldb_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret, cq_alloc_depth;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = false;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	cq_alloc_depth = RTE_MAX(cfg->cq_depth, DLB2_MIN_HARDWARE_CQ_DEPTH);
+	alloc_sz = cq_alloc_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].cq_base = (void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+	return ret;
+}
+
+static int
+dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_dir_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = true;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	alloc_sz = cfg->cq_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
+		(void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -240,6 +418,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

Re: [dpdk-dev] [PATCH v6 13/23] event/dlb2: add port setup

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, October 30, 2020 1:29 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com; thomas@monjalon.net
> Subject: [PATCH v6 13/23] event/dlb2: add port setup
> 
> Configure the load balanded (ldb) or directed (dir) port.
> The consumer queue (CQ) and producer port (PP) are also
> set up here.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Typo in the commit message ("balanded" -> "balanced"), but that can
be fixed in the merge.

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v6 14/23] event/dlb2: add port link

[Timothy McDaniel]

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 302 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 633 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
 6 files changed, 1007 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7898e42..5068d6d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1496,6 +1496,307 @@ dlb2_eventdev_port_setup(struct rte_eventdev *dev,
 	return 0;
 }
 
+static int16_t
+dlb2_hw_map_ldb_qid_to_port(struct dlb2_hw_dev *handle,
+			    uint32_t qm_port_id,
+			    uint16_t qm_qid,
+			    uint8_t priority)
+{
+	struct dlb2_map_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	/* Build message */
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+	cfg.priority = EV_TO_DLB2_PRIO(priority);
+
+	ret = dlb2_iface_map_qid(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: map qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		DLB2_LOG_ERR("dlb2: grp=%d, qm_port=%d, qm_qid=%d prio=%d\n",
+			     handle->domain_id, cfg.port_id,
+			     cfg.qid,
+			     cfg.priority);
+	} else {
+		DLB2_LOG_DBG("dlb2: mapped queue %d to qm_port %d\n",
+			     qm_qid, qm_port_id);
+	}
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port,
+			  struct dlb2_eventdev_queue *ev_queue,
+			  uint8_t priority)
+{
+	int first_avail = -1;
+	int ret, i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid) {
+			if (ev_port->link[i].queue_id == ev_queue->id &&
+			    ev_port->link[i].priority == priority) {
+				if (ev_port->link[i].mapped)
+					return 0; /* already mapped */
+				first_avail = i;
+			}
+		} else if (first_avail == -1)
+			first_avail = i;
+	}
+	if (first_avail == -1) {
+		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
+			     ev_port->qm_port.id);
+		return -EINVAL;
+	}
+
+	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
+					  ev_port->qm_port.id,
+					  ev_queue->qm_queue.id,
+					  priority);
+
+	if (!ret)
+		ev_port->link[first_avail].mapped = true;
+
+	return ret;
+}
+
+static int32_t
+dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 int32_t qm_port_id)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_queue_args cfg;
+	int32_t ret;
+
+	/* The directed port is always configured before its queue */
+	cfg.port_id = qm_port_id;
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_dir_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	int32_t qm_qid;
+
+	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port->qm_port.id);
+
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the DIR queue\n");
+		return qm_qid;
+	}
+
+	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int
+dlb2_do_port_link(struct rte_eventdev *dev,
+		  struct dlb2_eventdev_queue *ev_queue,
+		  struct dlb2_eventdev_port *ev_port,
+		  uint8_t prio)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int err;
+
+	/* Don't link until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	if (ev_queue->qm_queue.is_directed)
+		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue, ev_port);
+	else
+		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue, prio);
+
+	if (err) {
+		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
+			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
+			     ev_queue->id, ev_port->id);
+
+		rte_errno = err;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
+			uint8_t queue_id,
+			bool link_exists,
+			int index)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	bool port_is_dir, queue_is_dir;
+
+	if (queue_id > dlb2->num_queues) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	ev_queue = &dlb2->ev_queues[queue_id];
+
+	if (!ev_queue->setup_done &&
+	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	port_is_dir = ev_port->qm_port.is_directed;
+	queue_is_dir = ev_queue->qm_queue.is_directed;
+
+	if (port_is_dir != queue_is_dir) {
+		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
+			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
+			     port_is_dir ? "DIR" : "LDB", ev_port->id);
+
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if there is space for the requested link */
+	if (!link_exists && index == -1) {
+		DLB2_LOG_ERR("no space for new link\n");
+		rte_errno = -ENOSPC;
+		return -1;
+	}
+
+	/* Check if the directed port is already linked */
+	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if the directed queue is already linked */
+	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
+			     ev_queue->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
+			const uint8_t queues[], const uint8_t priorities[],
+			uint16_t nb_links)
+
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i, j;
+
+	RTE_SET_USED(dev);
+
+	if (ev_port == NULL) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!ev_port->setup_done &&
+	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
+	 * queues pointer.
+	 */
+	if (nb_links == 0) {
+		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
+		     nb_links,
+		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
+		     ev_port->id);
+
+	for (i = 0; i < nb_links; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		uint8_t queue_id, prio;
+		bool found = false;
+		int index = -1;
+
+		queue_id = queues[i];
+		prio = priorities[i];
+
+		/* Check if the link already exists. */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].valid) {
+				if (ev_port->link[j].queue_id == queue_id) {
+					found = true;
+					index = j;
+					break;
+				}
+			} else if (index == -1) {
+				index = j;
+			}
+
+		/* could not link */
+		if (index == -1)
+			break;
+
+		/* Check if already linked at the requested priority */
+		if (found && ev_port->link[j].priority == prio)
+			continue;
+
+		if (dlb2_validate_port_link(ev_port, queue_id, found, index))
+			break; /* return index of offending queue */
+
+		ev_queue = &dlb2->ev_queues[queue_id];
+
+		if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+			break; /* return index of offending queue */
+
+		ev_queue->num_links++;
+
+		ev_port->link[index].queue_id = queue_id;
+		ev_port->link[index].priority = prio;
+		ev_port->link[index].valid = true;
+		/* Entry already exists?  If so, then must be prio change */
+		if (!found)
+			ev_port->num_links++;
+	}
+	return i;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1507,6 +1808,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_link        = dlb2_eventdev_port_link,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 01818c7..7d1ba73 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -51,3 +51,9 @@ int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 				  struct dlb2_create_dir_port_args *cfg,
 				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_dir_queue_args *cfg);
+
+int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+			  struct dlb2_map_qid_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index a6b923a..4eddd14 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -50,4 +50,10 @@ extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 struct dlb2_create_dir_port_args *cfg,
 					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+					struct dlb2_create_dir_queue_args *cfg);
+
+extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+				 struct dlb2_map_qid_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index a633d3e..14ad262 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -4895,3 +4895,636 @@ int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_dir_pq_pair *queue,
+				     struct dlb2_create_dir_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_dir_vasqid_v r0 = { {0} };
+	union dlb2_sys_dir_qid_its r1 = { {0} };
+	union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+	union dlb2_sys_dir_qid_v r5 = { {0} };
+
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r0.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r1.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+		    r1.val);
+
+	r2.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+		    r2.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		r3.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+		r4.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+	}
+
+	r5.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+	queue->queue_configured = true;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_dir_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_dir_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = dlb2_get_domain_used_dir_pq(args->port_id,
+						   vdev_req,
+						   domain);
+
+		if (port == NULL || port->domain_id.phys_id !=
+				domain->id.phys_id || !port->port_configured) {
+			resp->status = DLB2_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the queue's port is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->port_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->port_id != -1)
+		queue = dlb2_get_domain_used_dir_pq(args->port_id,
+						    vdev_req,
+						    domain);
+	else
+		queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					   typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &queue->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs,
+			      &queue->domain_list);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot,
+					      struct dlb2_map_qid_args *args)
+{
+	union dlb2_lsp_cq2priov r0;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+	r0.field.v |= 1 << slot;
+	r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+	dlb2_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+					      struct dlb2_ldb_queue *queue,
+					      struct dlb2_cmd_response *resp)
+{
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB2_QUEUE_UNMAPPED;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+			  bool vdev_req,
+			  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter)
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+			      bool vdev_req,
+			      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+
+		DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+	}
+
+	return NULL;
+}
+
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+				    u32 domain_id,
+				    struct dlb2_map_qid_args *args,
+				    struct dlb2_cmd_response *resp,
+				    bool vdev_req,
+				    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	struct dlb2_ldb_queue *queue;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB2_QID_PRIORITIES) {
+		resp->status = DLB2_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_map_qid_args *args,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	DLB2_HW_DBG(hw, "\tPriority:  %d\n",
+		    args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	int ret, i, id;
+	u8 prio;
+
+	dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_map_qid_args(hw,
+				       domain_id,
+				       args,
+				       resp,
+				       vdev_req,
+				       vdev_id);
+	if (ret)
+		return ret;
+
+	prio = args->priority;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	if (port->num_pending_removals)
+		dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].pending_priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+		if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb2_qid_map_state st;
+
+			if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+				DLB2_HW_ERR(hw,
+					    "[%s():%d] Internal error: port slot tracking failed\n",
+					    __func__, __LINE__);
+				return -EFAULT;
+			}
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb2_port_slot_state_transition(hw, port, queue,
+							      i, st);
+			if (ret)
+				return ret;
+
+			DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 2a089f6..a010e25 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -651,3 +651,13 @@ dlb2_pf_create_dir_port(struct dlb2_hw *hw,
 				       NOT_VF_REQ,
 				       PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_dir_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 2a82b8f..37be4b1 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -406,6 +406,54 @@ dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_dir_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_dir_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_dir_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
+		struct dlb2_map_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_map_qid(&dlb2_dev->hw,
+			      handle->domain_id,
+			      cfg,
+			      &response,
+			      false,
+			      0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -419,7 +467,9 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
 	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
+	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v6 15/23] event/dlb2: add port unlink and port unlinks in progress

[Timothy McDaniel]

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 163 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 283 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  51 ++++++
 5 files changed, 509 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 5068d6d..c221a61 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1797,6 +1797,166 @@ dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
 	return i;
 }
 
+static int16_t
+dlb2_hw_unmap_ldb_qid_from_port(struct dlb2_hw_dev *handle,
+				uint32_t qm_port_id,
+				uint16_t qm_qid)
+{
+	struct dlb2_unmap_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+
+	ret = dlb2_iface_unmap_qid(handle, &cfg);
+	if (ret < 0)
+		DLB2_LOG_ERR("dlb2: unmap qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_detach_ldb(struct dlb2_eventdev *dlb2,
+			    struct dlb2_eventdev_port *ev_port,
+			    struct dlb2_eventdev_queue *ev_queue)
+{
+	int ret, i;
+
+	/* Don't unlink until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid &&
+		    ev_port->link[i].queue_id == ev_queue->id)
+			break; /* found */
+	}
+
+	/* This is expected with eventdev API!
+	 * It blindly attemmpts to unmap all queues.
+	 */
+	if (i == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_LOG_DBG("dlb2: ignoring LB QID %d not mapped for qm_port %d.\n",
+			     ev_queue->qm_queue.id,
+			     ev_port->qm_port.id);
+		return 0;
+	}
+
+	ret = dlb2_hw_unmap_ldb_qid_from_port(&dlb2->qm_instance,
+					      ev_port->qm_port.id,
+					      ev_queue->qm_queue.id);
+	if (!ret)
+		ev_port->link[i].mapped = false;
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
+			  uint8_t queues[], uint16_t nb_unlinks)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (queues == NULL || nb_unlinks == 0) {
+		DLB2_LOG_DBG("dlb2: queues is NULL or nb_unlinks is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	/* FIXME: How to handle unlink on a directed port? */
+	if (ev_port->qm_port.is_directed) {
+		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
+			     ev_port->id);
+		rte_errno = 0;
+		return nb_unlinks; /* as if success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	for (i = 0; i < nb_unlinks; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		int ret, j;
+
+		if (queues[i] >= dlb2->num_queues) {
+			DLB2_LOG_ERR("dlb2: invalid queue id %d\n", queues[i]);
+			rte_errno = -EINVAL;
+			return i; /* return index of offending queue */
+		}
+
+		ev_queue = &dlb2->ev_queues[queues[i]];
+
+		/* Does a link exist? */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].queue_id == queues[i] &&
+			    ev_port->link[j].valid)
+				break;
+
+		if (j == DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+			continue;
+
+		ret = dlb2_event_queue_detach_ldb(dlb2, ev_port, ev_queue);
+		if (ret) {
+			DLB2_LOG_ERR("unlink err=%d for port %d queue %d\n",
+				     ret, ev_port->id, queues[i]);
+			rte_errno = -ENOENT;
+			return i; /* return index of offending queue */
+		}
+
+		ev_port->link[j].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+
+	return nb_unlinks;
+}
+
+static int
+dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
+				       void *event_port)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	struct dlb2_pending_port_unmaps_args cfg;
+	int ret;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	cfg.port_id = ev_port->qm_port.id;
+	dlb2 = ev_port->dlb2;
+	handle = &dlb2->qm_instance;
+	ret = dlb2_iface_pending_port_unmaps(handle, &cfg);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: num_unlinks_in_progress ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1809,6 +1969,9 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
 		.port_link        = dlb2_eventdev_port_link,
+		.port_unlink      = dlb2_eventdev_port_unlink,
+		.port_unlinks_in_progress =
+				    dlb2_eventdev_port_unlinks_in_progress,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 7d1ba73..bd241f3 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -57,3 +57,9 @@ int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 			  struct dlb2_map_qid_args *cfg);
+
+int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+			    struct dlb2_unmap_qid_args *cfg);
+
+int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4eddd14..e892036 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -56,4 +56,10 @@ extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 				 struct dlb2_map_qid_args *cfg);
+
+extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+				   struct dlb2_unmap_qid_args *cfg);
+
+extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 14ad262..4a3d96d 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5528,3 +5528,286 @@ int dlb2_hw_map_qid(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_unmap_qid_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+		DLB2_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			    hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+				      u32 domain_id,
+				      struct dlb2_unmap_qid_args *args,
+				      struct dlb2_cmd_response *resp,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	struct dlb2_ldb_port *port;
+	int slot;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			    __func__, args->qid);
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		return 0;
+
+	resp->status = DLB2_ST_INVALID_QID;
+	return -EINVAL;
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	bool unmap_complete;
+	int i, ret, id;
+
+	dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_unmap_qid_args(hw,
+					 domain_id,
+					 args,
+					 resp,
+					 vdev_req,
+					 vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+		st = DLB2_QUEUE_UNMAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		st = DLB2_QUEUE_UNMAP_IN_PROG;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB2_QUEUE_MAPPED;
+	if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: no available CQ slots\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asynchronous procedure. It requires
+	 * stopping the DLB2 from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb2_ldb_port_cq_disable(hw, port);
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete && !os_worker_active(hw))
+		os_schedule_work(hw);
+
+unmap_qid_done:
+	resp->status = 0;
+
+	return 0;
+}
+
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+				  struct dlb2_pending_port_unmaps_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+
+	dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 37be4b1..33ea73e 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -454,6 +454,55 @@ dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw_dev *handle,
+		  struct dlb2_unmap_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_unmap_qid(&dlb2_dev->hw,
+				handle->domain_id,
+				cfg,
+				&response,
+				false,
+				0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
+			    struct dlb2_pending_port_unmaps_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_pending_port_unmaps(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -470,6 +519,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
+	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v6 16/23] event/dlb2: add eventdev start

[Timothy McDaniel]

Add support for the eventdev start entry point.
We delay initializing some resources until
eventdev start, since the number of linked queues can be
used to determine if we are dealing with a ldb or dir resource.
If this is a device restart, then the previous configuration
will be reapplied.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 133 +++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   3 +
 drivers/event/dlb2/dlb2_iface.h            |   3 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 123 ++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +++
 drivers/event/dlb2/pf/dlb2_pf.c            |  25 ++++++
 6 files changed, 297 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index c221a61..051836d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1957,6 +1957,138 @@ dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
 	return cfg.response.id;
 }
 
+static int
+dlb2_eventdev_reapply_configuration(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret, i;
+
+	/* If an event queue or port was previously configured, but hasn't been
+	 * reconfigured, reapply its original configuration.
+	 */
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+
+		ev_queue = &dlb2->ev_queues[i];
+
+		if (ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_queue_setup(dev, i, &ev_queue->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure queue %d", i);
+			return ret;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+
+		if (ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_port_setup(dev, i, &ev_port->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure ev_port %d",
+				     i);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_apply_port_links(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int i;
+
+	/* Perform requested port->queue links */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+		int j;
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			struct dlb2_eventdev_queue *ev_queue;
+			uint8_t prio, queue_id;
+
+			if (!ev_port->link[j].valid)
+				continue;
+
+			prio = ev_port->link[j].priority;
+			queue_id = ev_port->link[j].queue_id;
+
+			if (dlb2_validate_port_link(ev_port, queue_id, true, j))
+				return -EINVAL;
+
+			ev_queue = &dlb2->ev_queues[queue_id];
+
+			if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_start(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_start_domain_args cfg;
+	int ret, i;
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_ERR("bad state %d for dev_start\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return -EINVAL;
+	}
+	dlb2->run_state = DLB2_RUN_STATE_STARTING;
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	/* If the device was configured more than once, some event ports and/or
+	 * queues may need to be reconfigured.
+	 */
+	ret = dlb2_eventdev_reapply_configuration(dev);
+	if (ret)
+		return ret;
+
+	/* The DLB PMD delays port links until the device is started. */
+	ret = dlb2_eventdev_apply_port_links(dev);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		if (!dlb2->ev_ports[i].setup_done) {
+			DLB2_LOG_ERR("dlb2: port %d not setup", i);
+			return -ESTALE;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0) {
+			DLB2_LOG_ERR("dlb2: queue %d is not linked", i);
+			return -ENOLINK;
+		}
+	}
+
+	ret = dlb2_iface_sched_domain_start(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: sched_domain_start ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STARTED;
+	DLB2_LOG_DBG("dlb2: sched_domain_start completed OK\n");
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1964,6 +2096,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.dev_start        = dlb2_eventdev_start,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index bd241f3..a86191d 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -63,3 +63,6 @@ int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				     struct dlb2_start_domain_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index e892036..7ead374 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -62,4 +62,7 @@ extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				struct dlb2_start_domain_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 4a3d96d..c14a2f3 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5811,3 +5811,126 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 struct dlb2_cmd_response *resp,
+					 bool vdev_req,
+					 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_start_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 start domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_hw_start_domain() - Lock the domain configuration
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @arg: User-provided arguments (unused, here for ioctl callback template).
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int
+dlb2_hw_start_domain(struct dlb2_hw *hw,
+		     u32 domain_id,
+		     __attribute((unused)) struct dlb2_start_domain_args *arg,
+		     struct dlb2_cmd_response *resp,
+		     bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_queue;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+	RTE_SET_USED(arg);
+	RTE_SET_USED(iter);
+
+	dlb2_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb2_verify_start_domain_args(hw,
+					    domain_id,
+					    resp,
+					    vdev_req,
+					    vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB2 will drop all
+	 * incoming traffic to those queues.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, ldb_queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r0.val);
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+	}
+
+	dlb2_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index a010e25..aee8336 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -661,3 +661,13 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_start_domain(struct dlb2_hw *hw,
+		     u32 id,
+		     struct dlb2_start_domain_args *args,
+		     struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_start_domain(hw, id, args, resp, NOT_VF_REQ,
+				    PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 33ea73e..fff2e57 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -503,6 +503,30 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
 
 	return ret;
 }
+
+static int
+dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
+			   struct dlb2_start_domain_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_start_domain(&dlb2_dev->hw,
+				   handle->domain_id,
+				   cfg,
+				   &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -520,6 +544,7 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v6 17/23] event/dlb2: add enqueue and its burst variants

[Timothy McDaniel]

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst | 118 +++++++++
 drivers/event/dlb2/dlb2.c     | 578 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 696 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index bbd6ac8..aa8bf01 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -163,6 +163,124 @@ Flow ID
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Hardware Credits
+~~~~~~~~~~~~~~~~
+
+DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
+event storage, with each unit of storage represented by a credit. A port spends
+a credit to enqueue an event, and hardware refills the ports with credits as the
+events are scheduled to ports. Refills come from credit pools, and each port is
+a member of a load-balanced credit pool and a directed credit pool. The
+load-balanced credits are used to enqueue to load-balanced queues, and directed
+credits are used for directed queues.
+
+A DLB2 eventdev contains one load-balanced and one directed credit pool. These
+pools' sizes are controlled by the nb_events_limit field in struct
+rte_event_dev_config. The load-balanced pool is sized to contain
+nb_events_limit credits, and the directed pool is sized to contain
+nb_events_limit/4 credits. The directed pool size can be overridden with the
+num_dir_credits vdev argument, like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,num_dir_credits=<value>
+
+This can be used if the default allocation is too low or too high for the
+specific application needs. The PMD also supports a vdev arg that limits the
+max_num_events reported by rte_event_dev_info_get():
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,max_num_events=<value>
+
+By default, max_num_events is reported as the total available load-balanced
+credits. If multiple DLB2-based applications are being used, it may be desirable
+to control how many load-balanced credits each application uses, particularly
+when application(s) are written to configure nb_events_limit equal to the
+reported max_num_events.
+
+Each port is a member of both credit pools. A port's credit allocation is
+defined by its low watermark, high watermark, and refill quanta. These three
+parameters are calculated by the dlb PMD like so:
+
+- The load-balanced high watermark is set to the port's enqueue_depth.
+  The directed high watermark is set to the minimum of the enqueue_depth and
+  the directed pool size divided by the total number of ports.
+- The refill quanta is set to half the high watermark.
+- The low watermark is set to the minimum of 16 and the refill quanta.
+
+When the eventdev is started, each port is pre-allocated a high watermark's
+worth of credits. For example, if an eventdev contains four ports with enqueue
+depths of 32 and a load-balanced credit pool size of 4096, each port will start
+with 32 load-balanced credits, and there will be 3968 credits available to
+replenish the ports. Thus, a single port is not capable of enqueueing up to the
+nb_events_limit (without any events being dequeued), since the other ports are
+retaining their initial credit allocation; in short, all ports must enqueue in
+order to reach the limit.
+
+If a port attempts to enqueue and has no credits available, the enqueue
+operation will fail and the application must retry the enqueue. Credits are
+replenished asynchronously by the DLB2 hardware.
+
+Software Credits
+~~~~~~~~~~~~~~~~
+
+The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
+credit scheme on top of the hardware credit scheme in order to comply with
+the per-port backpressure described in the eventdev API.
+
+The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
+credit when it enqueues to a queue, and credits are later replenished after the
+events are dequeued and released.
+
+In the software credit scheme, a credit is consumed when a new (.op =
+RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
+replenished when the event is released from the system (either explicitly with
+RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
+
+In this model, an event is "in the system" from its first enqueue into eventdev
+until it is last dequeued. If the event goes through multiple event queues, it
+is still considered "in the system" while a worker thread is processing it.
+
+A port will fail to enqueue if the number of events in the system exceeds its
+``new_event_threshold`` (specified at port setup time). A port will also fail
+to enqueue if it lacks enough hardware credits to enqueue; load-balanced
+credits are used to enqueue to a load-balanced queue, and directed credits are
+used to enqueue to a directed queue.
+
+The out-of-credit situations are typically transient, and an eventdev
+application using the DLB2 ought to retry its enqueues if they fail.
+If enqueue fails, DLB2 PMD sets rte_errno as follows:
+
+- -ENOSPC: Credit exhaustion (either hardware or software)
+- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
+
+Depending on the pipeline the application has constructed, it's possible to
+enter a credit deadlock scenario wherein the worker thread lacks the credit
+to enqueue an event, and it must dequeue an event before it can recover the
+credit. If the worker thread retries its enqueue indefinitely, it will not
+make forward progress. Such deadlock is possible if the application has event
+"loops", in which an event in dequeued from queue A and later enqueued back to
+queue A.
+
+Due to this, workers should stop retrying after a time, release the events it
+is attempting to enqueue, and dequeue more events. It is important that the
+worker release the events and don't simply set them aside to retry the enqueue
+again later, because the port has limited history list size (by default, twice
+the port's dequeue_depth).
+
+Priority
+~~~~~~~~
+
+The DLB2 supports event priority and per-port queue service priority, as
+described in the eventdev header file. The DLB2 does not support 'global' event
+queue priority established at queue creation time.
+
+DLB2 supports 8 event and queue service priority levels. For both priority
+types, the PMD uses the upper three bits of the priority field to determine the
+DLB2 priority, discarding the 5 least significant bits. The 5 least significant
+event priority bits are not preserved when an event is enqueued.
+
 Reconfiguration
 ~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 051836d..7d2cf76 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2089,6 +2089,578 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
 	return 0;
 }
 
+static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
+	{
+		/* Load-balanced cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
+	},
+	{
+		/* Directed cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
+	},
+};
+
+static inline uint32_t
+dlb2_port_credits_get(struct dlb2_port *qm_port,
+		      enum dlb2_hw_queue_types type)
+{
+	uint32_t credits = *qm_port->credit_pool[type];
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	if (unlikely(credits < batch_size))
+		batch_size = credits;
+
+	if (likely(credits &&
+		   __atomic_compare_exchange_n(
+			qm_port->credit_pool[type],
+			&credits, credits - batch_size, false,
+			__ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
+		return batch_size;
+	else
+		return 0;
+}
+
+static inline void
+dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port)
+{
+	uint16_t quanta = ev_port->credit_update_quanta;
+
+	if (ev_port->inflight_credits >= quanta * 2) {
+		/* Replenish credits, saving one quanta for enqueues */
+		uint16_t val = ev_port->inflight_credits - quanta;
+
+		__atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits -= val;
+	}
+}
+
+static inline int
+dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
+						__ATOMIC_SEQ_CST);
+	const int num = 1;
+
+	if (unlikely(ev_port->inflight_max < sw_inflights)) {
+		DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
+		rte_errno = -ENOSPC;
+		return 1;
+	}
+
+	if (ev_port->inflight_credits < num) {
+		/* check if event enqueue brings ev_port over max threshold */
+		uint32_t credit_update_quanta = ev_port->credit_update_quanta;
+
+		if (sw_inflights + credit_update_quanta >
+				dlb2->new_event_limit) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_new_event_limit,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+
+		__atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
+				   __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits += (credit_update_quanta);
+
+		if (ev_port->inflight_credits < num) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_inflight_credits,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_ldb_credits == 0)) {
+		qm_port->cached_ldb_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_LDB_QUEUE);
+		if (unlikely(qm_port->cached_ldb_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
+			1);
+			DLB2_LOG_DBG("ldb credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_dir_credits == 0)) {
+		qm_port->cached_dir_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_DIR_QUEUE);
+		if (unlikely(qm_port->cached_dir_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
+			1);
+			DLB2_LOG_DBG("dir credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static __rte_always_inline void
+dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
+	      struct process_local_port_data *port_data)
+{
+	dlb2_movdir64b(port_data->pp_addr, qe4);
+}
+
+static inline int
+dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_cq_pop_qe *qe;
+
+	RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
+
+	qe = qm_port->consume_qe;
+
+	qe->tokens = num - 1;
+
+	/* No store fence needed since no pointer is being sent, and CQ token
+	 * pops can be safely reordered with other HCWs.
+	 */
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	dlb2_movntdq_single(port_data->pp_addr, qe);
+
+	DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
+
+	qm_port->owed_tokens = 0;
+
+	return 0;
+}
+
+static inline void
+dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
+		   bool do_sfence,
+		   struct process_local_port_data *port_data)
+{
+	/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
+	 * application writes complete before enqueueing the QE.
+	 */
+	if (do_sfence)
+		rte_wmb();
+
+	dlb2_pp_write(qm_port->qe4, port_data);
+}
+
+static inline void
+dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
+{
+	struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
+	int num = qm_port->owed_tokens;
+
+	qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
+	qe[idx].tokens = num - 1;
+
+	qm_port->owed_tokens = 0;
+}
+
+static inline void
+dlb2_event_build_hcws(struct dlb2_port *qm_port,
+		      const struct rte_event ev[],
+		      int num,
+		      uint8_t *sched_type,
+		      uint8_t *queue_id)
+{
+	struct dlb2_enqueue_qe *qe;
+	uint16_t sched_word[4];
+	__m128i sse_qe[2];
+	int i;
+
+	qe = qm_port->qe4;
+
+	sse_qe[0] = _mm_setzero_si128();
+	sse_qe[1] = _mm_setzero_si128();
+
+	switch (num) {
+	case 4:
+		/* Construct the metadata portion of two HCWs in one 128b SSE
+		 * register. HCW metadata is constructed in the SSE registers
+		 * like so:
+		 * sse_qe[0][63:0]:   qe[0]'s metadata
+		 * sse_qe[0][127:64]: qe[1]'s metadata
+		 * sse_qe[1][63:0]:   qe[2]'s metadata
+		 * sse_qe[1][127:64]: qe[3]'s metadata
+		 */
+
+		/* Convert the event operation into a command byte and store it
+		 * in the metadata:
+		 * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
+		 * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
+		 * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
+		 * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
+		 */
+#define DLB2_QE_CMD_BYTE 7
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[0].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[1].op],
+				DLB2_QE_CMD_BYTE + 8);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[2].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[3].op],
+				DLB2_QE_CMD_BYTE + 8);
+
+		/* Store priority, scheduling type, and queue ID in the sched
+		 * word array because these values are re-used when the
+		 * destination is a directed queue.
+		 */
+		sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
+				sched_type[0] << 8 |
+				queue_id[0];
+		sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
+				sched_type[1] << 8 |
+				queue_id[1];
+		sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
+				sched_type[2] << 8 |
+				queue_id[2];
+		sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
+				sched_type[3] << 8 |
+				queue_id[3];
+
+		/* Store the event priority, scheduling type, and queue ID in
+		 * the metadata:
+		 * sse_qe[0][31:16] = sched_word[0]
+		 * sse_qe[0][95:80] = sched_word[1]
+		 * sse_qe[1][31:16] = sched_word[2]
+		 * sse_qe[1][95:80] = sched_word[3]
+		 */
+#define DLB2_QE_QID_SCHED_WORD 1
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[0],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[1],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[2],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[3],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+
+		/* If the destination is a load-balanced queue, store the lock
+		 * ID. If it is a directed queue, DLB places this field in
+		 * bytes 10-11 of the received QE, so we format it accordingly:
+		 * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
+		 * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
+		 * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
+		 * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
+		 */
+#define DLB2_QE_LOCK_ID_WORD 2
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[0] == DLB2_SCHED_DIRECTED) ?
+					sched_word[0] : ev[0].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[1] == DLB2_SCHED_DIRECTED) ?
+					sched_word[1] : ev[1].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[2] == DLB2_SCHED_DIRECTED) ?
+					sched_word[2] : ev[2].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[3] == DLB2_SCHED_DIRECTED) ?
+					sched_word[3] : ev[3].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+
+		/* Store the event type and sub event type in the metadata:
+		 * sse_qe[0][15:0]  = flow_id[0]
+		 * sse_qe[0][79:64] = flow_id[1]
+		 * sse_qe[1][15:0]  = flow_id[2]
+		 * sse_qe[1][79:64] = flow_id[3]
+		 */
+#define DLB2_QE_EV_TYPE_WORD 0
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[0].sub_event_type << 8 |
+						ev[0].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[1].sub_event_type << 8 |
+						ev[1].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[2].sub_event_type << 8 |
+						ev[2].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[3].sub_event_type << 8 |
+						ev[3].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+
+		/* Store the metadata to memory (use the double-precision
+		 * _mm_storeh_pd because there is no integer function for
+		 * storing the upper 64b):
+		 * qe[0] metadata = sse_qe[0][63:0]
+		 * qe[1] metadata = sse_qe[0][127:64]
+		 * qe[2] metadata = sse_qe[1][63:0]
+		 * qe[3] metadata = sse_qe[1][127:64]
+		 */
+		_mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
+		_mm_storeh_pd((double *)&qe[1].u.opaque_data,
+			      (__m128d)sse_qe[0]);
+		_mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
+		_mm_storeh_pd((double *)&qe[3].u.opaque_data,
+			      (__m128d)sse_qe[1]);
+
+		qe[0].data = ev[0].u64;
+		qe[1].data = ev[1].u64;
+		qe[2].data = ev[2].u64;
+		qe[3].data = ev[3].u64;
+
+		break;
+	case 3:
+	case 2:
+	case 1:
+		/* At least one QE will be valid, so only zero out three */
+		qe[1].cmd_byte = 0;
+		qe[2].cmd_byte = 0;
+		qe[3].cmd_byte = 0;
+
+		for (i = 0; i < num; i++) {
+			qe[i].cmd_byte =
+				cmd_byte_map[qm_port->is_directed][ev[i].op];
+			qe[i].sched_type = sched_type[i];
+			qe[i].data = ev[i].u64;
+			qe[i].qid = queue_id[i];
+			qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
+			qe[i].lock_id = ev[i].flow_id;
+			if (sched_type[i] == DLB2_SCHED_DIRECTED) {
+				struct dlb2_msg_info *info =
+					(struct dlb2_msg_info *)&qe[i].lock_id;
+
+				info->qid = queue_id[i];
+				info->sched_type = DLB2_SCHED_DIRECTED;
+				info->priority = qe[i].priority;
+			}
+			qe[i].u.event_type.major = ev[i].event_type;
+			qe[i].u.event_type.sub = ev[i].sub_event_type;
+		}
+		break;
+	}
+}
+
+static inline int
+dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
+			struct dlb2_port *qm_port,
+			const struct rte_event ev[],
+			uint8_t *sched_type,
+			uint8_t *queue_id)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	uint16_t *cached_credits = NULL;
+	struct dlb2_queue *qm_queue;
+
+	ev_queue = &dlb2->ev_queues[ev->queue_id];
+	qm_queue = &ev_queue->qm_queue;
+	*queue_id = qm_queue->id;
+
+	/* Ignore sched_type and hardware credits on release events */
+	if (ev->op == RTE_EVENT_OP_RELEASE)
+		goto op_check;
+
+	if (!qm_queue->is_directed) {
+		/* Load balanced destination queue */
+
+		if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_ldb_credits;
+
+		switch (ev->sched_type) {
+		case RTE_SCHED_TYPE_ORDERED:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
+			if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
+				DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
+					     *queue_id);
+				rte_errno = -EINVAL;
+				return 1;
+			}
+			*sched_type = DLB2_SCHED_ORDERED;
+			break;
+		case RTE_SCHED_TYPE_ATOMIC:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
+			*sched_type = DLB2_SCHED_ATOMIC;
+			break;
+		case RTE_SCHED_TYPE_PARALLEL:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
+			if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
+				*sched_type = DLB2_SCHED_ORDERED;
+			else
+				*sched_type = DLB2_SCHED_UNORDERED;
+			break;
+		default:
+			DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
+			DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
+			rte_errno = -EINVAL;
+			return 1;
+		}
+	} else {
+		/* Directed destination queue */
+
+		if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_dir_credits;
+
+		DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
+
+		*sched_type = DLB2_SCHED_DIRECTED;
+	}
+
+op_check:
+	switch (ev->op) {
+	case RTE_EVENT_OP_NEW:
+		/* Check that a sw credit is available */
+		if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		ev_port->inflight_credits--;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_FORWARD:
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_RELEASE:
+		ev_port->inflight_credits++;
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+
+		/* Replenish s/w credits if enough are cached */
+		dlb2_replenish_sw_credits(dlb2, ev_port);
+		break;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+	if (ev->op != RTE_EVENT_OP_RELEASE) {
+		DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
+		DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
+	}
+#endif
+
+	return 0;
+}
+
+static inline uint16_t
+dlb2_event_enqueue_burst(void *event_port,
+			 const struct rte_event events[],
+			 uint16_t num)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct process_local_port_data *port_data;
+	int i, cnt;
+
+	RTE_ASSERT(ev_port->enq_configured);
+	RTE_ASSERT(events != NULL);
+
+	cnt = 0;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
+		int j = 0;
+
+		for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
+			const struct rte_event *ev = &events[i + j];
+
+			if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
+						    &sched_types[j],
+						    &queue_ids[j]))
+				break;
+		}
+
+		if (j == 0)
+			break;
+
+		dlb2_event_build_hcws(qm_port, &events[i], j,
+				      sched_types, queue_ids);
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+
+		if (j < DLB2_NUM_QES_PER_CACHE_LINE)
+			break;
+	}
+
+	return cnt;
+}
+
+static inline uint16_t
+dlb2_event_enqueue(void *event_port,
+		   const struct rte_event events[])
+{
+	return dlb2_event_enqueue_burst(event_port, events, 1);
+}
+
+static uint16_t
+dlb2_event_enqueue_new_burst(void *event_port,
+			     const struct rte_event events[],
+			     uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
+static uint16_t
+dlb2_event_enqueue_forward_burst(void *event_port,
+				 const struct rte_event events[],
+				 uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2112,7 +2684,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
 	};
 
+	/* Expose PMD's eventdev interface */
+
 	dev->dev_ops = &dlb2_eventdev_entry_ops;
+	dev->enqueue = dlb2_event_enqueue;
+	dev->enqueue_burst = dlb2_event_enqueue_burst;
+	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
+	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v6 18/23] event/dlb2: add dequeue and its burst variants

[Timothy McDaniel]

Add support for dequeue, dequeue_burst, ...

DLB2 does not currently support interrupts, but instead use
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB2 supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst |  21 ++
 drivers/event/dlb2/dlb2.c     | 769 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 790 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index aa8bf01..b9f57fd 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -314,6 +314,27 @@ The PMD does not support the following configuration sequences:
 This sequence is not supported because the event device must be reconfigured
 before its ports or queues can be.
 
+Deferred Scheduling
+~~~~~~~~~~~~~~~~~~~
+
+The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
+dequeued event before returning from rte_event_dequeue_burst(). This frees the
+corresponding entries in the CQ, which enables the DLB2 to schedule more events
+to it.
+
+To support applications seeking finer-grained scheduling control -- for example
+deferring scheduling to get the best possible priority scheduling and
+load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
+the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
+call. This mode only applies to load-balanced event ports with dequeue depth of
+1.
+
+To enable deferred scheduling, use the defer_sched vdev argument like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,defer_sched=on
+
 Atomic Inflights Allocation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7d2cf76..3d82ed5 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -26,6 +26,7 @@
 #include <rte_log.h>
 #include <rte_malloc.h>
 #include <rte_mbuf.h>
+#include <rte_power_intrinsics.h>
 #include <rte_prefetch.h>
 #include <rte_ring.h>
 #include <rte_string_fns.h>
@@ -2661,9 +2662,768 @@ dlb2_event_enqueue_forward_burst(void *event_port,
 	return dlb2_event_enqueue_burst(event_port, events, num);
 }
 
+static inline void
+dlb2_port_credits_inc(struct dlb2_port *qm_port, int num)
+{
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	/* increment port credits, and return to pool if exceeds threshold */
+	if (!qm_port->is_directed) {
+		qm_port->cached_ldb_credits += num;
+		if (qm_port->cached_ldb_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_LDB_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_ldb_credits -= batch_size;
+		}
+	} else {
+		qm_port->cached_dir_credits += num;
+		if (qm_port->cached_dir_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_DIR_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_dir_credits -= batch_size;
+		}
+	}
+}
+
+static inline bool
+dlb2_cq_is_empty(struct dlb2_port *qm_port)
+{
+	volatile struct dlb2_dequeue_qe *qe_ptr;
+	struct dlb2_dequeue_qe qe;
+
+	qe_ptr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	qe = qe_ptr[qm_port->cq_idx];
+
+	return (qe.cq_gen != qm_port->gen_bit);
+}
+
+static inline int
+dlb2_dequeue_wait(struct dlb2_eventdev *dlb2,
+		  struct dlb2_eventdev_port *ev_port,
+		  struct dlb2_port *qm_port,
+		  uint64_t timeout,
+		  uint64_t start_ticks)
+{
+	struct process_local_port_data *port_data;
+	uint64_t elapsed_ticks;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	elapsed_ticks = rte_get_timer_cycles() - start_ticks;
+
+	/* Wait/poll time expired */
+	if (elapsed_ticks >= timeout) {
+		return 1;
+	} else if (dlb2->umwait_allowed) {
+		volatile struct dlb2_dequeue_qe *cq_base;
+		union {
+			uint64_t raw_qe[2];
+			struct dlb2_dequeue_qe qe;
+		} qe_mask;
+		uint64_t expected_value;
+		volatile uint64_t *monitor_addr;
+
+		qe_mask.qe.cq_gen = 1; /* set mask */
+
+		cq_base = port_data->cq_base;
+		monitor_addr = (volatile uint64_t *)(volatile void *)
+			&cq_base[qm_port->cq_idx];
+		monitor_addr++; /* cq_gen bit is in second 64bit location */
+
+		if (qm_port->gen_bit)
+			expected_value = qe_mask.raw_qe[1];
+		else
+			expected_value = 0;
+
+		rte_power_monitor(monitor_addr, expected_value,
+				  qe_mask.raw_qe[1], timeout + start_ticks,
+				  sizeof(uint64_t));
+
+		DLB2_INC_STAT(ev_port->stats.traffic.rx_umonitor_umwait, 1);
+	} else {
+		uint64_t poll_interval = RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL;
+		uint64_t curr_ticks = rte_get_timer_cycles();
+		uint64_t init_ticks = curr_ticks;
+
+		while ((curr_ticks - start_ticks < timeout) &&
+		       (curr_ticks - init_ticks < poll_interval))
+			curr_ticks = rte_get_timer_cycles();
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
+			 struct dlb2_port *qm_port,
+			 struct rte_event *events,
+			 struct dlb2_dequeue_qe *qes,
+			 int cnt)
+{
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	int i, num, evq_id;
+
+	for (i = 0, num = 0; i < cnt; i++) {
+		struct dlb2_dequeue_qe *qe = &qes[i];
+		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
+			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+			[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+		};
+
+		/* Fill in event information.
+		 * Note that flow_id must be embedded in the data by
+		 * the app, such as the mbuf RSS hash field if the data
+		 * buffer is a mbuf.
+		 */
+		if (unlikely(qe->error)) {
+			DLB2_LOG_ERR("QE error bit ON\n");
+			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
+			dlb2_consume_qe_immediate(qm_port, 1);
+			continue; /* Ignore */
+		}
+
+		events[num].u64 = qe->data;
+		events[num].flow_id = qe->flow_id;
+		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
+		events[num].event_type = qe->u.event_type.major;
+		events[num].sub_event_type = qe->u.event_type.sub;
+		events[num].sched_type = sched_type_map[qe->sched_type];
+		events[num].impl_opaque = qe->qid_depth;
+
+		/* qid not preserved for directed queues */
+		if (qm_port->is_directed)
+			evq_id = ev_port->link[0].queue_id;
+		else
+			evq_id = qid_mappings[qe->qid];
+
+		events[num].queue_id = evq_id;
+		DLB2_INC_STAT(
+			ev_port->stats.queue[evq_id].qid_depth[qe->qid_depth],
+			1);
+		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type], 1);
+		num++;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num);
+
+	return num;
+}
+
+static inline int
+dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
+			      struct dlb2_port *qm_port,
+			      struct rte_event *events,
+			      struct dlb2_dequeue_qe *qes)
+{
+	int sched_type_map[] = {
+		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+	};
+	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	__m128i sse_evt[2];
+
+	/* In the unlikely case that any of the QE error bits are set, process
+	 * them one at a time.
+	 */
+	if (unlikely(qes[0].error || qes[1].error ||
+		     qes[2].error || qes[3].error))
+		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
+						 qes, num_events);
+
+	events[0].u64 = qes[0].data;
+	events[1].u64 = qes[1].data;
+	events[2].u64 = qes[2].data;
+	events[3].u64 = qes[3].data;
+
+	/* Construct the metadata portion of two struct rte_events
+	 * in one 128b SSE register. Event metadata is constructed in the SSE
+	 * registers like so:
+	 * sse_evt[0][63:0]:   event[0]'s metadata
+	 * sse_evt[0][127:64]: event[1]'s metadata
+	 * sse_evt[1][63:0]:   event[2]'s metadata
+	 * sse_evt[1][127:64]: event[3]'s metadata
+	 */
+	sse_evt[0] = _mm_setzero_si128();
+	sse_evt[1] = _mm_setzero_si128();
+
+	/* Convert the hardware queue ID to an event queue ID and store it in
+	 * the metadata:
+	 * sse_evt[0][47:40]   = qid_mappings[qes[0].qid]
+	 * sse_evt[0][111:104] = qid_mappings[qes[1].qid]
+	 * sse_evt[1][47:40]   = qid_mappings[qes[2].qid]
+	 * sse_evt[1][111:104] = qid_mappings[qes[3].qid]
+	 */
+#define DLB_EVENT_QUEUE_ID_BYTE 5
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[0].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[1].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[2].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[3].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+
+	/* Convert the hardware priority to an event priority and store it in
+	 * the metadata, while also returning the queue depth status
+	 * value captured by the hardware, storing it in impl_opaque, which can
+	 * be read by the application but not modified
+	 * sse_evt[0][55:48]   = DLB2_TO_EV_PRIO(qes[0].priority)
+	 * sse_evt[0][63:56]   = qes[0].qid_depth
+	 * sse_evt[0][119:112] = DLB2_TO_EV_PRIO(qes[1].priority)
+	 * sse_evt[0][127:120] = qes[1].qid_depth
+	 * sse_evt[1][55:48]   = DLB2_TO_EV_PRIO(qes[2].priority)
+	 * sse_evt[1][63:56]   = qes[2].qid_depth
+	 * sse_evt[1][119:112] = DLB2_TO_EV_PRIO(qes[3].priority)
+	 * sse_evt[1][127:120] = qes[3].qid_depth
+	 */
+#define DLB_EVENT_PRIO_IMPL_OPAQUE_WORD 3
+#define DLB_BYTE_SHIFT 8
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[0].priority) |
+			(qes[0].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[1].priority) |
+			(qes[1].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[2].priority) |
+			(qes[2].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[3].priority) |
+			(qes[3].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+
+	/* Write the event type, sub event type, and flow_id to the event
+	 * metadata.
+	 * sse_evt[0][31:0]   = qes[0].flow_id |
+	 *			qes[0].u.event_type.major << 28 |
+	 *			qes[0].u.event_type.sub << 20;
+	 * sse_evt[0][95:64]  = qes[1].flow_id |
+	 *			qes[1].u.event_type.major << 28 |
+	 *			qes[1].u.event_type.sub << 20;
+	 * sse_evt[1][31:0]   = qes[2].flow_id |
+	 *			qes[2].u.event_type.major << 28 |
+	 *			qes[2].u.event_type.sub << 20;
+	 * sse_evt[1][95:64]  = qes[3].flow_id |
+	 *			qes[3].u.event_type.major << 28 |
+	 *			qes[3].u.event_type.sub << 20;
+	 */
+#define DLB_EVENT_EV_TYPE_DW 0
+#define DLB_EVENT_EV_TYPE_SHIFT 28
+#define DLB_EVENT_SUB_EV_TYPE_SHIFT 20
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[0].flow_id |
+			qes[0].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[0].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[1].flow_id |
+			qes[1].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[1].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[2].flow_id |
+			qes[2].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[2].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[3].flow_id |
+			qes[3].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT  |
+			qes[3].u.event_type.sub << DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+
+	/* Write the sched type to the event metadata. 'op' and 'rsvd' are not
+	 * set:
+	 * sse_evt[0][39:32]  = sched_type_map[qes[0].sched_type] << 6
+	 * sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6
+	 * sse_evt[1][39:32]  = sched_type_map[qes[2].sched_type] << 6
+	 * sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6
+	 */
+#define DLB_EVENT_SCHED_TYPE_BYTE 4
+#define DLB_EVENT_SCHED_TYPE_SHIFT 6
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[0].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[1].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[2].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[3].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+
+	/* Store the metadata to the event (use the double-precision
+	 * _mm_storeh_pd because there is no integer function for storing the
+	 * upper 64b):
+	 * events[0].event = sse_evt[0][63:0]
+	 * events[1].event = sse_evt[0][127:64]
+	 * events[2].event = sse_evt[1][63:0]
+	 * events[3].event = sse_evt[1][127:64]
+	 */
+	_mm_storel_epi64((__m128i *)&events[0].event, sse_evt[0]);
+	_mm_storeh_pd((double *)&events[1].event, (__m128d) sse_evt[0]);
+	_mm_storel_epi64((__m128i *)&events[2].event, sse_evt[1]);
+	_mm_storeh_pd((double *)&events[3].event, (__m128d) sse_evt[1]);
+
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[0].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[1].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[2].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[3].sched_type], 1);
+
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[0].queue_id].
+			qid_depth[qes[0].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[1].queue_id].
+			qid_depth[qes[1].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[2].queue_id].
+			qid_depth[qes[2].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[3].queue_id].
+			qid_depth[qes[3].qid_depth],
+		1);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num_events);
+
+	return num_events;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe_sparse(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe)
+{
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	uint8_t xor_mask[2] = {0x0F, 0x00};
+	const uint8_t and_mask = 0x0F;
+	__m128i *qes = (__m128i *)qe;
+	uint8_t gen_bits, gen_bit;
+	uintptr_t addr[4];
+	uint16_t idx;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+
+	idx = qm_port->cq_idx;
+
+	/* Load the next 4 QEs */
+	addr[0] = (uintptr_t)&cq_addr[idx];
+	addr[1] = (uintptr_t)&cq_addr[(idx +  4) & qm_port->cq_depth_mask];
+	addr[2] = (uintptr_t)&cq_addr[(idx +  8) & qm_port->cq_depth_mask];
+	addr[3] = (uintptr_t)&cq_addr[(idx + 12) & qm_port->cq_depth_mask];
+
+	/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */
+	rte_prefetch0(&cq_addr[(idx + 16) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 20) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 24) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 28) & qm_port->cq_depth_mask]);
+
+	/* Correct the xor_mask for wrap-around QEs */
+	gen_bit = qm_port->gen_bit;
+	xor_mask[gen_bit] ^= !!((idx +  4) > qm_port->cq_depth_mask) << 1;
+	xor_mask[gen_bit] ^= !!((idx +  8) > qm_port->cq_depth_mask) << 2;
+	xor_mask[gen_bit] ^= !!((idx + 12) > qm_port->cq_depth_mask) << 3;
+
+	/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is
+	 * valid, then QEs[0:N-1] are too.
+	 */
+	qes[3] = _mm_load_si128((__m128i *)(void *)addr[3]);
+	rte_compiler_barrier();
+	qes[2] = _mm_load_si128((__m128i *)(void *)addr[2]);
+	rte_compiler_barrier();
+	qes[1] = _mm_load_si128((__m128i *)(void *)addr[1]);
+	rte_compiler_barrier();
+	qes[0] = _mm_load_si128((__m128i *)(void *)addr[0]);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[gen_bit];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask;
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline void
+dlb2_inc_cq_idx(struct dlb2_port *qm_port, int cnt)
+{
+	uint16_t idx = qm_port->cq_idx_unmasked + cnt;
+
+	qm_port->cq_idx_unmasked = idx;
+	qm_port->cq_idx = idx & qm_port->cq_depth_mask;
+	qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) & 0x1;
+}
+
+static int
+dlb2_event_release(struct dlb2_eventdev *dlb2,
+		   uint8_t port_id,
+		   int n)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_eventdev_port *ev_port;
+	struct dlb2_port *qm_port;
+	int i, cnt;
+
+	if (port_id > dlb2->num_ports) {
+		DLB2_LOG_ERR("Invalid port id %d in dlb2-event_release\n",
+			     port_id);
+		rte_errno = -EINVAL;
+		return rte_errno;
+	}
+
+	ev_port = &dlb2->ev_ports[port_id];
+	qm_port = &ev_port->qm_port;
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	cnt = 0;
+
+	if (qm_port->is_directed) {
+		cnt = n;
+		goto sw_credit_update;
+	}
+
+	for (i = 0; i < n; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		int j;
+
+		/* Zero-out QEs */
+		qm_port->qe4[0].cmd_byte = 0;
+		qm_port->qe4[1].cmd_byte = 0;
+		qm_port->qe4[2].cmd_byte = 0;
+		qm_port->qe4[3].cmd_byte = 0;
+
+		for (j = 0; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++)
+			qm_port->qe4[j].cmd_byte = DLB2_COMP_CMD_BYTE;
+
+		qm_port->issued_releases += j;
+
+		if (j == 0)
+			break;
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+	}
+
+sw_credit_update:
+	/* each release returns one credit */
+	if (!ev_port->outstanding_releases) {
+		DLB2_LOG_ERR("Unrecoverable application error. Outstanding releases underflowed.\n");
+		rte_errno = -ENOTRECOVERABLE;
+		return rte_errno;
+	}
+
+	ev_port->outstanding_releases -= cnt;
+	ev_port->inflight_credits += cnt;
+
+	/* Replenish s/w credits if enough releases are performed */
+	dlb2_replenish_sw_credits(dlb2, ev_port);
+	return 0;
+}
+
+static inline int16_t
+dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
+		       struct dlb2_eventdev_port *ev_port,
+		       struct rte_event *events,
+		       uint16_t max_num,
+		       uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe_sparse(qm_port, qes);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail << 2);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							      qm_port,
+							      &events[num],
+							      &qes[0]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							 qm_port,
+							 &events[num],
+							 &qes[0],
+							 num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe,
+	     uint8_t *offset)
+{
+	uint8_t xor_mask[2][4] = { {0x0F, 0x0E, 0x0C, 0x08},
+				   {0x00, 0x01, 0x03, 0x07} };
+	uint8_t and_mask[4] = {0x0F, 0x0E, 0x0C, 0x08};
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	__m128i *qes = (__m128i *)qe;
+	uint64_t *cache_line_base;
+	uint8_t gen_bits;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	cq_addr = &cq_addr[qm_port->cq_idx];
+
+	cache_line_base = (void *)(((uintptr_t)cq_addr) & ~0x3F);
+	*offset = ((uintptr_t)cq_addr & 0x30) >> 4;
+
+	/* Load the next CQ cache line from memory. Pack these reads as tight
+	 * as possible to reduce the chance that DLB invalidates the line while
+	 * the CPU is reading it. Read the cache line backwards to ensure that
+	 * if QE[N] (N > 0) is valid, then QEs[0:N-1] are too.
+	 *
+	 * (Valid QEs start at &qe[offset])
+	 */
+	qes[3] = _mm_load_si128((__m128i *)&cache_line_base[6]);
+	qes[2] = _mm_load_si128((__m128i *)&cache_line_base[4]);
+	qes[1] = _mm_load_si128((__m128i *)&cache_line_base[2]);
+	qes[0] = _mm_load_si128((__m128i *)&cache_line_base[0]);
+
+	/* Evict the cache line ASAP */
+	dlb2_cldemote(cache_line_base);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[qm_port->gen_bit][*offset];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask[*offset];
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline int16_t
+dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
+		struct dlb2_eventdev_port *ev_port,
+		struct rte_event *events,
+		uint16_t max_num,
+		uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t offset;
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe(qm_port, qes, &offset);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							     qm_port,
+							     &events[num],
+							     &qes[offset]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							qm_port,
+							&events[num],
+							&qes[offset],
+							num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static uint16_t
+dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
+			 uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue(void *event_port, struct rte_event *ev, uint64_t wait)
+{
+	return dlb2_event_dequeue_burst(event_port, ev, 1, wait);
+}
+
+static uint16_t
+dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
+				uint16_t num, uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
+			  uint64_t wait)
+{
+	return dlb2_event_dequeue_burst_sparse(event_port, ev, 1, wait);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
+	struct dlb2_eventdev *dlb2;
+
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
@@ -2691,6 +3451,15 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	dev->enqueue_burst = dlb2_event_enqueue_burst;
 	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
 	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
+
+	dlb2 = dev->data->dev_private;
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE) {
+		dev->dequeue = dlb2_event_dequeue_sparse;
+		dev->dequeue_burst = dlb2_event_dequeue_burst_sparse;
+	} else {
+		dev->dequeue = dlb2_event_dequeue;
+		dev->dequeue_burst = dlb2_event_dequeue_burst;
+	}
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v6 19/23] event/dlb2: add eventdev stop and close

[Timothy McDaniel]

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 257 +++++++++++++++++++++++++++--
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c |  91 ++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  52 ++++++
 5 files changed, 396 insertions(+), 16 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 3d82ed5..1944082 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -71,21 +71,6 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
-/*
- * DUMMY - added so that xstats path will compile/link.
- * Will be replaced by real version in a subsequent
- * patch.
- */
-uint32_t
-dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
-		     struct dlb2_eventdev_queue *queue)
-{
-	RTE_SET_USED(dlb2);
-	RTE_SET_USED(queue);
-
-	return 0;
-}
-
 static void
 dlb2_free_qe_mem(struct dlb2_port *qm_port)
 {
@@ -1878,7 +1863,6 @@ dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
 		return 0; /* Ignore and return success */
 	}
 
-	/* FIXME: How to handle unlink on a directed port? */
 	if (ev_port->qm_port.is_directed) {
 		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
 			     ev_port->id);
@@ -3420,6 +3404,245 @@ dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
 }
 
 static void
+dlb2_flush_port(struct rte_eventdev *dev, int port_id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	eventdev_stop_flush_t flush;
+	struct rte_event ev;
+	uint8_t dev_id;
+	void *arg;
+	int i;
+
+	flush = dev->dev_ops->dev_stop_flush;
+	dev_id = dev->data->dev_id;
+	arg = dev->data->dev_stop_flush_arg;
+
+	while (rte_event_dequeue_burst(dev_id, port_id, &ev, 1, 0)) {
+		if (flush)
+			flush(dev_id, ev, arg);
+
+		if (dlb2->ev_ports[port_id].qm_port.is_directed)
+			continue;
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+	}
+
+	/* Enqueue any additional outstanding releases */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = dlb2->ev_ports[port_id].outstanding_releases; i > 0; i--)
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+}
+
+static uint32_t
+dlb2_get_ldb_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_ldb_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_ldb_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_ldb_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static uint32_t
+dlb2_get_dir_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_dir_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_dir_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_dir_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	if (queue->qm_queue.is_directed)
+		return dlb2_get_dir_queue_depth(dlb2, queue);
+	else
+		return dlb2_get_ldb_queue_depth(dlb2, queue);
+}
+
+static bool
+dlb2_queue_is_empty(struct dlb2_eventdev *dlb2,
+		    struct dlb2_eventdev_queue *queue)
+{
+	return dlb2_get_queue_depth(dlb2, queue) == 0;
+}
+
+static bool
+dlb2_linked_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0)
+			continue;
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static bool
+dlb2_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static void
+dlb2_drain(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_port *ev_port = NULL;
+	uint8_t dev_id;
+	int i;
+
+	dev_id = dev->data->dev_id;
+
+	while (!dlb2_linked_queues_empty(dlb2)) {
+		/* Flush all the ev_ports, which will drain all their connected
+		 * queues.
+		 */
+		for (i = 0; i < dlb2->num_ports; i++)
+			dlb2_flush_port(dev, i);
+	}
+
+	/* The queues are empty, but there may be events left in the ports. */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_flush_port(dev, i);
+
+	/* If the domain's queues are empty, we're done. */
+	if (dlb2_queues_empty(dlb2))
+		return;
+
+	/* Else, there must be at least one unlinked load-balanced queue.
+	 * Select a load-balanced port with which to drain the unlinked
+	 * queue(s).
+	 */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		if (!ev_port->qm_port.is_directed)
+			break;
+	}
+
+	if (i == dlb2->num_ports) {
+		DLB2_LOG_ERR("internal error: no LDB ev_ports\n");
+		return;
+	}
+
+	rte_errno = 0;
+	rte_event_port_unlink(dev_id, ev_port->id, NULL, 0);
+
+	if (rte_errno) {
+		DLB2_LOG_ERR("internal error: failed to unlink ev_port %d\n",
+			     ev_port->id);
+		return;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		uint8_t qid, prio;
+		int ret;
+
+		if (dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			continue;
+
+		qid = i;
+		prio = 0;
+
+		/* Link the ev_port to the queue */
+		ret = rte_event_port_link(dev_id, ev_port->id, &qid, &prio, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to link ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+
+		/* Flush the queue */
+		while (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			dlb2_flush_port(dev, ev_port->id);
+
+		/* Drain any extant events in the ev_port. */
+		dlb2_flush_port(dev, ev_port->id);
+
+		/* Unlink the ev_port from the queue */
+		ret = rte_event_port_unlink(dev_id, ev_port->id, &qid, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to unlink ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+	}
+}
+
+static void
+dlb2_eventdev_stop(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_DBG("Internal error: already stopped\n");
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	} else if (dlb2->run_state != DLB2_RUN_STATE_STARTED) {
+		DLB2_LOG_ERR("Internal error: bad state %d for dev_stop\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPING;
+
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	dlb2_drain(dev);
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPED;
+}
+
+static int
+dlb2_eventdev_close(struct rte_eventdev *dev)
+{
+	dlb2_hw_reset_sched_domain(dev, false);
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3429,6 +3652,8 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.dev_start        = dlb2_eventdev_start,
+		.dev_stop         = dlb2_eventdev_stop,
+		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a86191d..5471dd8 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -66,3 +66,9 @@ int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				     struct dlb2_start_domain_args *cfg);
+
+int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 7ead374..b508eb0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -65,4 +65,10 @@ extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				struct dlb2_start_domain_args *cfg);
+
+extern int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+extern int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index c14a2f3..ae5ef2f 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5934,3 +5934,94 @@ dlb2_hw_start_domain(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_dir_queue_depth(hw, queue);
+
+	return 0;
+}
+
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+
+	dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index fff2e57..632c4e0 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -527,6 +527,56 @@ dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_ldb_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_ldb_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_dir_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret = 0;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_dir_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -544,6 +594,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth;
+	dlb2_iface_get_dir_queue_depth = dlb2_pf_get_dir_queue_depth;
 	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v6 20/23] event/dlb2: add PMD's token pop public interface

[Timothy McDaniel]

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/api/doxy-api-index.md         |  1 +
 drivers/event/dlb2/dlb2.c         | 53 ++++++++++++++++++++++++----
 drivers/event/dlb2/dlb2_priv.h    |  3 ++
 drivers/event/dlb2/meson.build    |  5 ++-
 drivers/event/dlb2/rte_pmd_dlb2.c | 39 +++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2.h | 72 +++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/version.map    |  6 ++++
 7 files changed, 172 insertions(+), 7 deletions(-)
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index a9c12d1..00f13e2 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -52,6 +52,7 @@ The public API headers are grouped by topics:
   [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
   [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
+  [dlb2]	       (@ref rte_pmd_dlb2.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 1944082..5818d81 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1197,7 +1197,7 @@ dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
 	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
 
 	qm_port->dequeue_depth = dequeue_depth;
-
+	qm_port->token_pop_thresh = dequeue_depth;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -1365,6 +1365,8 @@ dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
 
 	qm_port->dequeue_depth = dequeue_depth;
 
+	/* Directed ports are auto-pop, by default. */
+	qm_port->token_pop_mode = AUTO_POP;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -2612,6 +2614,14 @@ dlb2_event_enqueue_burst(void *event_port,
 		dlb2_event_build_hcws(qm_port, &events[i], j,
 				      sched_types, queue_ids);
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
@@ -2620,6 +2630,11 @@ dlb2_event_enqueue_burst(void *event_port,
 			break;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
 	return cnt;
 }
 
@@ -3109,11 +3124,25 @@ dlb2_event_release(struct dlb2_eventdev *dlb2,
 		if (j == 0)
 			break;
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
 sw_credit_update:
 	/* each release returns one credit */
 	if (!ev_port->outstanding_releases) {
@@ -3197,8 +3226,8 @@ dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3324,8 +3353,8 @@ dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3340,6 +3369,7 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 			 uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3355,6 +3385,9 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3374,6 +3407,7 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 				uint16_t num, uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3389,6 +3423,9 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3693,7 +3730,7 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    struct dlb2_devargs *dlb2_args)
 {
 	struct dlb2_eventdev *dlb2;
-	int err;
+	int err, i;
 
 	dlb2 = dev->data->dev_private;
 
@@ -3743,6 +3780,10 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Initialize each port's token pop mode */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
+		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 61567a6..b73cf3f 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -12,6 +12,7 @@
 #include <rte_config.h>
 #include "dlb2_user.h"
 #include "dlb2_log.h"
+#include "rte_pmd_dlb2.h"
 
 #ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
 #define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
@@ -290,6 +291,7 @@ struct dlb2_port {
 	bool gen_bit;
 	uint16_t dir_credits;
 	uint32_t dequeue_depth;
+	enum dlb2_token_pop_mode token_pop_mode;
 	union dlb2_port_config cfg;
 	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
 	uint16_t cached_ldb_credits;
@@ -298,6 +300,7 @@ struct dlb2_port {
 	bool int_armed;
 	uint16_t owed_tokens;
 	int16_t issued_releases;
+	int16_t token_pop_thresh;
 	int cq_depth;
 	uint16_t cq_idx;
 	uint16_t cq_idx_unmasked;
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index dc16c96..b734ff6 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,3 +1,4 @@
+
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
@@ -12,7 +13,9 @@ sources = files('dlb2.c',
 		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
-		'pf/base/dlb2_resource.c'
+		'pf/base/dlb2_resource.c',
+		'rte_pmd_dlb2.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
+install_headers('rte_pmd_dlb2.h')
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
new file mode 100644
index 0000000..b09b585
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "rte_pmd_dlb2.h"
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode)
+{
+	struct dlb2_eventdev *dlb2;
+	struct rte_eventdev *dev;
+
+	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+	dev = &rte_eventdevs[dev_id];
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (mode >= NUM_TOKEN_POP_MODES)
+		return -EINVAL;
+
+	/* The event device must be configured, but not yet started */
+	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
+		return -EINVAL;
+
+	/* The token pop mode must be set before configuring the port */
+	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
+		return -EINVAL;
+
+	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
new file mode 100644
index 0000000..74399db
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+/*!
+ *  @file      rte_pmd_dlb2.h
+ *
+ *  @brief     DLB PMD-specific functions
+ */
+
+#ifndef _RTE_PMD_DLB2_H_
+#define _RTE_PMD_DLB2_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Selects the token pop mode for a DLB2 port.
+ */
+enum dlb2_token_pop_mode {
+	/* Pop the CQ tokens immediately after dequeueing. */
+	AUTO_POP,
+	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
+	 * Supported on load-balanced ports only.
+	 */
+	DELAYED_POP,
+	/* Pop the CQ tokens during next dequeue operation. */
+	DEFERRED_POP,
+
+	/* NUM_TOKEN_POP_MODES must be last */
+	NUM_TOKEN_POP_MODES
+};
+
+/*!
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Configure the token pop mode for a DLB2 port. By default, all ports use
+ * AUTO_POP. This function must be called before calling rte_event_port_setup()
+ * for the port, but after calling rte_event_dev_configure().
+ *
+ * @param dev_id
+ *    The identifier of the event device.
+ * @param port_id
+ *    The identifier of the event port.
+ * @param mode
+ *    The token pop mode.
+ *
+ * @return
+ * - 0: Success
+ * - EINVAL: Invalid dev_id, port_id, or mode
+ * - EINVAL: The DLB2 is not configured, is already running, or the port is
+ *   already setup
+ */
+
+__rte_experimental
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_DLB2_H_ */
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
index 4a76d1d..b1e4dff 100644
--- a/drivers/event/dlb2/version.map
+++ b/drivers/event/dlb2/version.map
@@ -1,3 +1,9 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmd_dlb2_set_token_pop_mode;
+};
-- 
2.6.4

[dpdk-dev] [PATCH v6 21/23] event/dlb2: add PMD self-tests

[Timothy McDaniel]

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 app/test/test_eventdev.c           |    7 +
 drivers/event/dlb2/dlb2.c          |    1 +
 drivers/event/dlb2/dlb2_selftest.c | 1524 ++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build     |    3 +-
 4 files changed, 1534 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c

diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
index 62019c1..bcfaa53 100644
--- a/app/test/test_eventdev.c
+++ b/app/test/test_eventdev.c
@@ -1030,6 +1030,12 @@ test_eventdev_selftest_dpaa2(void)
 	return test_eventdev_selftest_impl("event_dpaa2", "");
 }
 
+static int
+test_eventdev_selftest_dlb2(void)
+{
+	return test_eventdev_selftest_impl("dlb2_event", "");
+}
+
 REGISTER_TEST_COMMAND(eventdev_common_autotest, test_eventdev_common);
 REGISTER_TEST_COMMAND(eventdev_selftest_sw, test_eventdev_selftest_sw);
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
@@ -1037,3 +1043,4 @@ REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
 		test_eventdev_selftest_octeontx2);
 REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2, test_eventdev_selftest_dpaa2);
+REGISTER_TEST_COMMAND(eventdev_selftest_dlb2, test_eventdev_selftest_dlb2);
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 5818d81..e4bd8cf 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3704,6 +3704,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
 		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+		.dev_selftest     = test_dlb2_eventdev,
 	};
 
 	/* Expose PMD's eventdev interface */
diff --git a/drivers/event/dlb2/dlb2_selftest.c b/drivers/event/dlb2/dlb2_selftest.c
new file mode 100644
index 0000000..e300067
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_selftest.c
@@ -0,0 +1,1524 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <unistd.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_eventdev.h>
+#include <rte_pause.h>
+
+#include "dlb2_priv.h"
+#include "rte_pmd_dlb2.h"
+
+#define MAX_PORTS 32
+#define MAX_QIDS 32
+#define DEFAULT_NUM_SEQ_NUMS 64
+
+static struct rte_mempool *eventdev_func_mempool;
+static int evdev;
+
+struct test {
+	struct rte_mempool *mbuf_pool;
+	int nb_qids;
+};
+
+/* initialization and config */
+static inline int
+init(struct test *t, int nb_queues, int nb_ports)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	memset(t, 0, sizeof(*t));
+
+	t->mbuf_pool = eventdev_func_mempool;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = nb_queues;
+	config.nb_event_ports = nb_ports;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0)
+		printf("%d: Error configuring device\n", __LINE__);
+
+	return ret;
+}
+
+static inline int
+create_ports(int num_ports)
+{
+	int i;
+
+	if (num_ports > MAX_PORTS)
+		return -1;
+
+	for (i = 0; i < num_ports; i++) {
+		struct rte_event_port_conf conf;
+
+		if (rte_event_port_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default port conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		if (rte_event_port_setup(evdev, i, &conf) < 0) {
+			printf("%d: Error setting up port %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+create_lb_qids(struct test *t, int num_qids, uint32_t flags)
+{
+	int i;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		struct rte_event_queue_conf conf;
+
+		if (rte_event_queue_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default queue conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		conf.schedule_type = flags;
+
+		if (conf.schedule_type == RTE_SCHED_TYPE_PARALLEL)
+			conf.nb_atomic_order_sequences = 0;
+		else
+			conf.nb_atomic_order_sequences = DEFAULT_NUM_SEQ_NUMS;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+static inline int
+create_atomic_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ATOMIC);
+}
+
+/* destruction */
+static inline void
+cleanup(void)
+{
+	int ret = 0;
+
+	rte_event_dev_stop(evdev);
+	ret = rte_event_dev_close(evdev);
+
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+};
+
+static inline int
+enqueue_timeout(uint8_t port_id, struct rte_event *ev, uint64_t tmo_us)
+{
+	const uint64_t start = rte_get_timer_cycles();
+	const uint64_t ticks = (tmo_us * rte_get_timer_hz()) / 1E6;
+
+	while ((rte_get_timer_cycles() - start) < ticks) {
+		if (rte_event_enqueue_burst(evdev, port_id, ev, 1) == 1)
+			return 0;
+
+		if (rte_errno != -ENOSPC) {
+			printf("enqueue_burst returned rte_errno %d\n",
+			       rte_errno);
+			return -1;
+		}
+	}
+	printf("%s time out\n", __func__);
+	return -1;
+}
+
+static void
+flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused)
+{
+	rte_pktmbuf_free(event.mbuf);
+}
+
+static int
+test_stop_flush(struct test *t) /* test to check we can properly flush events */
+{
+	struct rte_event ev;
+	uint32_t dequeue_depth;
+	unsigned int i, count;
+	uint8_t queue_id;
+
+	ev.op = RTE_EVENT_OP_NEW;
+
+	if (init(t, 2, 1) < 0 ||
+	    create_ports(1) < 0 ||
+	    create_atomic_qids(t, 2) < 0) {
+		printf("%d: Error initializing device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_port_link(evdev, 0, NULL, NULL, 0) != 2) {
+		printf("%d: Error linking queues to the port\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: Error with start call\n", __LINE__);
+		goto err;
+	}
+
+	/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked
+	 * queue.
+	 */
+	queue_id = 1;
+
+	if (rte_event_port_unlink(evdev, 0, &queue_id, 1) != 1) {
+		printf("%d: Error unlinking queue 1 from port\n", __LINE__);
+		goto err;
+	}
+
+	count = rte_mempool_avail_count(t->mbuf_pool);
+
+	if (rte_event_port_attr_get(evdev,
+				    0,
+				    RTE_EVENT_PORT_ATTR_DEQ_DEPTH,
+				    &dequeue_depth)) {
+		printf("%d: Error retrieveing dequeue depth\n", __LINE__);
+		goto err;
+	}
+
+	/* Send QEs to queue 0 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 0;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Send QEs to queue 1 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 1;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Now the DLB is scheduling events from the port to the IQ, and at
+	 * least one event should be remaining in each queue.
+	 */
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, flush, NULL)) {
+		printf("%d: Error installing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+
+	if (count != rte_mempool_avail_count(t->mbuf_pool)) {
+		printf("%d: Error executing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, NULL, NULL)) {
+		printf("%d: Error uninstalling the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	return 0;
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_single_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Create a directed port */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) == 0) {
+		printf("%d: port 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Create a load-balanced port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a directed queue */
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed queue */
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) == 0) {
+		printf("%d: queue 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a load-balanced queue */
+	queue_conf.event_queue_cfg = 0;
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to link directed and load-balanced resources */
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 0 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 1 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Link ports to queues */
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+#define NUM_LDB_PORTS 64
+#define NUM_LDB_QUEUES 32
+
+static int
+test_info_get(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	config.nb_event_ports = info.max_event_ports;
+	config.nb_event_queues = NUM_LDB_QUEUES + info.max_event_ports / 2;
+	config.nb_single_link_event_port_queues = info.max_event_ports / 2;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		goto err;
+	}
+
+	/* The DLB2 PMD only reports load-balanced ports and queues in its
+	 * info_get function. Confirm that these values don't include the
+	 * directed port or queue counts.
+	 */
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret) {
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+		return -1;
+	}
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+static int
+test_reconfiguration_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 2 LDB queues */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+			printf("%d: P0->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P1->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+			printf("%d: P1->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Configure device with 2 DIR ports and 2 DIR queues */
+	config.nb_single_link_event_port_queues = 2;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q1 */
+	queue_id = 1;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: P0->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Link P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: P1->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_queues = 5;
+	config.nb_event_ports = 5;
+	config.nb_single_link_event_port_queues = 1;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < config.nb_event_queues - 1; i++) {
+		port_conf.event_port_cfg = 0;
+		queue_conf.event_queue_cfg = 0;
+
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: P%d->Q%d link expected to succeed\n",
+			       __LINE__, i, i);
+			goto err;
+		}
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+		printf("%d: port %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+		printf("%d: queue %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	queue_id = i;
+
+	if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+		printf("%d: P%d->Q%d link expected to succeed\n",
+		       __LINE__, i, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_ports += 1;
+
+	/* Reconfigure device with 1 more load-balanced port */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Configure the new port */
+	if (rte_event_port_setup(evdev, config.nb_event_ports - 1,
+				 &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_load_balanced_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_directed_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 DIR port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (ev.queue_id != 0) {
+		printf("%d: invalid dequeued event queue ID (%d)\n",
+		       __LINE__, ev.queue_id);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_deferred_sched(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	const int num_events = 128;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 1 queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 1, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.dequeue_depth = 1;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.schedule_type = RTE_SCHED_TYPE_PARALLEL;
+	queue_conf.nb_atomic_order_sequences = 0;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 128 NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_PARALLEL;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue one event from port 0 */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue (and release) all other events from port 1. Deferred
+	 * scheduling ensures no other events are scheduled to port 0 without a
+	 * subsequent rte_event_dequeue_burst() call.
+	 */
+	for (i = 0; i < num_events - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 1, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		if (rte_event_enqueue_burst(evdev, 1, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_delayed_pop(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	int ret, i, num_events;
+	struct rte_event ev;
+	uint8_t queue_id;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DELAYED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	num_events = 2 * port_conf.dequeue_depth;
+
+	/* Enqueue 2 * dequeue_depth NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue dequeue_depth events but only release dequeue_depth - 2.
+	 * Delayed pop won't perform the pop and no more events will be
+	 * scheduled.
+	 */
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = 0; i < port_conf.dequeue_depth - 2; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	ret = rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout);
+	if (ret != 0) {
+		printf("%d: event dequeue expected to fail (ret = %d)\n",
+		       __LINE__, ret);
+		goto err;
+	}
+
+	/* Release one more event. This will trigger the token pop, and
+	 * dequeue_depth - 1 more events will be scheduled to the device.
+	 */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: RELEASE enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 0) {
+		printf("%d: event dequeue expected to fail\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+do_selftest(void)
+{
+	struct test t;
+	int ret;
+
+	/* Only create mbuf pool once, reuse for each test run */
+	if (!eventdev_func_mempool) {
+		eventdev_func_mempool =
+			rte_pktmbuf_pool_create("EVENTDEV_DLB2_ST_POOL",
+						(1 << 12), /* 4k buffers */
+						32 /*MBUF_CACHE_SIZE*/,
+						0,
+						512, /* use very small mbufs */
+						rte_socket_id());
+		if (!eventdev_func_mempool) {
+			printf("ERROR creating mempool\n");
+			goto test_fail;
+		}
+	}
+	t.mbuf_pool = eventdev_func_mempool;
+
+	printf("*** Running Stop Flush test...\n");
+	ret = test_stop_flush(&t);
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Single Link test...\n");
+	ret = test_single_link();
+	if (ret != 0) {
+		printf("ERROR - Single Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Info Get test...\n");
+	ret = test_info_get();
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Reconfiguration Link test...\n");
+	ret = test_reconfiguration_link();
+	if (ret != 0) {
+		printf("ERROR - Reconfiguration Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Load-Balanced Traffic test...\n");
+	ret = test_load_balanced_traffic();
+	if (ret != 0) {
+		printf("ERROR - Load-Balanced Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Directed Traffic test...\n");
+	ret = test_directed_traffic();
+	if (ret != 0) {
+		printf("ERROR - Directed Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Deferred Scheduling test...\n");
+	ret = test_deferred_sched();
+	if (ret != 0) {
+		printf("ERROR - Deferred Scheduling test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Delayed Pop test...\n");
+	ret = test_delayed_pop();
+	if (ret != 0) {
+		printf("ERROR - Delayed Pop test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	return 0;
+
+test_fail:
+	return -1;
+}
+
+int
+test_dlb2_eventdev(void)
+{
+	const char *dlb2_eventdev_name = "dlb2_event";
+	uint8_t num_evdevs = rte_event_dev_count();
+	int i, ret = 0;
+	int found = 0, skipped = 0, passed = 0, failed = 0;
+	struct rte_event_dev_info info;
+
+	for (i = 0; found + skipped < num_evdevs && i < RTE_EVENT_MAX_DEVS;
+	     i++) {
+		ret = rte_event_dev_info_get(i, &info);
+		if (ret < 0)
+			continue;
+
+		/* skip non-dlb2 event devices */
+		if (strncmp(info.driver_name, dlb2_eventdev_name,
+			    sizeof(*info.driver_name)) != 0) {
+			skipped++;
+			continue;
+		}
+
+		evdev = rte_event_dev_get_dev_id(info.driver_name);
+		if (evdev < 0) {
+			printf("Could not get dev_id for eventdev with name %s, i=%d\n",
+			       info.driver_name, i);
+			skipped++;
+			continue;
+		}
+		found++;
+		printf("Running selftest on eventdev %s\n", info.driver_name);
+		ret = do_selftest();
+		if (ret == 0) {
+			passed++;
+			printf("Selftest passed for eventdev %s\n",
+			       info.driver_name);
+		} else {
+			failed++;
+			printf("Selftest failed for eventdev %s, err=%d\n",
+			       info.driver_name, ret);
+		}
+	}
+
+	printf("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n",
+	       found, skipped, passed, failed);
+	return ret;
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index b734ff6..340bc57 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -14,7 +14,8 @@ sources = files('dlb2.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c',
-		'rte_pmd_dlb2.c'
+		'rte_pmd_dlb2.c',
+		'dlb2_selftest.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v6 22/23] event/dlb2: add queue and port release

[Timothy McDaniel]

DLB does not support reconfiguring individual queues
or ports on the fly. The entire device must be reconfigured.
Previously allocated port QE and memzone memory
is freed in this patch.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index e4bd8cf..4b2509b 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3680,6 +3680,28 @@ dlb2_eventdev_close(struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(id);
+
+	/* This function intentionally left blank. */
+}
+
+static void
+dlb2_eventdev_port_release(void *port)
+{
+	struct dlb2_eventdev_port *ev_port = port;
+	struct dlb2_port *qm_port;
+
+	if (ev_port) {
+		qm_port = &ev_port->qm_port;
+		if (qm_port->config_state == DLB2_CONFIGURED)
+			dlb2_free_qe_mem(qm_port);
+	}
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3693,8 +3715,10 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
+		.queue_release    = dlb2_eventdev_queue_release,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_release     = dlb2_eventdev_port_release,
 		.port_link        = dlb2_eventdev_port_link,
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
-- 
2.6.4

[dpdk-dev] [PATCH v6 23/23] event/dlb2: add timeout ticks entry point

[Timothy McDaniel]

Adds the timeout ticks conversion function.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 4b2509b..7890b54 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3701,6 +3701,18 @@ dlb2_eventdev_port_release(void *port)
 	}
 }
 
+static int
+dlb2_eventdev_timeout_ticks(struct rte_eventdev *dev, uint64_t ns,
+			    uint64_t *timeout_ticks)
+{
+	RTE_SET_USED(dev);
+	uint64_t cycles_per_ns = rte_get_timer_hz() / 1E9;
+
+	*timeout_ticks = ns * cycles_per_ns;
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -3723,6 +3735,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
 				    dlb2_eventdev_port_unlinks_in_progress,
+		.timeout_ticks    = dlb2_eventdev_timeout_ticks,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v7 00/23] Add DLB2 PMD

[Timothy McDaniel]

The following patch series adds support for a new eventdev PMD. The DLB2
PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
hardware.
The DLB2 is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB2 hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB2 ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a
directed queue. Additionally, even though LDB ports may link multiple
queues,
the number of queues that may be linked is limited by hardware.

While reviewing the code, please be aware that this PMD has full
control over the DLB2 hardware. Intel will be extending the DLB2 PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.
Note that the DLB2 hardware is a successor of the DLB hardware, and
as such is structured similarly, both in terms of code layout and
implementation.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Changes since V6:
=====================
- removed unused function, fixing build error
- fixed typo in port_setup commit message
- this patch series is based on dpdk-next-eventdev

Changes since V5:
================
- convert to use rte_power_monitor patches
- replace __builtin_ia32_movntdq() with _mm_stream_si128()
- remove unused functions in dlb_selftest.c
- workaround for RHEL 7 gcc brace bug

Changes since V4:
================
- moved introduction of dlb in relnotes_20_11 to first patch in series
- fixed underlines in dlb.rst that were too short
- note that the code still uses its private byte-encoded versions of
  umonitor/umwait, rather than the new functions in the power
  patch that are built on top of those intrinsics. This is intentional.

Changes since V3:
================
- updated MAINTAINERS file to alphabetically insert DLB
- don't create RTE_ symbols in pmd
- converted to use version.map scheme
- converted to use .._master_lcore instead of .._main_lcore
- this patch set is based on dpdk-next-eventdev

Changes since V2:
================
- fixed meson conditional build. Moved test into driver’s meson.build
  file instead of event/meson.build
- documentation is populated as associated code is introduced
- add log_register in add dynamic logging patch
- rename RTE_xxx symbol(s) as DLB2_xxx
- replaced function ptr enqueue_four with direct call to movdir64b
- remove unused port_pages
- broke up probe patch into 3 smaller patches for easier review
- changed param order of movdir64b/movntdq to match intrinsics
- added self to MAINTAINERS files
- squashed announcement of availability into last patch in series
- correct spelling errors and delete repeated words
- DPDK_21.0 -> DPDK 21 in map file
- add experimental banner to public structs and APIs

Changes since V1:
=================
- implement changes requested in code reviews by Gage Eads and Mike
  Chen
- fix a memzone leak
- convert to use eal rte-cpuflags patch from Liang Ma

Known Issues:
- the documentation contained in dlb2.rst is not complete, and
  should be updated to include command line parameters (class of service,
  etc ...).

Depends-on: patch-82202 ("eventdev: increase MAX QUEUES PER DEV to 255")

Timothy McDaniel (23):
  event/dlb2: add documentation and meson build infrastructure
  event/dlb2: add dynamic logging
  event/dlb2: add private data structures and constants
  event/dlb2: add definitions shared with LKM or shared code
  event/dlb2: add inline functions
  event/dlb2: add eventdev probe
  event/dlb2: add flexible interface
  event/dlb2: add probe-time hardware init
  event/dlb2: add xstats
  event/dlb2: add infos get and configure
  event/dlb2: add queue and port default conf
  event/dlb2: add queue setup
  event/dlb2: add port setup
  event/dlb2: add port link
  event/dlb2: add port unlink and port unlinks in progress
  event/dlb2: add eventdev start
  event/dlb2: add enqueue and its burst variants
  event/dlb2: add dequeue and its burst variants
  event/dlb2: add eventdev stop and close
  event/dlb2: add PMD's token pop public interface
  event/dlb2: add PMD self-tests
  event/dlb2: add queue and port release
  event/dlb2: add timeout ticks entry point

 MAINTAINERS                                    |    6 +-
 app/test/test_eventdev.c                       |    7 +
 config/rte_config.h                            |    7 +
 doc/api/doxy-api-index.md                      |    1 +
 doc/guides/eventdevs/dlb2.rst                  |  365 ++
 doc/guides/eventdevs/index.rst                 |    1 +
 doc/guides/rel_notes/release_20_11.rst         |    5 +
 drivers/event/dlb2/dlb2.c                      | 3947 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c                |   74 +
 drivers/event/dlb2/dlb2_iface.h                |   74 +
 drivers/event/dlb2/dlb2_inline_fns.h           |   61 +
 drivers/event/dlb2/dlb2_log.h                  |   25 +
 drivers/event/dlb2/dlb2_priv.h                 |  578 +++
 drivers/event/dlb2/dlb2_selftest.c             | 1524 ++++++
 drivers/event/dlb2/dlb2_user.h                 |  679 +++
 drivers/event/dlb2/dlb2_xstats.c               | 1240 +++++
 drivers/event/dlb2/meson.build                 |   22 +
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 +++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 +
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c     | 6027 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  673 +++
 drivers/event/dlb2/pf/dlb2_main.h              |   97 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  728 +++
 drivers/event/dlb2/rte_pmd_dlb2.c              |   39 +
 drivers/event/dlb2/rte_pmd_dlb2.h              |   72 +
 drivers/event/dlb2/version.map                 |    9 +
 drivers/event/meson.build                      |    3 +-
 33 files changed, 22497 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
 create mode 100644 drivers/event/dlb2/dlb2_log.h
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c
 create mode 100644 drivers/event/dlb2/dlb2_user.h
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
 create mode 100644 drivers/event/dlb2/version.map

-- 
2.6.4

[dpdk-dev] [PATCH v7 01/23] event/dlb2: add documentation and meson build infrastructure

[Timothy McDaniel]

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux 64 bit X86 platforms at this time.

Adds announcement of availabililty for the new driver
for Intel Dynamic Load Balancer 2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 MAINTAINERS                            |  6 ++++-
 config/rte_config.h                    |  7 ++++++
 doc/guides/eventdevs/dlb2.rst          | 40 ++++++++++++++++++++++++++++++++++
 doc/guides/eventdevs/index.rst         |  1 +
 doc/guides/rel_notes/release_20_11.rst |  5 +++++
 drivers/event/dlb2/meson.build         | 13 +++++++++++
 drivers/event/dlb2/version.map         |  3 +++
 drivers/event/meson.build              |  3 ++-
 8 files changed, 76 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index a3d1927..727451b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1162,6 +1162,11 @@ Cavium OCTEON TX timvf
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 F: drivers/event/octeontx/timvf_*
 
+Intel DLB2
+M: Timothy McDaniel <timothy.mcdaniel@intel.com>
+F: drivers/event/dlb2/
+F: doc/guides/eventdevs/dlb2.rst
+
 Marvell OCTEON TX2
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 M: Jerin Jacob <jerinj@marvell.com>
@@ -1198,7 +1203,6 @@ M: Peter Mccarthy <peter.mccarthy@intel.com>
 F: drivers/event/opdl/
 F: doc/guides/eventdevs/opdl.rst
 
-
 Rawdev Drivers
 --------------
 
diff --git a/config/rte_config.h b/config/rte_config.h
index b78c6aa..64aa333 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -135,4 +135,11 @@
 /* QEDE PMD defines */
 #define RTE_LIBRTE_QEDE_FW ""
 
+/* DLB2 defines */
+#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
+#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
+#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
+#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
+
 #endif /* _RTE_CONFIG_H_ */
diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
new file mode 100644
index 0000000..e3091c9
--- /dev/null
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -0,0 +1,40 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Driver for the Intel® Dynamic Load Balancer (DLB2)
+==================================================
+
+The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
+
+Prerequisites
+-------------
+
+Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
+the basic DPDK environment.
+
+Configuration
+-------------
+
+The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
+device access. To use this operation mode, the PCIe PF device must be bound
+to a DPDK-compatible VFIO driver, such as vfio-pci.
+
+Eventdev API Notes
+------------------
+
+The DLB2 provides the functions of a DPDK event device; specifically, it
+supports atomic, ordered, and parallel scheduling events from queues to ports.
+However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
+features are abstracted by the PMD such as directed ports.
+
+In general the dlb PMD is designed for ease-of-use and does not require a
+detailed understanding of the hardware, but these details are important when
+writing high-performance code. This section describes the places where the
+eventdev API and DLB2 misalign.
+
+Flow ID
+~~~~~~~
+
+The flow ID field is preserved in the event when it is scheduled in the
+DLB2.
+
diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
index bb66a5e..738788d 100644
--- a/doc/guides/eventdevs/index.rst
+++ b/doc/guides/eventdevs/index.rst
@@ -11,6 +11,7 @@ application through the eventdev API.
     :maxdepth: 2
     :numbered:
 
+    dlb2
     dpaa
     dpaa2
     dsw
diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
index d8ac359..dbce3b1 100644
--- a/doc/guides/rel_notes/release_20_11.rst
+++ b/doc/guides/rel_notes/release_20_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
+
+  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
+  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
+
 * **Added write combining store APIs.**
 
   Added ``rte_write32_wc`` and ``rte_write32_wc_relaxed`` APIs
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
new file mode 100644
index 0000000..5324043
--- /dev/null
+++ b/drivers/event/dlb2/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2020 Intel Corporation
+
+if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
+        build = false
+        reason = 'only supported on ARCH_X86_64 Linux'
+        subdir_done()
+endif
+
+sources = files(
+)
+
+deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
new file mode 100644
index 0000000..4a76d1d
--- /dev/null
+++ b/drivers/event/dlb2/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index a7dac99..c433c10 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -5,7 +5,8 @@ if is_windows
 	subdir_done()
 endif
 
-drivers = ['dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw', 'dsw']
+drivers = ['dlb2', 'dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw',
+	   'dsw']
 if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
-- 
2.6.4

[dpdk-dev] [PATCH v7 02/23] event/dlb2: add dynamic logging

[Timothy McDaniel]

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c      |  7 +++++++
 drivers/event/dlb2/dlb2_log.h  | 25 +++++++++++++++++++++++++
 drivers/event/dlb2/meson.build |  3 +--
 3 files changed, 33 insertions(+), 2 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_log.h

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
new file mode 100644
index 0000000..2082dd6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2.c
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <rte_log.h>
+
+RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
new file mode 100644
index 0000000..dc1481e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_log.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_EVDEV_LOG_H_
+#define _DLB2_EVDEV_LOG_H_
+
+extern int eventdev_dlb2_log_level;
+
+/* Dynamic logging */
+#define DLB2_LOG_IMPL(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
+		__func__, ##args)
+
+#define DLB2_LOG_INFO(fmt, args...) \
+	DLB2_LOG_IMPL(INFO, fmt, ## args)
+
+#define DLB2_LOG_ERR(fmt, args...) \
+	DLB2_LOG_IMPL(ERR, fmt, ## args)
+
+/* remove debug logs at compile time unless actually debugging */
+#define DLB2_LOG_DBG(fmt, args...) \
+	RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
+
+#endif /* _DLB2_EVDEV_LOG_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 5324043..435c1ee 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,7 +7,6 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files(
-)
+sources = files('dlb2.c')
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v7 03/23] event/dlb2: add private data structures and constants

[Timothy McDaniel]

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_priv.h | 575 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 575 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_priv.h

diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
new file mode 100644
index 0000000..61567a6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -0,0 +1,575 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_PRIV_H_
+#define _DLB2_PRIV_H_
+
+#include <emmintrin.h>
+#include <stdbool.h>
+
+#include <rte_eventdev.h>
+#include <rte_config.h>
+#include "dlb2_user.h"
+#include "dlb2_log.h"
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
+#else
+#define DLB2_INC_STAT(_stat, _incr_val)
+#endif
+
+#define EVDEV_DLB2_NAME_PMD dlb2_event
+
+/*  command line arg strings */
+#define NUMA_NODE_ARG "numa_node"
+#define DLB2_MAX_NUM_EVENTS "max_num_events"
+#define DLB2_NUM_DIR_CREDITS "num_dir_credits"
+#define DEV_ID_ARG "dev_id"
+#define DLB2_DEFER_SCHED_ARG "defer_sched"
+#define DLB2_QID_DEPTH_THRESH_ARG "qid_depth_thresh"
+#define DLB2_COS_ARG "cos"
+
+/* Begin HW related defines and structs */
+
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_VFS 16
+#define DLB2_MAX_NUM_LDB_QUEUES 32
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS 64
+#define DLB2_MAX_NUM_DIR_QUEUES 64
+#define DLB2_MAX_NUM_FLOWS (64 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDITS (8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS (2 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_DIR_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQOS_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_MAX_DEVICE_PATH 32
+#define DLB2_MIN_DEQUEUE_TIMEOUT_NS 1
+/* Note: "- 1" here to support the timeout range check in eventdev_autotest */
+#define DLB2_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1)
+#define DLB2_SW_CREDIT_BATCH_SZ 32
+#define DLB2_NUM_SN_GROUPS 2
+#define DLB2_MAX_LDB_SN_ALLOC 1024
+#define DLB2_MAX_QUEUE_DEPTH_THRESHOLD 8191
+
+/* 2048 total hist list entries and 64 total ldb ports, which
+ * makes for 2048/64 == 32 hist list entries per port. However, CQ
+ * depth must be a power of 2 and must also be >= HIST LIST entries.
+ * As a result we just limit the maximum dequeue depth to 32.
+ */
+#define DLB2_MIN_CQ_DEPTH 1
+#define DLB2_MAX_CQ_DEPTH 32
+#define DLB2_MIN_HARDWARE_CQ_DEPTH 8
+#define DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \
+	DLB2_MAX_CQ_DEPTH
+
+/*
+ * Static per queue/port provisioning values
+ */
+#define DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 64
+
+#define CQ_BASE(is_dir) ((is_dir) ? DLB2_DIR_CQ_BASE : DLB2_LDB_CQ_BASE)
+#define CQ_SIZE(is_dir) ((is_dir) ? DLB2_DIR_CQ_MAX_SIZE : \
+				    DLB2_LDB_CQ_MAX_SIZE)
+#define PP_BASE(is_dir) ((is_dir) ? DLB2_DIR_PP_BASE : DLB2_LDB_PP_BASE)
+
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+
+#define DLB2_NUM_QES_PER_CACHE_LINE 4
+
+#define DLB2_MAX_ENQUEUE_DEPTH 64
+#define DLB2_MIN_ENQUEUE_DEPTH 4
+
+#define DLB2_NAME_SIZE 64
+
+#define DLB2_1K 1024
+#define DLB2_2K (2 * DLB2_1K)
+#define DLB2_4K (4 * DLB2_1K)
+#define DLB2_16K (16 * DLB2_1K)
+#define DLB2_32K (32 * DLB2_1K)
+#define DLB2_1MB (DLB2_1K * DLB2_1K)
+#define DLB2_16MB (16 * DLB2_1MB)
+
+/* Use the upper 3 bits of the event priority to select the DLB2 priority */
+#define EV_TO_DLB2_PRIO(x) ((x) >> 5)
+#define DLB2_TO_EV_PRIO(x) ((x) << 5)
+
+enum dlb2_hw_port_types {
+	DLB2_LDB_PORT,
+	DLB2_DIR_PORT,
+	DLB2_NUM_PORT_TYPES /* Must be last */
+};
+
+enum dlb2_hw_queue_types {
+	DLB2_LDB_QUEUE,
+	DLB2_DIR_QUEUE,
+	DLB2_NUM_QUEUE_TYPES /* Must be last */
+};
+
+#define PORT_TYPE(p) ((p)->is_directed ? DLB2_DIR_PORT : DLB2_LDB_PORT)
+
+/* Do not change - must match hardware! */
+enum dlb2_hw_sched_type {
+	DLB2_SCHED_ATOMIC = 0,
+	DLB2_SCHED_UNORDERED,
+	DLB2_SCHED_ORDERED,
+	DLB2_SCHED_DIRECTED,
+	/* DLB2_NUM_HW_SCHED_TYPES must be last */
+	DLB2_NUM_HW_SCHED_TYPES
+};
+
+struct dlb2_hw_rsrcs {
+	int32_t nb_events_limit;
+	uint32_t num_queues;		/* Total queues (lb + dir) */
+	uint32_t num_ldb_queues;	/* Number of available ldb queues */
+	uint32_t num_ldb_ports;         /* Number of load balanced ports */
+	uint32_t num_dir_ports;         /* Number of directed ports */
+	uint32_t num_ldb_credits;       /* Number of load balanced credits */
+	uint32_t num_dir_credits;       /* Number of directed credits */
+	uint32_t reorder_window_size;   /* Size of reorder window */
+};
+
+struct dlb2_hw_resource_info {
+	/**> Max resources that can be provided */
+	struct dlb2_hw_rsrcs hw_rsrc_max;
+	int num_sched_domains;
+	uint32_t socket_id;
+};
+
+enum dlb2_enqueue_type {
+	/**>
+	 * New : Used to inject a new packet into the QM.
+	 */
+	DLB2_ENQ_NEW,
+	/**>
+	 * Forward : Enqueues a packet, and
+	 *  - if atomic: release any lock it holds in the QM
+	 *  - if ordered: release the packet for egress re-ordering
+	 */
+	DLB2_ENQ_FWD,
+	/**>
+	 * Enqueue Drop : Release an inflight packet. Must be called with
+	 * event == NULL. Used to drop a packet.
+	 *
+	 * Note that all packets dequeued from a load-balanced port must be
+	 * released, either with DLB2_ENQ_DROP or DLB2_ENQ_FWD.
+	 */
+	DLB2_ENQ_DROP,
+
+	/* marker for array sizing etc. */
+	_DLB2_NB_ENQ_TYPES
+};
+
+/* hw-specific format - do not change */
+
+struct dlb2_event_type {
+	uint8_t major:4;
+	uint8_t unused:4;
+	uint8_t sub;
+};
+
+union dlb2_opaque_data {
+	uint16_t opaque_data;
+	struct dlb2_event_type event_type;
+};
+
+struct dlb2_msg_info {
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+};
+
+#define DLB2_NEW_CMD_BYTE 0x08
+#define DLB2_FWD_CMD_BYTE 0x0A
+#define DLB2_COMP_CMD_BYTE 0x02
+#define DLB2_POP_CMD_BYTE 0x01
+#define DLB2_NOOP_CMD_BYTE 0x00
+
+/* hw-specific format - do not change */
+struct dlb2_enqueue_qe {
+	uint64_t data;
+	/* Word 3 */
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	/* Word 4 */
+	uint16_t lock_id;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_cq_pop_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t tokens:10;
+	uint16_t rsvd2:6;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_dequeue_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t flow_id:16; /* was pp_id in v1 */
+	uint8_t debug;
+	uint8_t cq_gen:1;
+	uint8_t qid_depth:2; /* 2 bits in v2 */
+	uint8_t rsvd1:2;
+	uint8_t error:1;
+	uint8_t rsvd2:2;
+};
+
+union dlb2_port_config {
+	struct dlb2_create_ldb_port_args ldb;
+	struct dlb2_create_dir_port_args dir;
+};
+
+enum dlb2_port_state {
+	PORT_CLOSED,
+	PORT_STARTED,
+	PORT_STOPPED
+};
+
+enum dlb2_configuration_state {
+	/* The resource has not been configured */
+	DLB2_NOT_CONFIGURED,
+	/* The resource was configured, but the device was stopped */
+	DLB2_PREV_CONFIGURED,
+	/* The resource is currently configured */
+	DLB2_CONFIGURED
+};
+
+struct dlb2_port {
+	uint32_t id;
+	bool is_directed;
+	bool gen_bit;
+	uint16_t dir_credits;
+	uint32_t dequeue_depth;
+	union dlb2_port_config cfg;
+	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
+	uint16_t cached_ldb_credits;
+	uint16_t ldb_credits;
+	uint16_t cached_dir_credits;
+	bool int_armed;
+	uint16_t owed_tokens;
+	int16_t issued_releases;
+	int cq_depth;
+	uint16_t cq_idx;
+	uint16_t cq_idx_unmasked;
+	uint16_t cq_depth_mask;
+	uint16_t gen_bit_shift;
+	enum dlb2_port_state state;
+	enum dlb2_configuration_state config_state;
+	int num_mapped_qids;
+	uint8_t *qid_mappings;
+	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
+	struct dlb2_enqueue_qe *int_arm_qe;
+	struct dlb2_cq_pop_qe *consume_qe;
+	struct dlb2_eventdev *dlb2; /* back ptr */
+	struct dlb2_eventdev_port *ev_port; /* back ptr */
+};
+
+/* Per-process per-port mmio and memory pointers */
+struct process_local_port_data {
+	uint64_t *pp_addr;
+	struct dlb2_dequeue_qe *cq_base;
+	const struct rte_memzone *mz;
+	bool mmaped;
+};
+
+struct dlb2_eventdev;
+
+struct dlb2_config {
+	int configured;
+	int reserved;
+	uint32_t num_ldb_credits;
+	uint32_t num_dir_credits;
+	struct dlb2_create_sched_domain_args resources;
+};
+
+enum dlb2_cos {
+	DLB2_COS_DEFAULT = -1,
+	DLB2_COS_0 = 0,
+	DLB2_COS_1,
+	DLB2_COS_2,
+	DLB2_COS_3
+};
+
+struct dlb2_hw_dev {
+	struct dlb2_config cfg;
+	struct dlb2_hw_resource_info info;
+	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
+	uint32_t domain_id;
+	enum dlb2_cos cos_id;
+	rte_spinlock_t resource_lock; /* for MP support */
+} __rte_cache_aligned;
+
+/* End HW related defines and structs */
+
+/* Begin DLB2 PMD Eventdev related defines and structs */
+
+#define DLB2_MAX_NUM_QUEUES \
+	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
+
+#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS + DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
+
+/** Structure to hold the queue to port link establishment attributes */
+
+struct dlb2_event_queue_link {
+	uint8_t queue_id;
+	uint8_t priority;
+	bool mapped;
+	bool valid;
+};
+
+struct dlb2_traffic_stats {
+	uint64_t rx_ok;
+	uint64_t rx_drop;
+	uint64_t rx_interrupt_wait;
+	uint64_t rx_umonitor_umwait;
+	uint64_t tx_ok;
+	uint64_t total_polls;
+	uint64_t zero_polls;
+	uint64_t tx_nospc_ldb_hw_credits;
+	uint64_t tx_nospc_dir_hw_credits;
+	uint64_t tx_nospc_inflight_max;
+	uint64_t tx_nospc_new_event_limit;
+	uint64_t tx_nospc_inflight_credits;
+};
+
+/* DLB2 HW sets the 2bit qid_depth in rx QEs based on the programmable depth
+ * threshold. The global default value in config/common_base (or rte_config.h)
+ * can be overridden on a per-qid basis using a vdev command line parameter.
+ * 3: depth > threshold
+ * 2: threshold >= depth > 3/4 threshold
+ * 1: 3/4 threshold >= depth > 1/2 threshold
+ * 0: depth <= 1/2 threshold.
+ */
+#define DLB2_QID_DEPTH_LE50 0
+#define DLB2_QID_DEPTH_GT50_LE75 1
+#define DLB2_QID_DEPTH_GT75_LE100 2
+#define DLB2_QID_DEPTH_GT100 3
+#define DLB2_NUM_QID_DEPTH_STAT_VALS 4 /* 2 bits */
+
+struct dlb2_queue_stats {
+	uint64_t enq_ok;
+	uint64_t qid_depth[DLB2_NUM_QID_DEPTH_STAT_VALS];
+};
+
+struct dlb2_port_stats {
+	struct dlb2_traffic_stats traffic;
+	uint64_t tx_op_cnt[4]; /* indexed by rte_event.op */
+	uint64_t tx_implicit_rel;
+	uint64_t tx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t tx_invalid;
+	uint64_t rx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t rx_sched_invalid;
+	struct dlb2_queue_stats queue[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_eventdev_port {
+	struct dlb2_port qm_port; /* hw specific data structure */
+	struct rte_event_port_conf conf; /* user-supplied configuration */
+	uint16_t inflight_credits; /* num credits this port has right now */
+	uint16_t credit_update_quanta;
+	struct dlb2_eventdev *dlb2; /* backlink optimization */
+	struct dlb2_port_stats stats __rte_cache_aligned;
+	struct dlb2_event_queue_link link[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	int num_links;
+	uint32_t id; /* port id */
+	/* num releases yet to be completed on this port.
+	 * Only applies to load-balanced ports.
+	 */
+	uint16_t outstanding_releases;
+	uint16_t inflight_max; /* app requested max inflights for this port */
+	/* setup_done is set when the event port is setup */
+	bool setup_done;
+	/* enq_configured is set when the qm port is created */
+	bool enq_configured;
+	uint8_t implicit_release; /* release events before dequeueing */
+}  __rte_cache_aligned;
+
+struct dlb2_queue {
+	uint32_t num_qid_inflights; /* User config */
+	uint32_t num_atm_inflights; /* User config */
+	enum dlb2_configuration_state config_state;
+	int sched_type; /* LB queue only */
+	uint32_t id;
+	bool is_directed;
+};
+
+struct dlb2_eventdev_queue {
+	struct dlb2_queue qm_queue;
+	struct rte_event_queue_conf conf; /* User config */
+	int depth_threshold; /* use default if 0 */
+	uint32_t id;
+	bool setup_done;
+	uint8_t num_links;
+};
+
+enum dlb2_run_state {
+	DLB2_RUN_STATE_STOPPED = 0,
+	DLB2_RUN_STATE_STOPPING,
+	DLB2_RUN_STATE_STARTING,
+	DLB2_RUN_STATE_STARTED
+};
+
+struct dlb2_eventdev {
+	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
+	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each queue */
+	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
+	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each port */
+	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
+	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
+	struct dlb2_get_num_resources_args hw_rsrc_query_results;
+	uint32_t xstats_count_mode_queue;
+	struct dlb2_hw_dev qm_instance; /* strictly hw related */
+	uint64_t global_dequeue_wait_ticks;
+	struct dlb2_xstats_entry *xstats;
+	struct rte_eventdev *event_dev; /* backlink to dev */
+	uint32_t xstats_count_mode_dev;
+	uint32_t xstats_count_mode_port;
+	uint32_t xstats_count;
+	uint32_t inflights; /* use __atomic builtins */
+	uint32_t new_event_limit;
+	int max_num_events_override;
+	int num_dir_credits_override;
+	volatile enum dlb2_run_state run_state;
+	uint16_t num_dir_queues; /* total num of evdev dir queues requested */
+	uint16_t num_dir_credits;
+	uint16_t num_ldb_credits;
+	uint16_t num_queues; /* total queues */
+	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested */
+	uint16_t num_ports; /* total num of evdev ports requested */
+	uint16_t num_ldb_ports; /* total num of ldb ports requested */
+	uint16_t num_dir_ports; /* total num of dir ports requested */
+	bool umwait_allowed;
+	bool global_dequeue_wait; /* Not using per dequeue wait if true */
+	bool defer_sched;
+	enum dlb2_cq_poll_modes poll_mode;
+	uint8_t revision;
+	bool configured;
+	uint16_t max_ldb_credits;
+	uint16_t max_dir_credits;
+
+	/* force hw credit pool counters into exclusive cache lines */
+
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t ldb_credit_pool __rte_cache_aligned;
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t dir_credit_pool __rte_cache_aligned;
+};
+
+/* used for collecting and passing around the dev args */
+struct dlb2_qid_depth_thresholds {
+	int val[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_devargs {
+	int socket_id;
+	int max_num_events;
+	int num_dir_credits_override;
+	int dev_id;
+	int defer_sched;
+	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
+	enum dlb2_cos cos_id;
+};
+
+/* End Eventdev related defines and structs */
+
+/* Forwards for non-inlined functions */
+
+void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
+
+int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
+
+void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
+
+int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n);
+
+int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstat_names,
+		unsigned int *ids, unsigned int size);
+
+uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+					  const char *name, unsigned int *id);
+
+int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode,
+		int16_t queue_port_id,
+		const uint32_t ids[],
+		uint32_t nb_ids);
+
+int test_dlb2_eventdev(void);
+
+int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+				const char *name,
+				struct dlb2_devargs *dlb2_args);
+
+int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+				  const char *name);
+
+uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *queue);
+
+int dlb2_parse_params(const char *params,
+		      const char *name,
+		      struct dlb2_devargs *dlb2_args);
+
+/* Extern globals */
+extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
+
+#endif	/* _DLB2_PRIV_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v7 04/23] event/dlb2: add definitions shared with LKM or shared code

[Timothy McDaniel]

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_user.h | 679 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 679 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_user.h

diff --git a/drivers/event/dlb2/dlb2_user.h b/drivers/event/dlb2/dlb2_user.h
new file mode 100644
index 0000000..f4bda78
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_user.h
@@ -0,0 +1,679 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#define DLB2_MAX_NAME_LEN 64
+
+#include <linux/types.h>
+
+enum dlb2_error {
+	DLB2_ST_SUCCESS = 0,
+	DLB2_ST_NAME_EXISTS,
+	DLB2_ST_DOMAIN_UNAVAILABLE,
+	DLB2_ST_LDB_PORTS_UNAVAILABLE,
+	DLB2_ST_DIR_PORTS_UNAVAILABLE,
+	DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB2_ST_INVALID_DOMAIN_ID,
+	DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB2_ST_INVALID_LDB_QUEUE_ID,
+	DLB2_ST_INVALID_CQ_DEPTH,
+	DLB2_ST_INVALID_CQ_VIRT_ADDR,
+	DLB2_ST_INVALID_PORT_ID,
+	DLB2_ST_INVALID_QID,
+	DLB2_ST_INVALID_PRIORITY,
+	DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+	DLB2_ST_INVALID_DIR_QUEUE_ID,
+	DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB2_ST_DOMAIN_NOT_CONFIGURED,
+	DLB2_ST_INTERNAL_ERROR,
+	DLB2_ST_DOMAIN_IN_USE,
+	DLB2_ST_DOMAIN_NOT_FOUND,
+	DLB2_ST_QUEUE_NOT_FOUND,
+	DLB2_ST_DOMAIN_STARTED,
+	DLB2_ST_DOMAIN_NOT_STARTED,
+	DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB2_ST_DOMAIN_RESET_FAILED,
+	DLB2_ST_MBOX_ERROR,
+	DLB2_ST_INVALID_HIST_LIST_DEPTH,
+	DLB2_ST_NO_MEMORY,
+	DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB2_ST_INVALID_COS_ID,
+};
+
+static const char dlb2_error_strings[][128] = {
+	"DLB2_ST_SUCCESS",
+	"DLB2_ST_NAME_EXISTS",
+	"DLB2_ST_DOMAIN_UNAVAILABLE",
+	"DLB2_ST_LDB_PORTS_UNAVAILABLE",
+	"DLB2_ST_DIR_PORTS_UNAVAILABLE",
+	"DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+	"DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+	"DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+	"DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+	"DLB2_ST_INVALID_DOMAIN_ID",
+	"DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+	"DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+	"DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+	"DLB2_ST_INVALID_LDB_QUEUE_ID",
+	"DLB2_ST_INVALID_CQ_DEPTH",
+	"DLB2_ST_INVALID_CQ_VIRT_ADDR",
+	"DLB2_ST_INVALID_PORT_ID",
+	"DLB2_ST_INVALID_QID",
+	"DLB2_ST_INVALID_PRIORITY",
+	"DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+	"DLB2_ST_INVALID_DIR_QUEUE_ID",
+	"DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+	"DLB2_ST_DOMAIN_NOT_CONFIGURED",
+	"DLB2_ST_INTERNAL_ERROR",
+	"DLB2_ST_DOMAIN_IN_USE",
+	"DLB2_ST_DOMAIN_NOT_FOUND",
+	"DLB2_ST_QUEUE_NOT_FOUND",
+	"DLB2_ST_DOMAIN_STARTED",
+	"DLB2_ST_DOMAIN_NOT_STARTED",
+	"DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+	"DLB2_ST_DOMAIN_RESET_FAILED",
+	"DLB2_ST_MBOX_ERROR",
+	"DLB2_ST_INVALID_HIST_LIST_DEPTH",
+	"DLB2_ST_NO_MEMORY",
+	"DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+	"DLB2_ST_INVALID_COS_ID",
+};
+
+struct dlb2_cmd_response {
+	__u32 status; /* Interpret using enum dlb2_error */
+	__u32 id;
+};
+
+/*******************/
+/* 'dlb2' commands */
+/*******************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+	DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	This ioctl interface is the same in all driver versions and is always
+ *	the first ioctl.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+	/* Output parameters */
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ *	load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ *		percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ *	load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ *   ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+enum dlb2_cq_poll_modes {
+	DLB2_CQ_POLL_MODE_STD,
+	DLB2_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB2_CQ_POLL_MODE must be last */
+	NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/********************************/
+/* 'scheduling domain' commands */
+/********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_start_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
+#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
+
+
+#endif /* __DLB2_USER_H */
-- 
2.6.4

[dpdk-dev] [PATCH v7 05/23] event/dlb2: add inline functions

[Timothy McDaniel]

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_inline_fns.h | 61 ++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h

diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
new file mode 100644
index 0000000..10bd062
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_inline_fns.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_INLINE_FNS_H_
+#define _DLB2_INLINE_FNS_H_
+
+/* Inline functions required in more than one source file. */
+
+static inline struct dlb2_eventdev *
+dlb2_pmd_priv(const struct rte_eventdev *eventdev)
+{
+	return eventdev->data->dev_private;
+}
+
+static inline void
+dlb2_umonitor(volatile void *addr)
+{
+	asm volatile(".byte 0xf3, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (addr));
+}
+
+static inline void
+dlb2_umwait(int state, uint64_t timeout)
+{
+	uint32_t eax = timeout & UINT32_MAX;
+	uint32_t edx = timeout >> 32;
+
+	asm volatile(".byte 0xf2, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (state),  "a" (eax), "d" (edx));
+}
+
+static inline void
+dlb2_movntdq_single(void *pp_addr, void *qe4)
+{
+	long long *_qe  = (long long *)qe4;
+	__v2di src_data0 = (__v2di){_qe[0], _qe[1]};
+
+	_mm_stream_si128(pp_addr, src_data0);
+}
+
+static inline void
+dlb2_cldemote(void *addr)
+{
+	/* Load addr into RSI, then demote the cache line of the address
+	 * contained in that register.
+	 */
+	asm volatile(".byte 0x0f, 0x1c, 0x06" :: "S" (addr));
+}
+
+static inline void
+dlb2_movdir64b(void *pp_addr, void *qe4)
+{
+	asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
+		     :
+		     : "a" (pp_addr), "d" (qe4));
+}
+
+#endif /* _DLB2_INLINE_FNS_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v7 06/23] event/dlb2: add eventdev probe

[Timothy McDaniel]

Add the eventdev portion of probe, and parse command line
options, but do not initialize hardware.

Changes since v2 patch-set probe:

Primary and secondary probe-time init has been removed, and
will be introduced in subsequent patches contained in
the v3 patch-set.

Hardware init has been moved to a subsequent patch in order to
minimize the patch size. The files dlb2/pf/dlb2_pf.c and
dlb2/pf/dlb_main.c include stubs of hardware init functions,
and these will be replaced with the real versions when the
hardware layer is committed.

Initialization of the flexible interface layer has been moved to
a subsequent patch in order to minimize patch size.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                      |  343 ++++
 drivers/event/dlb2/meson.build                 |    5 +-
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +++
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  621 ++++++
 drivers/event/dlb2/pf/dlb2_main.h              |   98 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  196 ++
 13 files changed, 7497 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2082dd6..1b11979 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2,6 +2,349 @@
  * Copyright(c) 2016-2020 Intel Corporation
  */
 
+#include <assert.h>
+#include <errno.h>
+#include <nmmintrin.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_cycles.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_io.h>
+#include <rte_kvargs.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_ring.h>
+#include <rte_string_fns.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+/*
+ * Resources exposed to eventdev. Some values overridden at runtime using
+ * values returned by the DLB kernel driver.
+ */
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
+#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
+#endif
+
+struct process_local_port_data
+dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
+
+#define DLB2_BASE_10 10
+
+static int
+dlb2_string_to_int(int *result, const char *str)
+{
+	long ret;
+	char *endptr;
+
+	if (str == NULL || result == NULL)
+		return -EINVAL;
+
+	errno = 0;
+	ret = strtol(str, &endptr, DLB2_BASE_10);
+	if (errno)
+		return -errno;
+
+	/* long int and int may be different width for some architectures */
+	if (ret < INT_MIN || ret > INT_MAX || endptr == str)
+		return -EINVAL;
+
+	*result = ret;
+	return 0;
+}
+
+static int
+set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
+{
+	int *socket_id = opaque;
+	int ret;
+
+	ret = dlb2_string_to_int(socket_id, value);
+	if (ret < 0)
+		return ret;
+
+	if (*socket_id > RTE_MAX_NUMA_NODES)
+		return -EINVAL;
+	return 0;
+}
+
+static int
+set_max_num_events(const char *key __rte_unused,
+		   const char *value,
+		   void *opaque)
+{
+	int *max_num_events = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(max_num_events, value);
+	if (ret < 0)
+		return ret;
+
+	if (*max_num_events < 0 || *max_num_events >
+			DLB2_MAX_NUM_LDB_CREDITS) {
+		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
+			     DLB2_MAX_NUM_LDB_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_num_dir_credits(const char *key __rte_unused,
+		    const char *value,
+		    void *opaque)
+{
+	int *num_dir_credits = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(num_dir_credits, value);
+	if (ret < 0)
+		return ret;
+
+	if (*num_dir_credits < 0 ||
+	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
+		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
+			     DLB2_MAX_NUM_DIR_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_dev_id(const char *key __rte_unused,
+	   const char *value,
+	   void *opaque)
+{
+	int *dev_id = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(dev_id, value);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+set_cos(const char *key __rte_unused,
+	const char *value,
+	void *opaque)
+{
+	enum dlb2_cos *cos_id = opaque;
+	int x = 0;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(&x, value);
+	if (ret < 0)
+		return ret;
+
+	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
+		DLB2_LOG_ERR(
+			"COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
+			x);
+		return -EINVAL;
+	}
+
+	*cos_id = x;
+
+	return 0;
+}
+
+
+static int
+set_qid_depth_thresh(const char *key __rte_unused,
+		     const char *value,
+		     void *opaque)
+{
+	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
+	int first, last, thresh, i;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	/* command line override may take one of the following 3 forms:
+	 * qid_depth_thresh=all:<threshold_value> ... all queues
+	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
+	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
+	 */
+	if (sscanf(value, "all:%d", &thresh) == 1) {
+		first = 0;
+		last = DLB2_MAX_NUM_QUEUES - 1;
+	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
+		/* we have everything we need */
+	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
+		last = first;
+	} else {
+		DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val, qid-qid:val, or qid:val\n");
+		return -EINVAL;
+	}
+
+	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
+		return -EINVAL;
+	}
+
+	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
+			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
+		return -EINVAL;
+	}
+
+	for (i = first; i <= last; i++)
+		qid_thresh->val[i] = thresh; /* indexed by qid */
+
+	return 0;
+}
+
+int
+dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+			    const char *name,
+			    struct dlb2_devargs *dlb2_args)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+	RTE_SET_USED(dlb2_args);
+
+	return 0;
+}
+
+int
+dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+			      const char *name)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+int
+dlb2_parse_params(const char *params,
+		  const char *name,
+		  struct dlb2_devargs *dlb2_args)
+{
+	int ret = 0;
+	static const char * const args[] = { NUMA_NODE_ARG,
+					     DLB2_MAX_NUM_EVENTS,
+					     DLB2_NUM_DIR_CREDITS,
+					     DEV_ID_ARG,
+					     DLB2_QID_DEPTH_THRESH_ARG,
+					     DLB2_COS_ARG,
+					     NULL };
+
+	if (params != NULL && params[0] != '\0') {
+		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
+
+		if (kvlist == NULL) {
+			RTE_LOG(INFO, PMD,
+				"Ignoring unsupported parameters when creating device '%s'\n",
+				name);
+		} else {
+			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
+						     set_numa_node,
+						     &dlb2_args->socket_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing numa node parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
+						 set_max_num_events,
+						 &dlb2_args->max_num_events);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist,
+					DLB2_NUM_DIR_CREDITS,
+					set_num_dir_credits,
+					&dlb2_args->num_dir_credits_override);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
+						 set_dev_id,
+						 &dlb2_args->dev_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(
+					kvlist,
+					DLB2_QID_DEPTH_THRESH_ARG,
+					set_qid_depth_thresh,
+					&dlb2_args->qid_depth_thresholds);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
+						 set_cos,
+						 &dlb2_args->cos_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing cos parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
 
+			rte_kvargs_free(kvlist);
+		}
+	}
+	return ret;
+}
 RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 435c1ee..1eab9b9 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,6 +7,9 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files('dlb2.c')
+sources = files('dlb2.c',
+		'pf/dlb2_main.c',
+		'pf/dlb2_pf.c'
+)
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
new file mode 100644
index 0000000..1d99f1e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_osdep_list.h"
+#include "dlb2_osdep_types.h"
+
+#define DLB2_MAX_NUM_VDEVS			16
+#define DLB2_MAX_NUM_DOMAINS			32
+#define DLB2_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS			64
+#define DLB2_MAX_NUM_DIR_PORTS			64
+#define DLB2_MAX_NUM_LDB_CREDITS		(8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS		(2 * 1024)
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB2_MAX_NUM_AQED_ENTRIES		2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB2_QID_PRIORITIES			8
+#define DLB2_NUM_ARB_WEIGHTS			8
+#define DLB2_MAX_WEIGHT				255
+#define DLB2_NUM_COS_DOMAINS			4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#ifdef FPGA
+#define DLB2_HZ					2000000
+#else
+#define DLB2_HZ					800000000
+#endif
+
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
+
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+	DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+	DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID		3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID		4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW		5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ	1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
+#define DLB2_VF_BASE_CQ_VECTOR_ID	     0
+#define DLB2_VF_LAST_CQ_VECTOR_ID	     30
+#define DLB2_VF_MBOX_VECTOR_ID		     31
+#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
+
+#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
+					     DLB2_MAX_NUM_DIR_PORTS + 1)
+
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE   0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
+				DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE   0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
+				DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE       0x2100000
+#define DLB2_LDB_PP_STRIDE     0x1000
+#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
+				DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE       0x2000000
+#define DLB2_DIR_PP_STRIDE     0x1000
+#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
+				DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
+struct dlb2_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb2_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb2_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
+struct dlb2_ldb_queue {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB2_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB2_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB2_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB2_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+	enum dlb2_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb2_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+	u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+	u32 bound[6] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & (1 << i))) {
+			group->slot_use_bitmap |= 1 << i;
+			return i;
+		}
+	}
+
+	return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+	int i, cnt = 0;
+
+	for (i = 0; i < 32; i++)
+		cnt += !!(group->slot_use_bitmap & (1 << i));
+
+	return cnt;
+}
+
+struct dlb2_hw_domain {
+	struct dlb2_function_resources *parent_func;
+	struct dlb2_list_entry func_list;
+	struct dlb2_list_head used_ldb_queues;
+	struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head used_dir_pq_pairs;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb2_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb2_bitmap;
+
+struct dlb2_function_resources {
+	struct dlb2_list_head avail_domains;
+	struct dlb2_list_head used_domains;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	struct dlb2_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+	struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+	struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+	struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
+struct dlb2_mbox {
+	u32 *mbox;
+	u32 *isr_in_progress;
+};
+
+struct dlb2_sw_mbox {
+	struct dlb2_mbox vdev_to_pf;
+	struct dlb2_mbox pf_to_vdev;
+	void (*pf_to_vdev_inject)(void *arg);
+	void *pf_to_vdev_inject_arg;
+};
+
+struct dlb2_hw {
+	/* BAR 0 address */
+	void  *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void  *func_kva;
+	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb2_hw_resources rsrcs;
+	struct dlb2_function_resources pf;
+	struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+	struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+	/* Virtualization */
+	int virt_mode;
+	struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
+	unsigned int pasid[DLB2_MAX_NUM_VDEVS];
+};
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
new file mode 100644
index 0000000..ce462c0
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
@@ -0,0 +1,596 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_BASE_DLB2_MBOX_H
+#define __DLB2_BASE_DLB2_MBOX_H
+
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+
+#define DLB2_MBOX_INTERFACE_VERSION 1
+
+/*
+ * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
+ * to send requests of its own (e.g. notifying a VF of an impending FLR).
+ * To avoid communication race conditions, e.g. the PF sends a response and then
+ * sends a request before the VF reads the response, the PF->VF mailbox is
+ * divided into two sections:
+ * - Bytes 0-47: PF responses
+ * - Bytes 48-63: PF requests
+ *
+ * Partitioning the PF->VF mailbox allows responses and requests to occupy the
+ * mailbox simultaneously.
+ */
+#define DLB2_PF2VF_RESP_BYTES	  48
+#define DLB2_PF2VF_RESP_BASE	  0
+#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
+
+#define DLB2_PF2VF_REQ_BYTES	  16
+#define DLB2_PF2VF_REQ_BASE	  (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
+#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
+
+/*
+ * Similarly, the VF->PF mailbox is divided into two sections:
+ * - Bytes 0-239: VF requests
+ * -- (Bytes 0-3 are unused due to a hardware errata)
+ * - Bytes 240-255: VF responses
+ */
+#define DLB2_VF2PF_REQ_BYTES	 236
+#define DLB2_VF2PF_REQ_BASE	 4
+#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
+
+#define DLB2_VF2PF_RESP_BYTES	  16
+#define DLB2_VF2PF_RESP_BASE	  (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
+#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
+
+/* VF-initiated commands */
+enum dlb2_mbox_cmd_type {
+	DLB2_MBOX_CMD_REGISTER,
+	DLB2_MBOX_CMD_UNREGISTER,
+	DLB2_MBOX_CMD_GET_NUM_RESOURCES,
+	DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
+	DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
+	DLB2_MBOX_CMD_CREATE_LDB_PORT,
+	DLB2_MBOX_CMD_CREATE_DIR_PORT,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT,
+	DLB2_MBOX_CMD_DISABLE_LDB_PORT,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT,
+	DLB2_MBOX_CMD_DISABLE_DIR_PORT,
+	DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_MAP_QID,
+	DLB2_MBOX_CMD_UNMAP_QID,
+	DLB2_MBOX_CMD_START_DOMAIN,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
+	DLB2_MBOX_CMD_ARM_CQ_INTR,
+	DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
+	DLB2_MBOX_CMD_GET_SN_ALLOCATION,
+	DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
+	DLB2_MBOX_CMD_GET_COS_BW,
+	DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
+	DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_QE_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_CMD_TYPES,
+};
+
+static const char dlb2_mbox_cmd_type_strings[][128] = {
+	"DLB2_MBOX_CMD_REGISTER",
+	"DLB2_MBOX_CMD_UNREGISTER",
+	"DLB2_MBOX_CMD_GET_NUM_RESOURCES",
+	"DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_LDB_PORT",
+	"DLB2_MBOX_CMD_CREATE_DIR_PORT",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_DISABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_DISABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_MAP_QID",
+	"DLB2_MBOX_CMD_UNMAP_QID",
+	"DLB2_MBOX_CMD_START_DOMAIN",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
+	"DLB2_MBOX_CMD_ARM_CQ_INTR",
+	"DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
+	"DLB2_MBOX_CMD_GET_SN_ALLOCATION",
+	"DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
+	"DLB2_MBOX_CMD_GET_COS_BW",
+	"DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
+	"DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
+};
+
+/* PF-initiated commands */
+enum dlb2_mbox_vf_cmd_type {
+	DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
+	DLB2_MBOX_VF_CMD_NOTIFICATION,
+	DLB2_MBOX_VF_CMD_IN_USE,
+
+	/* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_VF_CMD_TYPES,
+};
+
+static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
+	"DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
+	"DLB2_MBOX_VF_CMD_NOTIFICATION",
+	"DLB2_MBOX_VF_CMD_IN_USE",
+};
+
+#define DLB2_MBOX_CMD_TYPE(hdr) \
+	(((struct dlb2_mbox_req_hdr *)hdr)->type)
+#define DLB2_MBOX_CMD_STRING(hdr) \
+	dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
+
+enum dlb2_mbox_status_type {
+	DLB2_MBOX_ST_SUCCESS,
+	DLB2_MBOX_ST_INVALID_CMD_TYPE,
+	DLB2_MBOX_ST_VERSION_MISMATCH,
+	DLB2_MBOX_ST_INVALID_OWNER_VF,
+};
+
+static const char dlb2_mbox_status_type_strings[][128] = {
+	"DLB2_MBOX_ST_SUCCESS",
+	"DLB2_MBOX_ST_INVALID_CMD_TYPE",
+	"DLB2_MBOX_ST_VERSION_MISMATCH",
+	"DLB2_MBOX_ST_INVALID_OWNER_VF",
+};
+
+#define DLB2_MBOX_ST_TYPE(hdr) \
+	(((struct dlb2_mbox_resp_hdr *)hdr)->status)
+#define DLB2_MBOX_ST_STRING(hdr) \
+	dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
+
+/* This structure is always the first field in a request structure */
+struct dlb2_mbox_req_hdr {
+	u32 type;
+};
+
+/* This structure is always the first field in a response structure */
+struct dlb2_mbox_resp_hdr {
+	u32 status;
+};
+
+struct dlb2_mbox_register_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 min_interface_version;
+	u16 max_interface_version;
+};
+
+struct dlb2_mbox_register_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 interface_version;
+	u8 pf_id;
+	u8 vf_id;
+	u8 is_auxiliary_vf;
+	u8 primary_vf_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u16 num_sched_domains;
+	u16 num_ldb_queues;
+	u16 num_ldb_ports;
+	u16 num_cos_ldb_ports[4];
+	u16 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 max_contiguous_hist_list_entries;
+	u16 num_ldb_credits;
+	u16 num_dir_credits;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 num_ldb_queues;
+	u32 num_ldb_ports;
+	u32 num_cos_ldb_ports[4];
+	u32 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u8 cos_strict;
+	u8 padding0[3];
+	u32 padding1;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 num_sequence_numbers;
+	u32 num_qid_inflights;
+	u32 num_atomic_inflights;
+	u32 lock_id_comp_level;
+	u32 depth_threshold;
+	u32 padding;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 depth_threshold;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u16 cq_depth;
+	u16 cq_history_list_size;
+	u8 cos_id;
+	u8 cos_strict;
+	u16 padding1;
+	u64 cq_base_address;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u64 cq_base_address;
+	u16 cq_depth;
+	u16 padding0;
+	s32 queue_id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_map_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+	u32 priority;
+	u32 padding0;
+};
+
+struct dlb2_mbox_map_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_start_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+};
+
+struct dlb2_mbox_start_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 is_ldb;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding0;
+};
+
+/*
+ * The alert_id and aux_alert_data follows the format of the alerts defined in
+ * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
+ * the aux_alert_data value varies depending on the alert.
+ */
+struct dlb2_mbox_vf_alert_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 alert_id;
+	u32 aux_alert_data;
+};
+
+enum dlb2_mbox_vf_notification_type {
+	DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
+	DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
+
+	/* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
+	NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
+};
+
+struct dlb2_mbox_vf_notification_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 notification;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 in_use;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 num;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 cos_id;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 mode;
+};
+
+#endif /* __DLB2_BASE_DLB2_MBOX_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
new file mode 100644
index 0000000..c4c34eb
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
@@ -0,0 +1,230 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_H
+#define __DLB2_OSDEP_H
+
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <rte_string_fns.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include "../dlb2_main.h"
+#include "dlb2_resource.h"
+#include "../../dlb2_log.h"
+#include "../../dlb2_user.h"
+
+
+#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
+#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
+#define DLB2_CSR_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
+#define DLB2_FUNC_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
+/* Map to PMDs logging interface */
+#define DLB2_ERR(dev, fmt, args...) \
+	DLB2_LOG_ERR(fmt, ## args)
+
+#define DLB2_INFO(dev, fmt, args...) \
+	DLB2_LOG_INFO(fmt, ## args)
+
+#define DLB2_DEBUG(dev, fmt, args...) \
+	DLB2_LOG_DBG(fmt, ## args)
+
+/**
+ * os_udelay() - busy-wait for a number of microseconds
+ * @usecs: delay duration.
+ */
+static inline void os_udelay(int usecs)
+{
+	rte_delay_us(usecs);
+}
+
+/**
+ * os_msleep() - sleep for a number of milliseconds
+ * @usecs: delay duration.
+ */
+static inline void os_msleep(int msecs)
+{
+	rte_delay_ms(msecs);
+}
+
+#define DLB2_PP_BASE(__is_ldb) \
+	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
+
+/**
+ * os_map_producer_port() - map a producer port into the caller's address space
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ *
+ * This function maps the requested producer port memory into the caller's
+ * address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void *os_map_producer_port(struct dlb2_hw *hw,
+					 u8 port_id,
+					 bool is_ldb)
+{
+	uint64_t addr;
+	uint64_t pp_dma_base;
+
+	pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
+	addr = (pp_dma_base + (PAGE_SIZE * port_id));
+
+	return (void *)(uintptr_t)addr;
+}
+
+/**
+ * os_unmap_producer_port() - unmap a producer port
+ * @addr: mapped producer port address
+ *
+ * This function undoes os_map_producer_port() by unmapping the producer port
+ * memory from the caller's address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(addr);
+}
+
+/**
+ * os_fence_hcw() - fence an HCW to ensure it arrives at the device
+ * @hw: dlb2_hw handle for a particular device.
+ * @pp_addr: producer port address
+ */
+static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
+{
+	RTE_SET_USED(hw);
+
+	/* To ensure outstanding HCWs reach the device, read the PP address. IA
+	 * memory ordering prevents reads from passing older writes, and the
+	 * mfence also ensures this.
+	 */
+	rte_mb();
+
+	*(volatile u64 *)pp_addr;
+}
+
+/**
+ * DLB2_HW_ERR() - log an error message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_ERR(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_ERR(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/**
+ * DLB2_HW_DBG() - log an info message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_DBG(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_DEBUG(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/* The callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void *dlb2_complete_queue_map_unmap(void *__args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
+	int ret;
+
+	while (1) {
+		rte_spinlock_lock(&dlb2_dev->resource_mutex);
+
+		ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+		ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+		if (ret != 0) {
+			rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+			/* Relinquish the CPU so the application can process
+			 * its CQs, so this function doesn't deadlock.
+			 */
+			sched_yield();
+		} else {
+			break;
+		}
+	}
+
+	dlb2_dev->worker_launched = false;
+
+	rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+
+	return NULL;
+}
+
+
+/**
+ * os_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static inline void os_schedule_work(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+	pthread_t complete_queue_map_unmap_thread;
+	int ret;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
+				     "dlb_queue_unmap_waiter",
+				     NULL,
+				     dlb2_complete_queue_map_unmap,
+				     dlb2_dev);
+	if (ret)
+		DLB2_ERR(dlb2_dev,
+			 "Could not create queue complete map/unmap thread, err=%d\n",
+			 ret);
+	else
+		dlb2_dev->worker_launched = true;
+}
+
+/**
+ * os_worker_active() - query whether the map/unmap worker thread is active
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether a thread (launched by
+ * os_schedule_work()) is active. This function is used to determine
+ * whether or not to launch a worker thread.
+ */
+static inline bool os_worker_active(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	return dlb2_dev->worker_launched;
+}
+
+#endif /*  __DLB2_OSDEP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
new file mode 100644
index 0000000..423233b
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <rte_bitmap.h>
+#include <rte_string_fns.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include "../dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+	struct rte_bitmap *map;
+	unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+				    unsigned int len)
+{
+	struct dlb2_bitmap *bm;
+	void *mem;
+	uint32_t alloc_size;
+	uint32_t nbits = (uint32_t)len;
+
+	if (bitmap == NULL || nbits == 0)
+		return -EINVAL;
+
+	/* Allocate DLB2 bitmap control struct */
+	bm = rte_malloc("DLB2_PF",
+			sizeof(struct dlb2_bitmap),
+			RTE_CACHE_LINE_SIZE);
+
+	if (bm == NULL)
+		return -ENOMEM;
+
+	/* Allocate bitmap memory */
+	alloc_size = rte_bitmap_get_memory_footprint(nbits);
+	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
+	if (mem == NULL) {
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->map = rte_bitmap_init(len, mem, alloc_size);
+	if (bm->map == NULL) {
+		rte_free(mem);
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap == NULL)
+		return;
+
+	rte_free(bitmap->map);
+	rte_free(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++)
+		rte_bitmap_set(bitmap->map, i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	rte_bitmap_reset(bitmap->map);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set() - set a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
+				  unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_set(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+					unsigned int bit,
+					unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_set(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear() - clear a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
+				    unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_clear(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+					  unsigned int bit,
+					  unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_clear(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+						 unsigned int len)
+{
+	unsigned int i, j = 0;
+
+	if (bitmap  == NULL || bitmap->map == NULL || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			if (++j == len)
+				return i - j + 1;
+		} else {
+			j = 0;
+		}
+	}
+
+	/* No set bit range of length len? */
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			return i;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+	int weight = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			weight++;
+	}
+	return weight;
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				      bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+	int max_len = 0, len = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			len++;
+		} else {
+			if (len > max_len)
+				max_len = len;
+			len = 0;
+		}
+	}
+
+	if (len > max_len)
+		max_len = len;
+
+	return max_len;
+}
+
+/**
+ * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
+ * @dest: pointer to dlb2_bitmap structure, which will contain the results of
+ *	  the 'or' of src1 and src2.
+ * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
+ * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
+ *
+ * This function 'or's two bitmaps together and stores the result in a third
+ * bitmap. The source and destination bitmaps can be the same.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - One of the bitmaps is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
+				 struct dlb2_bitmap *src1,
+				 struct dlb2_bitmap *src2)
+{
+	unsigned int i, min;
+	int numset = 0;
+
+	if (dest  == NULL || dest->map == NULL ||
+	    src1  == NULL || src1->map == NULL ||
+	    src2  == NULL || src2->map == NULL)
+		return -EINVAL;
+
+	min = dest->len;
+	min = (min > src1->len) ? src1->len : min;
+	min = (min > src2->len) ? src2->len : min;
+
+	for (i = 0; i != min; i++) {
+		if  (rte_bitmap_get(src1->map, i) ||
+		     rte_bitmap_get(src2->map, i)) {
+			rte_bitmap_set(dest->map, i);
+			numset++;
+		} else {
+			rte_bitmap_clear(dest->map, i);
+		}
+	}
+
+	return numset;
+}
+
+#endif /*  __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
new file mode 100644
index 0000000..5531739
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_LIST_H
+#define __DLB2_OSDEP_LIST_H
+
+#include <rte_tailq.h>
+
+struct dlb2_list_entry {
+	TAILQ_ENTRY(dlb2_list_entry) node;
+};
+
+/* Dummy - just a struct definition */
+TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
+
+/* =================
+ * TAILQ Supplements
+ * =================
+ */
+
+#ifndef TAILQ_FOREACH_ENTRY
+#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)		\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter)						\
+		&& (ptr = container_of(iter, typeof(*(ptr)), name)); \
+	    (iter) = TAILQ_NEXT((iter), node))
+#endif
+
+#ifndef TAILQ_FOREACH_ENTRY_SAFE
+#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)	\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter) &&						\
+		(ptr = container_of(iter, typeof(*(ptr)), name)) &&\
+		((tvar) = TAILQ_NEXT((iter), node), 1);	\
+	    (iter) = (tvar))
+#endif
+
+/***********************/
+/*** List operations ***/
+/***********************/
+
+/**
+ * dlb2_list_init_head() - initialize the head of a list
+ * @head: list head
+ */
+static inline void dlb2_list_init_head(struct dlb2_list_head *head)
+{
+	TAILQ_INIT(head);
+}
+
+/**
+ * dlb2_list_add() - add an entry to a list
+ * @head: list head
+ * @entry: new list entry
+ */
+static inline void
+dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
+{
+	TAILQ_INSERT_TAIL(head, entry, node);
+}
+
+/**
+ * dlb2_list_del() - delete an entry from a list
+ * @entry: list entry
+ * @head: list head
+ */
+static inline void dlb2_list_del(struct dlb2_list_head *head,
+				 struct dlb2_list_entry *entry)
+{
+	TAILQ_REMOVE(head, entry, node);
+}
+
+/**
+ * dlb2_list_empty() - check if a list is empty
+ * @head: list head
+ *
+ * Return:
+ * Returns 1 if empty, 0 if not.
+ */
+static inline int dlb2_list_empty(struct dlb2_list_head *head)
+{
+	return TAILQ_EMPTY(head);
+}
+
+/**
+ * dlb2_list_splice() - splice a list
+ * @src_head: list to be added
+ * @ head: where src_head will be inserted
+ */
+static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
+				    struct dlb2_list_head *head)
+{
+	TAILQ_CONCAT(head, src_head, node);
+}
+
+/**
+ * DLB2_LIST_HEAD() - retrieve the head of the list
+ * @head: list head
+ * @type: type of the list variable
+ * @name: name of the list field within the containing struct
+ */
+#define DLB2_LIST_HEAD(head, type, name)                       \
+	(TAILQ_FIRST(&head) ?					\
+		container_of(TAILQ_FIRST(&head), type, name) :	\
+		NULL)
+
+/**
+ * DLB2_LIST_FOR_EACH() - iterate over a list
+ * @head: list head
+ * @ptr: pointer to struct containing a struct list
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ */
+#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
+	TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
+
+/**
+ * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
+ * an element is removed from the list while processing it.
+ * @ptr: pointer to struct containing a struct list
+ * @ptr_tmp: pointer to struct containing a struct list (temporary)
+ * @head: list head
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ * @iter_tmp: iterator variable (temporary)
+ */
+#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
+	TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
+
+#endif /*  __DLB2_OSDEP_LIST_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
new file mode 100644
index 0000000..0a48f7e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_TYPES_H
+#define __DLB2_OSDEP_TYPES_H
+
+#include <linux/types.h>
+
+#include <inttypes.h>
+#include <ctype.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Types for user mode PF PMD */
+typedef uint8_t         u8;
+typedef int8_t          s8;
+typedef uint16_t        u16;
+typedef int16_t         s16;
+typedef uint32_t        u32;
+typedef int32_t         s32;
+typedef uint64_t        u64;
+
+#define __iomem
+
+/* END types for user mode PF PMD */
+
+#endif /* __DLB2_OSDEP_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
new file mode 100644
index 0000000..43ecad4
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
@@ -0,0 +1,2527 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "dlb2_osdep_types.h"
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+	struct {
+		u32 isr_pend : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+	struct {
+		u32 vf0_reset_in_progress : 1;
+		u32 vf1_reset_in_progress : 1;
+		u32 vf2_reset_in_progress : 1;
+		u32 vf3_reset_in_progress : 1;
+		u32 vf4_reset_in_progress : 1;
+		u32 vf5_reset_in_progress : 1;
+		u32 vf6_reset_in_progress : 1;
+		u32 vf7_reset_in_progress : 1;
+		u32 vf8_reset_in_progress : 1;
+		u32 vf9_reset_in_progress : 1;
+		u32 vf10_reset_in_progress : 1;
+		u32 vf11_reset_in_progress : 1;
+		u32 vf12_reset_in_progress : 1;
+		u32 vf13_reset_in_progress : 1;
+		u32 vf14_reset_in_progress : 1;
+		u32 vf15_reset_in_progress : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+	struct {
+		u32 vec_mask : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+	struct {
+		u32 func_vf_bar_dis : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+	struct {
+		u32 total_vas : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+	struct {
+		u32 total_dir_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+	struct {
+		u32 total_ldb_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+	struct {
+		u32 total_dir_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+	struct {
+		u32 total_ldb_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+	struct {
+		u32 total_dir_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+	struct {
+		u32 total_ldb_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 meas : 1;
+		u32 debug : 7;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 cq_int_rearm : 1;
+		u32 dsi_error : 1;
+		u32 rsvd0 : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 rsvd0 : 3;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+	struct {
+		u32 qid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+	struct {
+		u32 vqid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+	struct {
+		u32 qid : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 debug : 8;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 isz : 1;
+		u32 dsi_error : 1;
+		u32 dlbrsvd : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 vf_synd0_parity : 1;
+		u32 vf_synd1_parity : 1;
+		u32 vf_synd2_parity : 1;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+	struct {
+		u32 sn_cfg_v : 1;
+		u32 fid_cfg_v : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS  0
+#define DLB2_CQ_ISR_MODE_MSI  1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI  3
+union dlb2_sys_ldb_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+	struct {
+		u32 keep_pf_ppid : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+	struct {
+		u32 illegal_hcw : 1;
+		u32 illegal_pp : 1;
+		u32 illegal_pasid : 1;
+		u32 illegal_qid : 1;
+		u32 disabled_qid : 1;
+		u32 illegal_ldb_qid_cfg : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+	struct {
+		u32 msix_0_ack : 1;
+		u32 msix_1_ack : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+	struct {
+		u32 msix_0_passthru : 1;
+		u32 msix_1_passthru : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED     0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+	struct {
+		u32 mode : 1;
+		u32 poll_mode : 1;
+		u32 poll_mask : 1;
+		u32 poll_lock : 1;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+	struct {
+		u32 cq : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 alarm : 1;
+		u32 cwd : 1;
+		u32 vf_pf_mb : 1;
+		u32 rsvd0 : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+	struct {
+		u32 qid_fid_limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+	struct {
+		u32 compress_code : 3;
+		u32 rsvd0 : 29;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+	(DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+	struct {
+		u32 count : 15;
+		u32 rsvd0 : 17;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+	struct {
+		u32 sn : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+	struct {
+		u32 mode : 3;
+		u32 slot : 4;
+		u32 rsvz0 : 1;
+		u32 grp : 1;
+		u32 rsvz1 : 1;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+	struct {
+		u32 en : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+	struct {
+		u32 write_pointer : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+	struct {
+		u32 base : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+	struct {
+		u32 limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+	struct {
+		u32 pop_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+	struct {
+		u32 push_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+	struct {
+		u32 depth : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+	struct {
+		u32 write_pointer : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 dlb_cor_alarm_enable : 1;
+		u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+		u32 cfg_64bytes_qe_dir_cq_mode : 1;
+		u32 pad_write_ldb : 1;
+		u32 pad_write_dir : 1;
+		u32 pad_first_write_ldb : 1;
+		u32 pad_first_write_dir : 1;
+		u32 rsvz0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+	struct {
+		u32 egress_credit_status_empty : 1;
+		u32 egress_credit_status_afull : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+		u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+		u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+		u32 chp_rop_pipe_credit_status_empty : 1;
+		u32 chp_rop_pipe_credit_status_afull : 1;
+		u32 qed_to_cq_pipe_credit_status_empty : 1;
+		u32 qed_to_cq_pipe_credit_status_afull : 1;
+		u32 egress_lsp_token_credit_status_empty : 1;
+		u32 egress_lsp_token_credit_status_afull : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 rsvz0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+	struct {
+		u32 sn_mode_0 : 3;
+		u32 rszv0 : 5;
+		u32 sn_mode_1 : 3;
+		u32 rszv1 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+	struct {
+		u32 unit_single_step_mode : 1;
+		u32 rr_en : 1;
+		u32 rszv0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+	struct {
+		u32 prio : 24;
+		u32 v : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+	struct {
+		u32 qid_p0 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p1 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p2 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p3 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+	struct {
+		u32 qid_p4 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p5 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p6 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p7 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+	struct {
+		u32 token_depth_select : 4;
+		u32 disable_wb_opt : 1;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+	struct {
+		u32 token_count : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+	struct {
+		u32 thresh : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+	(DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+	(DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+	struct {
+		u32 depth : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+	struct {
+		u32 cq : 8;
+		u32 qidix : 3;
+		u32 value : 1;
+		u32 nalb_haswork_v : 1;
+		u32 rlist_haswork_v : 1;
+		u32 slist_haswork_v : 1;
+		u32 inflight_ok_v : 1;
+		u32 aqed_nfull_v : 1;
+		u32 rsvz0 : 15;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+	struct {
+		u32 transfer : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+	struct {
+		u32 bw_range : 9;
+		u32 rsvz0 : 22;
+		u32 no_extra_credit : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+	struct {
+		u32 disab_atq_empty_arb : 1;
+		u32 inc_tok_unit_idle : 1;
+		u32 disab_rlist_pri : 1;
+		u32 inc_cmp_unit_idle : 1;
+		u32 rsvz0 : 2;
+		u32 dir_single_op : 1;
+		u32 dir_half_bw : 1;
+		u32 dir_single_out : 1;
+		u32 dir_disab_multi : 1;
+		u32 atq_single_op : 1;
+		u32 atq_half_bw : 1;
+		u32 atq_single_out : 1;
+		u32 atq_disab_multi : 1;
+		u32 dirrpl_single_op : 1;
+		u32 dirrpl_half_bw : 1;
+		u32 dirrpl_single_out : 1;
+		u32 lbrpl_single_op : 1;
+		u32 lbrpl_half_bw : 1;
+		u32 lbrpl_single_out : 1;
+		u32 ldb_single_op : 1;
+		u32 ldb_half_bw : 1;
+		u32 ldb_disab_multi : 1;
+		u32 atm_single_sch : 1;
+		u32 atm_single_cmp : 1;
+		u32 ldb_ce_tog_arb : 1;
+		u32 rsvz1 : 1;
+		u32 smon0_valid_sel : 2;
+		u32 smon0_value_sel : 1;
+		u32 smon0_compare_sel : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+	struct {
+		u32 chp_pf_reset_done : 1;
+		u32 rop_pf_reset_done : 1;
+		u32 lsp_pf_reset_done : 1;
+		u32 nalb_pf_reset_done : 1;
+		u32 ap_pf_reset_done : 1;
+		u32 dp_pf_reset_done : 1;
+		u32 qed_pf_reset_done : 1;
+		u32 dqed_pf_reset_done : 1;
+		u32 aqed_pf_reset_done : 1;
+		u32 sys_pf_reset_done : 1;
+		u32 pf_reset_active : 1;
+		u32 flrsm_state : 7;
+		u32 rsvd0 : 13;
+		u32 dlb_proc_reset_done : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+	struct {
+		u32 chp_pipeidle : 1;
+		u32 rop_pipeidle : 1;
+		u32 lsp_pipeidle : 1;
+		u32 nalb_pipeidle : 1;
+		u32 ap_pipeidle : 1;
+		u32 dp_pipeidle : 1;
+		u32 qed_pipeidle : 1;
+		u32 dqed_pipeidle : 1;
+		u32 aqed_pipeidle : 1;
+		u32 sys_pipeidle : 1;
+		u32 chp_unit_idle : 1;
+		u32 rop_unit_idle : 1;
+		u32 lsp_unit_idle : 1;
+		u32 nalb_unit_idle : 1;
+		u32 ap_unit_idle : 1;
+		u32 dp_unit_idle : 1;
+		u32 qed_unit_idle : 1;
+		u32 dqed_unit_idle : 1;
+		u32 aqed_unit_idle : 1;
+		u32 sys_unit_idle : 1;
+		u32 rsvd1 : 4;
+		u32 mstr_cfg_ring_idle : 1;
+		u32 mstr_cfg_mstr_idle : 1;
+		u32 mstr_flr_clkreq_b : 1;
+		u32 mstr_proc_idle : 1;
+		u32 mstr_proc_idle_masked : 1;
+		u32 rsvd0 : 2;
+		u32 dlb_func_idle : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+	struct {
+		u32 prochot : 1;
+		u32 pgcb_dlb_idle : 1;
+		u32 pgcb_dlb_pg_rdy_ack_b : 1;
+		u32 pmsm_pgcb_req_b : 1;
+		u32 pgbc_pmc_pg_req_b : 1;
+		u32 pmc_pgcb_pg_ack_b : 1;
+		u32 pmc_pgcb_fet_en_b : 1;
+		u32 pgcb_fet_en_b : 1;
+		u32 rsvz0 : 1;
+		u32 rsvz1 : 1;
+		u32 fuse_force_on : 1;
+		u32 fuse_proc_disable : 1;
+		u32 rsvz2 : 1;
+		u32 rsvz3 : 1;
+		u32 pm_fsm_d0tod3_ok : 1;
+		u32 pm_fsm_d3tod0_ok : 1;
+		u32 dlb_in_d3 : 1;
+		u32 rsvz4 : 7;
+		u32 pmsm : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+	struct {
+		u32 disable : 1;
+		u32 rsvz0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+	struct {
+		u32 isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+	struct {
+		u32 pf_isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+	struct {
+		u32 isr_pend : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+	struct {
+		u32 reset_in_progress : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+	struct {
+		u32 vf_msi_isr : 32;
+	} field;
+	u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
new file mode 100644
index 0000000..503fdf3
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
@@ -0,0 +1,1913 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_osdep_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_start_domain() - start a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: start domain arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int dlb2_hw_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp,
+			 bool vdev_request,
+			 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_request,
+		    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding unmap procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_finish_map_qid_procedures() - finish any pending map procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding map procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ *					interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+			  int port_id,
+			  bool is_ldb,
+			  bool vdev_request,
+			  unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+					 u32 *ldb_interrupts,
+					 u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+				 u32 *ldb_interrupts,
+				 u32 *dir_interrupts);
+
+/**
+ * dlb2_read_vf_intr_status() - read the VF interrupt status register
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function can be called from a VF's interrupt handler to determine
+ * which interrupts have fired. The first 31 bits correspond to CQ interrupt
+ * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
+ *
+ * Return:
+ * Returns a bit vector indicating which interrupt vectors are active.
+ */
+u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_intr_status() - ack VF interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function ACKs a VF's interrupts. Its interrupts argument should be the
+ * value returned by dlb2_read_vf_intr_status().
+ */
+void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function clears the VF's MSI interrupt pending register. Its interrupts
+ * argument should be contain the MSI vectors to ACK. For example, if MSI MME
+ * is in mode 0, then one bit 0 should ever be set.
+ */
+void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When done processing the PF mailbox request, this function unsets
+ * the PF's mailbox ISR register.
+ */
+void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
+
+/**
+ * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
+ *					vdevs
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the vdev->PF ISR fires, this function can be called to determine which
+ * vdev(s) are requesting service. This bitvector must be passed to
+ * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
+ * vdevs.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested service.
+ */
+u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ *
+ * When done processing all VF mailbox requests, this function unsets the VF's
+ * mailbox ISR register.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
+ *				    FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the VF FLR ISR fires, this function can be called to determine which
+ * VF(s) are requesting FLRs. This bitvector must passed to
+ * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
+ */
+u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF FLR requests, this function unsets the VF's FLR
+ * ISR register.
+ */
+void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF requests, this function communicates to the
+ * hardware that processing is complete.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
+			     u32 mbox_bitvec,
+			     u32 flr_bitvec);
+
+/**
+ * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the watchdog timer interrupt cause registers to
+ * determine which port(s) had a watchdog timeout, and notifies the
+ * application(s) that own the port(s).
+ */
+void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads and logs the alarm syndrome, then acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_request,
+			      unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of resources in use by the PF or a VF. It
+ * fills in the fields that args points to, except the following:
+ * - max_contiguous_atomic_inflights
+ * - max_contiguous_hist_list_entries
+ * - max_contiguous_ldb_credits
+ * - max_contiguous_dir_credits
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
+				   struct dlb2_get_num_resources_args *arg,
+				   bool vdev_request,
+				   unsigned int vdev_id);
+
+/**
+ * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
+ *				       the PF
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function sends a VF->PF mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the PF has processed
+ * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
+ * when the PF has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
+ *				 request
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * processing a VF->PF mailbox request. It should only be called after sending
+ * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vf_flr_complete() - check the status of a VF FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * executing the VF FLR. It should only be called after setting the VF's FLR
+ * bit.
+ */
+bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
+ *					vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function sends a PF->vdev mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the vdev has
+ * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
+ * determine when the vdev has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
+ *			       request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function returns a boolean indicating whether the vdev has finished
+ * processing a PF->vdev mailbox request. It should only be called after
+ * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
+			     unsigned int vf_id,
+			     void *data,
+			     int len);
+
+/**
+ * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
+			       unsigned int vf_id,
+			       void *data,
+			       int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_reset_vdev() - reset the hardware owned by a virtual device
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function resets the hardware owned by a vdev, by resetting the vdev's
+ * domains one by one.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function returns whether or not the vdev's resource assignments are
+ * locked. If locked, no resources can be added to or subtracted from the
+ * group.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_lock_vdev() - lock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sets a flag indicating that the vdev is using its resources.
+ * When the vdev is locked, its resource assignment cannot be changed.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_unlock_vdev() - unlock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function unlocks the vdev's resource assignment, allowing it to be
+ * modified.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of scheduling domains to assign to this vdev
+ *
+ * This function assigns num scheduling domains to the specified vdev. If the
+ * vdev already has domains assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB queues to assign to this vdev
+ *
+ * This function assigns num LDB queues to the specified vdev. If the vdev
+ * already has LDB queues assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports to the specified vdev. If the vdev
+ * already has LDB ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @cos: class-of-service ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports from class-of-service cos to the
+ * specified vdev. If the vdev already has LDB ports from this class-of-service
+ * assigned, this existing assignment is adjusted accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
+				   u32 id,
+				   u32 cos,
+				   u32 num);
+
+/**
+ * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR ports to assign to this vdev
+ *
+ * This function assigns num DIR ports to the specified vdev. If the vdev
+ * already has DIR ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB credit credits to assign to this vdev
+ *
+ * This function assigns num LDB credit to the specified vdev. If the vdev
+ * already has LDB credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR credits to assign to this vdev
+ *
+ * This function assigns num DIR credit to the specified vdev. If the vdev
+ * already has DIR credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of history list entries to assign to this vdev
+ *
+ * This function assigns num history list entries to the specified vdev. If the
+ * vdev already has history list entries assigned, this existing assignment is
+ * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
+ * this function may fail if a sufficiently large contiguous chunk is not
+ * available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of atomic inflights to assign to this vdev
+ *
+ * This function assigns num atomic inflights to the specified vdev. If the vdev
+ * already has atomic inflights assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of entries, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function takes any resources currently assigned to the vdev and
+ * reassigns them to the PF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_notify_vf() - send an alarm to a VF
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID
+ * @notification: notification
+ *
+ * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
+ *
+ * Return:
+ * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
+ */
+int dlb2_notify_vf(struct dlb2_hw *hw,
+		   unsigned int vf_id,
+		   u32 notification);
+
+/**
+ * dlb2_vdev_in_use() - query whether a virtual device is in use
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sends a mailbox request to the vdev to query whether the vdev
+ * is in use.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
+ * an internal error occurs.
+ */
+int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+enum dlb2_virt_mode {
+	DLB2_VIRT_NONE,
+	DLB2_VIRT_SRIOV,
+	DLB2_VIRT_SIOV,
+
+	/* NUM_DLB2_VIRT_MODES must be last */
+	NUM_DLB2_VIRT_MODES,
+};
+
+/**
+ * dlb2_hw_set_virt_mode() - set the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: either none, SR-IOV, or Scalable IOV.
+ *
+ * This function sets the virtualization mode of the device. This controls
+ * whether the device uses a software or hardware mailbox.
+ *
+ * This should be called by the PF driver when either SR-IOV or Scalable IOV is
+ * selected as the virtualization mechanism, and by the VF/VDEV driver during
+ * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
+ *
+ * Errors:
+ * EINVAL - Invalid mode.
+ */
+int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
+
+/**
+ * dlb2_hw_get_virt_mode() - get the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function gets the virtualization mode of the device.
+ */
+enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_register_sw_mbox() - register a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
+ * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
+ * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
+ * @inject_arg: user argument for pf2vdev_inject callback.
+ *
+ * When Scalable IOV is enabled, the VDCM must register a software mailbox for
+ * every virtual device during vdev creation.
+ *
+ * This function notifies the driver to use a software mailbox using the
+ * provided pointers, instead of the device's hardware mailbox. When the driver
+ * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
+ * go to the software mailbox instead of the hardware one. This is used in
+ * Scalable IOV virtualization.
+ */
+void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
+			      unsigned int vdev_id,
+			      u32 *vdev2pf_mbox,
+			      u32 *pf2vdev_mbox,
+			      void (*pf2vdev_inject)(void *),
+			      void *inject_arg);
+
+/**
+ * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function notifies the driver to stop using a previously registered
+ * software mailbox.
+ */
+void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ * @addr_lo: least-significant 32 bits of address.
+ * @data: 32 data bits.
+ *
+ * This sets up the CQ's IMS entry with the provided address and data values.
+ * This function should only be called if the device is configured for Scalable
+ * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
+ * not needed.
+ */
+void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb,
+				u32 addr_lo,
+				u32 data);
+
+/**
+ * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ *
+ * This clears the CQ's IMS entry, reverting it to its reset state.
+ */
+void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb);
+
+/**
+ * dlb2_hw_register_pasid() - register a vdev's PASID
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @pasid: the vdev's PASID.
+ *
+ * This function stores the user-supplied PASID, and uses it when configuring
+ * the vdev's CQs.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+int dlb2_hw_register_pasid(struct dlb2_hw *hw,
+			   unsigned int vdev_id,
+			   unsigned int pasid);
+
+/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vf_request,
+				unsigned int vf_id);
+
+/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ *	to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ *	port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ *	    cause the total bandwidth across all classes of service to exceed
+ *	    100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+enum dlb2_wd_tmo {
+	/* 40s watchdog timeout */
+	DLB2_WD_TMO_40S,
+	/* 10s watchdog timeout */
+	DLB2_WD_TMO_10S,
+	/* 1s watchdog timeout */
+	DLB2_WD_TMO_1S,
+
+	/* Must be last */
+	NUM_DLB2_WD_TMOS,
+};
+
+/**
+ * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
+ *	caller-specified timeout.
+ * @hw: dlb2_hw handle for a particular device.
+ * @tmo: watchdog timeout.
+ *
+ * This function should be called during device initialization and after reset.
+ * The watchdog timer interrupt must also be enabled per-CQ, using either
+ * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid timeout.
+ */
+int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
+
+/**
+ * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid directed port ID.
+ */
+int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid load-balanced port ID.
+ */
+int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *	ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * load-balanced CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_ldb_cq_interrupt().
+ */
+void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical directed port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * directed CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_dir_cq_interrupt().
+ */
+void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+#endif /* __DLB2_RESOURCE_H */
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
new file mode 100644
index 0000000..3c51cab
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -0,0 +1,621 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "base/dlb2_resource.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_regs.h"
+#include "dlb2_main.h"
+#include "../dlb2_user.h"
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+
+#define PF_ID_ZERO 0	/* PF ONLY! */
+#define NO_OWNER_VF 0	/* PF ONLY! */
+#define NOT_VF_REQ false /* PF ONLY! */
+
+#define DLB2_PCI_CFG_SPACE_SIZE 256
+#define DLB2_PCI_CAP_POINTER 0x34
+#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
+#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
+#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
+#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
+#define DLB2_PCI_EXT_CAP_ID_ERR 1
+#define DLB2_PCI_ERR_UNCOR_MASK 8
+#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
+
+#define DLB2_PCI_EXP_DEVCTL 8
+#define DLB2_PCI_LNKCTL 16
+#define DLB2_PCI_SLTCTL 24
+#define DLB2_PCI_RTCTL 28
+#define DLB2_PCI_EXP_DEVCTL2 40
+#define DLB2_PCI_LNKCTL2 48
+#define DLB2_PCI_SLTCTL2 56
+#define DLB2_PCI_CMD 4
+#define DLB2_PCI_X_CMD 2
+#define DLB2_PCI_EXP_DEVSTA 10
+#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
+#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
+
+#define DLB2_PCI_CAP_ID_EXP       0x10
+#define DLB2_PCI_CAP_ID_MSIX      0x11
+#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
+#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
+#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
+
+#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
+#define DLB2_PCI_PRI_ALLOC_REQ           0xC
+#define DLB2_PCI_PRI_CTRL                0x4
+#define DLB2_PCI_MSIX_FLAGS              0x2
+#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
+#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
+#define DLB2_PCI_ERR_ROOT_STATUS         0x30
+#define DLB2_PCI_ERR_COR_STATUS          0x10
+#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
+#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
+#define DLB2_PCI_ACS_CAP                 0x4
+#define DLB2_PCI_ACS_CTRL                0x6
+#define DLB2_PCI_ACS_SV                  0x1
+#define DLB2_PCI_ACS_RR                  0x4
+#define DLB2_PCI_ACS_CR                  0x8
+#define DLB2_PCI_ACS_UF                  0x10
+#define DLB2_PCI_ACS_EC                  0x20
+
+/* Stubs: Allow building partial probe patch */
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	int ret = 0;
+	RTE_SET_USED(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+/* End stubs */
+
+static int
+dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint32_t hdr;
+	size_t sz;
+	int pos;
+
+	pos = DLB2_PCI_CFG_SPACE_SIZE;
+	sz = sizeof(hdr);
+
+	while (pos > 0xFF) {
+		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
+			return -1;
+
+		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
+			return pos;
+
+		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint8_t pos;
+	int ret;
+	uint16_t hdr;
+
+	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
+	pos &= 0xFC;
+
+	if (ret != 1)
+		return -1;
+
+	while (pos > 0x3F) {
+		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
+		if (ret != 2)
+			return -1;
+
+		if (DLB2_PCI_CAP_ID(hdr) == id)
+			return pos;
+
+		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
+			return -1;
+
+		pos = DLB2_PCI_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+	rte_spinlock_init(&dlb2_dev->resource_mutex);
+	rte_spinlock_init(&dlb2_dev->measurement_lock);
+
+	return 0;
+}
+
+static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
+{
+	u32 retries = 0;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
+		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+		union dlb2_cfg_mstr_cfg_pm_status pm_st;
+		u32 addr;
+
+		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+			break;
+
+		rte_delay_ms(1);
+	};
+
+	if (retries == DLB2_READY_RETRY_LIMIT) {
+		printf("[%s()] wait for device ready timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+struct dlb2_dev *
+dlb2_probe(struct rte_pci_device *pdev)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret = 0;
+
+	DLB2_INFO(dlb2_dev, "probe\n");
+
+	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
+			      RTE_CACHE_LINE_SIZE);
+
+	if (dlb2_dev == NULL) {
+		ret = -ENOMEM;
+		goto dlb2_dev_malloc_fail;
+	}
+
+	/* PCI Bus driver has already mapped bar space into process.
+	 * Save off our IO register and FUNC addresses.
+	 */
+
+	/* BAR 0 */
+	if (pdev->mem_resource[0].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
+	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.func_kva,
+		  (void *)dlb2_dev->hw.func_phys_addr,
+		  (void *)(pdev->mem_resource[0].len));
+
+	/* BAR 2 */
+	if (pdev->mem_resource[2].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
+	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.csr_kva,
+		  (void *)dlb2_dev->hw.csr_phys_addr,
+		  (void *)(pdev->mem_resource[2].len));
+
+	dlb2_dev->pdev = pdev;
+
+	/* PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb2_pf_enable_pm(dlb2_dev);
+
+	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb2_pf_reset(dlb2_dev);
+	if (ret)
+		goto dlb2_reset_fail;
+
+	ret = dlb2_pf_init_driver_state(dlb2_dev);
+	if (ret)
+		goto init_driver_state_fail;
+
+	ret = dlb2_resource_init(&dlb2_dev->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	return dlb2_dev;
+
+resource_init_fail:
+	dlb2_resource_free(&dlb2_dev->hw);
+init_driver_state_fail:
+dlb2_reset_fail:
+pci_mmap_bad_addr:
+wait_for_device_ready_fail:
+	rte_free(dlb2_dev);
+dlb2_dev_malloc_fail:
+	rte_errno = ret;
+	return NULL;
+}
+
+int
+dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
+{
+	int ret = 0;
+	int i = 0;
+	uint32_t dword[16];
+	uint16_t cmd;
+	off_t off;
+
+	uint16_t dev_ctl_word;
+	uint16_t dev_ctl2_word;
+	uint16_t lnk_word;
+	uint16_t lnk_word2;
+	uint16_t slt_word;
+	uint16_t slt_word2;
+	uint16_t rt_ctl_word;
+	uint32_t pri_reqs_dword;
+	uint16_t pri_ctrl_word;
+
+	int pcie_cap_offset;
+	int pri_cap_offset;
+	int msix_cap_offset;
+	int err_cap_offset;
+	int acs_cap_offset;
+	int wait_count;
+
+	uint16_t devsta_busy_word;
+	uint16_t devctl_word;
+
+	struct rte_pci_device *pdev = dlb2_dev->pdev;
+
+	/* Save PCI config state */
+
+	for (i = 0; i < 16; i++) {
+		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
+			return ret;
+	}
+
+	pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
+
+	if (pcie_cap_offset < 0) {
+		printf("[%s()] failed to find the pcie capability\n",
+		       __func__);
+		return pcie_cap_offset;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
+		dev_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
+		lnk_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
+		slt_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_RTCTL;
+	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
+		rt_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
+		dev_ctl2_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
+		lnk_word2 = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
+		slt_word2 = 0;
+
+	off = DLB2_PCI_EXT_CAP_ID_PRI;
+	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (pri_cap_offset >= 0) {
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
+			pri_reqs_dword = 0;
+	}
+
+	/* clear the PCI command register before issuing the FLR */
+
+	off = DLB2_PCI_CMD;
+	cmd = 0;
+	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+		printf("[%s()] failed to write the pci command\n",
+		       __func__);
+		return ret;
+	}
+
+	/* issue the FLR */
+	for (wait_count = 0; wait_count < 4; wait_count++) {
+		int sleep_time;
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
+		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to read the pci device status\n",
+			       __func__);
+			return ret;
+		}
+
+		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
+			break;
+
+		sleep_time = (1 << (wait_count)) * 100;
+		rte_delay_ms(sleep_time);
+	}
+
+	if (wait_count == 4) {
+		printf("[%s()] wait for pci pending transactions timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to read the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
+
+	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to write the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	rte_delay_ms(100);
+
+	/* Restore PCI config state */
+
+	if (pcie_cap_offset >= 0) {
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie device control at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_RTCTL;
+		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	if (pri_cap_offset >= 0) {
+		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
+
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
+		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ERR;
+	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (err_cap_offset >= 0) {
+		uint32_t tmp;
+
+		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	for (i = 16; i > 0; i--) {
+		off = (i - 1) * 4;
+		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_CMD;
+	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
+		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+			printf("[%s()] failed to write the pci command\n",
+			       __func__);
+			return ret;
+		}
+	}
+
+	msix_cap_offset = dlb2_pci_find_capability(pdev,
+						   DLB2_PCI_CAP_ID_MSIX);
+	if (msix_cap_offset >= 0) {
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
+			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ACS;
+	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (acs_cap_offset >= 0) {
+		uint16_t acs_cap, acs_ctrl, acs_mask;
+		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
+		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
+			acs_cap = 0;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
+		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
+		acs_ctrl |= (acs_cap & acs_mask);
+
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
+		acs_mask |= DLB2_PCI_ACS_EC;
+		acs_ctrl &= ~acs_mask;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
new file mode 100644
index 0000000..f082a94
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -0,0 +1,98 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+#endif
+
+#include "base/dlb2_hw_types.h"
+#include "../dlb2_user.h"
+
+#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
+
+struct dlb2_dev;
+
+struct dlb2_port_memory {
+	struct dlb2_list_head list;
+	void *cq_base;
+	bool valid;
+};
+
+struct dlb2_dev {
+	struct rte_pci_device *pdev;
+	struct dlb2_hw hw;
+	/* struct list_head list; */
+	struct device *dlb2_device;
+	bool domain_reset_failed;
+	/* The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	rte_spinlock_t resource_mutex;
+	rte_spinlock_t measurement_lock;
+	bool worker_launched;
+	u8 revision;
+};
+
+struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
+int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
+int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+
+#endif /* __DLB2_MAIN_H */
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
new file mode 100644
index 0000000..ae4d785
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -0,0 +1,196 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_devargs.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_pci.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+#include "dlb2_main.h"
+#include "base/dlb2_hw_types.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_resource.h"
+
+static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
+
+/* Stubs: Allow building partial probe patch */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(arg);
+	RTE_SET_USED(vdev_req);
+	RTE_SET_USED(vdev_id);
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+/* End stubs */
+
+static void
+dlb2_pf_iface_fn_ptrs_init(void)
+{
+/* flexible iface fn ptr assignments will go here */
+}
+
+/* PCI DEV HOOKS */
+static int
+dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
+{
+	int ret = 0;
+	struct rte_pci_device *pci_dev;
+	struct dlb2_devargs dlb2_args = {
+		.socket_id = rte_socket_id(),
+		.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+		.num_dir_credits_override = -1,
+		.qid_depth_thresholds = { {0} },
+		.cos_id = DLB2_COS_DEFAULT
+	};
+	struct dlb2_eventdev *dlb2;
+
+	DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
+		     eventdev->data->dev_id, eventdev->data->socket_id);
+
+	dlb2_pf_iface_fn_ptrs_init();
+
+	pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
+
+		/* Probe the DLB2 PF layer */
+		dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
+
+		if (dlb2->qm_instance.pf_dev == NULL) {
+			DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
+				     rte_errno);
+			ret = -rte_errno;
+			goto dlb2_probe_failed;
+		}
+
+		/* Were we invoked with runtime parameters? */
+		if (pci_dev->device.devargs) {
+			ret = dlb2_parse_params(pci_dev->device.devargs->args,
+						pci_dev->device.devargs->name,
+						&dlb2_args);
+			if (ret) {
+				DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
+					     ret, rte_errno);
+				goto dlb2_probe_failed;
+			}
+		}
+
+		ret = dlb2_primary_eventdev_probe(eventdev,
+						  event_dlb2_pf_name,
+						  &dlb2_args);
+	} else {
+		ret = dlb2_secondary_eventdev_probe(eventdev,
+						    event_dlb2_pf_name);
+	}
+	if (ret)
+		goto dlb2_probe_failed;
+
+	DLB2_LOG_INFO("DLB2 PF Probe success\n");
+
+	return 0;
+
+dlb2_probe_failed:
+
+	DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
+
+	return ret;
+}
+
+#define EVENTDEV_INTEL_VENDOR_ID 0x8086
+
+static const struct rte_pci_id pci_id_dlb2_map[] = {
+	{
+		RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
+			       PCI_DEVICE_ID_INTEL_DLB2_PF)
+	},
+	{
+		.vendor_id = 0,
+	},
+};
+
+static int
+event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
+		     struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
+					     sizeof(struct dlb2_eventdev),
+					     dlb2_eventdev_pci_init,
+					     event_dlb2_pf_name);
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+}
+
+static int
+event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_remove(pci_dev, NULL);
+
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+
+}
+
+static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
+	.id_table = pci_id_dlb2_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe = event_dlb2_pci_probe,
+	.remove = event_dlb2_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
-- 
2.6.4

[dpdk-dev] [PATCH v7 07/23] event/dlb2: add flexible interface

[Timothy McDaniel]

This commit introduces the flexible interface. This
interface allows the core code to operate in PF mode (direct
hardware access) or bifurcated mode (hardware configured via
kernel driver). This driver currently only supports PF mode
but bifurcated mode will be added in a future DPDK patch-set.
Note that the flexible interface is not used for data path
operations, and thus there are no performance concerns
related to the use of function pointers.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_iface.c | 28 ++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.h | 29 +++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build  |  1 +
 3 files changed, 58 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h

diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
new file mode 100644
index 0000000..0fc9991
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+
+#include "dlb2_priv.h"
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+void (*dlb2_iface_low_level_io_init)(void);
+
+int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+				     uint8_t *revision);
+
+void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+				   enum dlb2_cq_poll_modes *mode);
+
+int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
new file mode 100644
index 0000000..4fb416e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_IFACE_H_
+#define _DLB2_IFACE_H_
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+extern void (*dlb2_iface_low_level_io_init)(void);
+
+extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+					    uint8_t *revision);
+
+extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+					  enum dlb2_cq_poll_modes *mode);
+
+extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
+
+#endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 1eab9b9..491e76d 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -8,6 +8,7 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
 endif
 
 sources = files('dlb2.c',
+		'dlb2_iface.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c'
 )
-- 
2.6.4

[dpdk-dev] [PATCH v7 08/23] event/dlb2: add probe-time hardware init

[Timothy McDaniel]

This commit adds probe-time low level hardware
initialization.  It also adds probe-time init for both
primary and secondary DPDK processes.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 188 +++++++++++++++++++-
 drivers/event/dlb2/meson.build             |   3 +-
 drivers/event/dlb2/pf/base/dlb2_resource.c | 274 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  23 +--
 drivers/event/dlb2/pf/dlb2_main.h          |   1 -
 drivers/event/dlb2/pf/dlb2_pf.c            |  78 ++++++--
 6 files changed, 523 insertions(+), 44 deletions(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 1b11979..16653db 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -31,6 +31,7 @@
 #include <rte_string_fns.h>
 
 #include "dlb2_priv.h"
+#include "dlb2_iface.h"
 #include "dlb2_inline_fns.h"
 
 /*
@@ -40,10 +41,108 @@
 #if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
 #error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
 #endif
+static struct rte_event_dev_info evdev_dlb2_default_info = {
+	.driver_name = "", /* probe will set */
+	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
+	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
+	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
+#else
+	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
+#endif
+	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
+	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
+	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
+	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
+	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
+	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+	.max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
+	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
+			  RTE_EVENT_DEV_CAP_EVENT_QOS |
+			  RTE_EVENT_DEV_CAP_BURST_MODE |
+			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
+			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
+			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
+};
 
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/* override defaults with value(s) provided on command line */
+static void
+dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
+				 int *qid_depth_thresholds)
+{
+	int q;
+
+	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
+		if (qid_depth_thresholds[q] != 0)
+			dlb2->ev_queues[q].depth_threshold =
+				qid_depth_thresholds[q];
+	}
+}
+
+static int
+dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
+	int ret;
+
+	/* Query driver resources provisioned for this device */
+
+	ret = dlb2_iface_get_num_resources(handle,
+					   &dlb2->hw_rsrc_query_results);
+	if (ret) {
+		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
+		return ret;
+	}
+
+	/* Complete filling in device resource info returned to evdev app,
+	 * overriding any default values.
+	 * The capabilities (CAPs) were set at compile time.
+	 */
+
+	evdev_dlb2_default_info.max_event_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	evdev_dlb2_default_info.max_event_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	evdev_dlb2_default_info.max_num_events =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	/* Save off values used when creating the scheduling domain. */
+
+	handle->info.num_sched_domains =
+		dlb2->hw_rsrc_query_results.num_sched_domains;
+
+	handle->info.hw_rsrc_max.nb_events_limit =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	handle->info.hw_rsrc_max.num_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues +
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.num_ldb_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	handle->info.hw_rsrc_max.num_ldb_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	handle->info.hw_rsrc_max.num_dir_ports =
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.reorder_window_size =
+		dlb2->hw_rsrc_query_results.num_hist_list_entries;
+
+	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
+
+	return 0;
+}
+
 #define DLB2_BASE_10 10
 
 static int
@@ -235,14 +334,71 @@ set_qid_depth_thresh(const char *key __rte_unused,
 	return 0;
 }
 
+static void
+dlb2_entry_points_init(struct rte_eventdev *dev)
+{
+	RTE_SET_USED(dev);
+
+	/* Eventdev PMD entry points */
+}
+
 int
 dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    const char *name,
 			    struct dlb2_devargs *dlb2_args)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
-	RTE_SET_USED(dlb2_args);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	dlb2->event_dev = dev; /* backlink */
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	dlb2->max_num_events_override = dlb2_args->max_num_events;
+	dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
+	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
+					    &dlb2->revision);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_hardware_init(&dlb2->qm_instance);
+
+	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
+			     err);
+		return err;
+	}
+
+	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	dlb2_init_queue_depth_thresholds(dlb2,
+					 dlb2_args->qid_depth_thresholds.val);
 
 	return 0;
 }
@@ -251,8 +407,30 @@ int
 dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
 			      const char *name)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 491e76d..28fef3f 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -10,7 +10,8 @@ endif
 sources = files('dlb2.c',
 		'dlb2_iface.c',
 		'pf/dlb2_main.c',
-		'pf/dlb2_pf.c'
+		'pf/dlb2_pf.c',
+		'pf/base/dlb2_resource.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
new file mode 100644
index 0000000..6de8b95
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_mbox.h"
+#include "dlb2_osdep.h"
+#include "dlb2_osdep_bitmap.h"
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+	int i;
+
+	dlb2_list_init_head(&domain->used_ldb_queues);
+	dlb2_list_init_head(&domain->used_dir_pq_pairs);
+	dlb2_list_init_head(&domain->avail_ldb_queues);
+	dlb2_list_init_head(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+	int i;
+
+	dlb2_list_init_head(&rsrc->avail_domains);
+	dlb2_list_init_head(&rsrc->used_domains);
+	dlb2_list_init_head(&rsrc->avail_ldb_queues);
+	dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
+	arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
+	arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
+	arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb2_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	struct dlb2_list_entry *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. This is application
+	 * dependent, but the driver interleaves port IDs as much as possible
+	 * to reduce the likelihood of this. This initial allocation maximizes
+	 * the average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	/* Zero-out resource tracking data structures */
+	memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+	memset(&hw->pf, 0, sizeof(hw->pf));
+
+	dlb2_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+		dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+		dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_domains, list);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_ldb_queues, list);
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+		struct dlb2_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
+			      &port->func_list);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+				DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+	if (ret)
+		goto unwind;
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+					DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+		if (ret)
+			goto unwind;
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb2_resource_free(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+	r0.field.disable = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 3c51cab..210b3186 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -19,6 +19,7 @@
 #include "dlb2_main.h"
 #include "../dlb2_user.h"
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 
 #define PF_ID_ZERO 0	/* PF ONLY! */
@@ -72,27 +73,6 @@
 #define DLB2_PCI_ACS_UF                  0x10
 #define DLB2_PCI_ACS_EC                  0x20
 
-/* Stubs: Allow building partial probe patch */
-void dlb2_resource_free(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-int dlb2_resource_init(struct dlb2_hw *hw)
-{
-	int ret = 0;
-	RTE_SET_USED(hw);
-
-	return ret;
-}
-
-void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-/* End stubs */
-
 static int
 dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
 {
@@ -149,7 +129,6 @@ static int
 dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
 {
 	rte_spinlock_init(&dlb2_dev->resource_mutex);
-	rte_spinlock_init(&dlb2_dev->measurement_lock);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index f082a94..f3bee71 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -38,7 +38,6 @@ struct dlb2_dev {
 	 * hardware registers.
 	 */
 	rte_spinlock_t resource_mutex;
-	rte_spinlock_t measurement_lock;
 	bool worker_launched;
 	u8 revision;
 };
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index ae4d785..7d64309 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -32,6 +32,7 @@
 #include <rte_string_fns.h>
 
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 #include "dlb2_main.h"
 #include "base/dlb2_hw_types.h"
@@ -40,35 +41,82 @@
 
 static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
 
-/* Stubs: Allow building partial probe patch */
-int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
-			      struct dlb2_get_num_resources_args *arg,
-			      bool vdev_req,
-			      unsigned int vdev_id)
+static void
+dlb2_pf_low_level_io_init(void)
+{
+	int i;
+	/* Addresses will be initialized at port create */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		/* First directed ports */
+		dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
+
+		/* Now load balanced ports */
+		dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
+	}
+}
+
+static int
+dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
+{
+	RTE_SET_USED(handle);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+static int
+dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
+			   uint8_t *revision)
 {
-	RTE_SET_USED(hw);
-	RTE_SET_USED(arg);
-	RTE_SET_USED(vdev_req);
-	RTE_SET_USED(vdev_id);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	*revision = dlb2_dev->revision;
 
 	return 0;
 }
 
-void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+static void
+dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+	dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_num_resources_args *rsrcs)
 {
-	RTE_SET_USED(hw);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
 }
 
-void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+static int
+dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
+			 enum dlb2_cq_poll_modes *mode)
 {
-	RTE_SET_USED(hw);
+	RTE_SET_USED(handle);
+
+	*mode = DLB2_CQ_POLL_MODE_SPARSE;
+
+	return 0;
 }
-/* End stubs */
 
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
-/* flexible iface fn ptr assignments will go here */
+	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
+	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
+	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
+	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
+	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v7 09/23] event/dlb2: add xstats

[Timothy McDaniel]

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
Reviewed-by: Chen, Mike Ximing <mike.ximing.chen@intel.com>
---
 drivers/event/dlb2/dlb2.c        |   35 +-
 drivers/event/dlb2/dlb2_xstats.c | 1240 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build   |    1 +
 3 files changed, 1273 insertions(+), 3 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 16653db..925824e 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,6 +70,21 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/*
+ * DUMMY - added so that xstats path will compile/link.
+ * Will be replaced by real version in a subsequent
+ * patch.
+ */
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	RTE_SET_USED(dlb2);
+	RTE_SET_USED(queue);
+
+	return 0;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -337,9 +352,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
-	RTE_SET_USED(dev);
-
-	/* Eventdev PMD entry points */
+	/* Expose PMD's eventdev interface */
+	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dump             = dlb2_eventdev_dump,
+		.xstats_get       = dlb2_eventdev_xstats_get,
+		.xstats_get_names = dlb2_eventdev_xstats_get_names,
+		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
+		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+	};
+
+	dev->dev_ops = &dlb2_eventdev_entry_ops;
 }
 
 int
@@ -391,6 +413,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Complete xtstats runtime initialization */
+	err = dlb2_xstats_init(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
+		return err;
+	}
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
new file mode 100644
index 0000000..21391f7
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_xstats.c
@@ -0,0 +1,1240 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eventdev.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+enum dlb2_xstats_type {
+	/* common to device and port */
+	rx_ok,				/**< Receive an event */
+	rx_drop,                        /**< Error bit set in received QE */
+	rx_interrupt_wait,		/**< Wait on an interrupt */
+	rx_umonitor_umwait,		/**< Block using umwait */
+	tx_ok,				/**< Transmit an event */
+	total_polls,			/**< Call dequeue_burst */
+	zero_polls,			/**< Call dequeue burst and return 0 */
+	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
+	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
+	tx_nospc_inflight_max,		/**< Reach the new_event_threshold */
+	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
+	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
+	/* device specific */
+	nb_events_limit,
+	inflight_events,
+	ldb_pool_size,
+	dir_pool_size,
+	/* port specific */
+	tx_new,				/**< Send an OP_NEW event */
+	tx_fwd,				/**< Send an OP_FORWARD event */
+	tx_rel,				/**< Send an OP_RELEASE event */
+	tx_implicit_rel,		/**< Issue an implicit event release */
+	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED event */
+	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL event */
+	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event */
+	tx_sched_directed,		/**< Send a directed event */
+	tx_invalid,                     /**< Send an event with an invalid op */
+	outstanding_releases,		/**< # of releases a port owes */
+	max_outstanding_releases,	/**< max # of releases a port can owe */
+	rx_sched_ordered,		/**< Dequeue an ordered event */
+	rx_sched_unordered,		/**< Dequeue an unordered event */
+	rx_sched_atomic,		/**< Dequeue an atomic event */
+	rx_sched_directed,		/**< Dequeue an directed event */
+	rx_sched_invalid,               /**< Dequeue event sched type invalid */
+	/* common to port and queue */
+	is_configured,			/**< Port is configured */
+	is_load_balanced,		/**< Port is LDB */
+	hw_id,				/**< Hardware ID */
+	/* queue specific */
+	num_links,			/**< Number of ports linked */
+	sched_type,			/**< Queue sched type */
+	enq_ok,				/**< # events enqueued to the queue */
+	current_depth,			/**< Current queue depth */
+	depth_threshold,		/**< Programmed depth threshold */
+	depth_le50_threshold,
+	/**< Depth LE to 50% of the configured hardware threshold */
+	depth_gt50_le75_threshold,
+	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold */
+	depth_gt75_le100_threshold,
+	/**< Depth GT 75%. but LE to the configured hardware threshold */
+	depth_gt100_threshold
+	/**< Depth GT 100% of the configured hw threshold */
+};
+
+typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
+		uint16_t obj_idx, /* port or queue id */
+		enum dlb2_xstats_type stat, int extra_arg);
+
+enum dlb2_xstats_fn_type {
+	DLB2_XSTATS_FN_DEV,
+	DLB2_XSTATS_FN_PORT,
+	DLB2_XSTATS_FN_QUEUE
+};
+
+struct dlb2_xstats_entry {
+	struct rte_event_dev_xstats_name name;
+	uint64_t reset_value; /* an offset to be taken away to emulate resets */
+	enum dlb2_xstats_fn_type fn_id;
+	enum dlb2_xstats_type stat;
+	enum rte_event_dev_xstats_mode mode;
+	int extra_arg;
+	uint16_t obj_idx;
+	uint8_t reset_allowed; /* when set, this value can be reset */
+};
+
+/* Some device stats are simply a summation of the corresponding port values */
+static uint64_t
+dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
+			     int which_stat)
+{
+	int i;
+	uint64_t val = 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
+
+		if (!port->setup_done)
+			continue;
+
+		switch (which_stat) {
+		case rx_ok:
+			val += port->stats.traffic.rx_ok;
+			break;
+		case rx_drop:
+			val += port->stats.traffic.rx_drop;
+			break;
+		case rx_interrupt_wait:
+			val += port->stats.traffic.rx_interrupt_wait;
+			break;
+		case rx_umonitor_umwait:
+			val += port->stats.traffic.rx_umonitor_umwait;
+			break;
+		case tx_ok:
+			val += port->stats.traffic.tx_ok;
+			break;
+		case total_polls:
+			val += port->stats.traffic.total_polls;
+			break;
+		case zero_polls:
+			val += port->stats.traffic.zero_polls;
+			break;
+		case tx_nospc_ldb_hw_credits:
+			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
+			break;
+		case tx_nospc_dir_hw_credits:
+			val += port->stats.traffic.tx_nospc_dir_hw_credits;
+			break;
+		case tx_nospc_inflight_max:
+			val += port->stats.traffic.tx_nospc_inflight_max;
+			break;
+		case tx_nospc_new_event_limit:
+			val += port->stats.traffic.tx_nospc_new_event_limit;
+			break;
+		case tx_nospc_inflight_credits:
+			val += port->stats.traffic.tx_nospc_inflight_credits;
+			break;
+		default:
+			return -1;
+		}
+	}
+	return val;
+}
+
+static uint64_t
+get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
+	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	switch (type) {
+	case rx_ok:
+	case rx_drop:
+	case rx_interrupt_wait:
+	case rx_umonitor_umwait:
+	case tx_ok:
+	case total_polls:
+	case zero_polls:
+	case tx_nospc_ldb_hw_credits:
+	case tx_nospc_dir_hw_credits:
+	case tx_nospc_inflight_max:
+	case tx_nospc_new_event_limit:
+	case tx_nospc_inflight_credits:
+		return dlb2_device_traffic_stat_get(dlb2, type);
+	case nb_events_limit:
+		return dlb2->new_event_limit;
+	case inflight_events:
+		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
+	case ldb_pool_size:
+		return dlb2->num_ldb_credits;
+	case dir_pool_size:
+		return dlb2->num_dir_credits;
+	default: return -1;
+	}
+}
+
+static uint64_t
+get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
+
+	switch (type) {
+	case rx_ok: return ev_port->stats.traffic.rx_ok;
+
+	case rx_drop: return ev_port->stats.traffic.rx_drop;
+
+	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
+
+	case rx_umonitor_umwait:
+		return ev_port->stats.traffic.rx_umonitor_umwait;
+
+	case tx_ok: return ev_port->stats.traffic.tx_ok;
+
+	case total_polls: return ev_port->stats.traffic.total_polls;
+
+	case zero_polls: return ev_port->stats.traffic.zero_polls;
+
+	case tx_nospc_ldb_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
+
+	case tx_nospc_dir_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
+
+	case tx_nospc_inflight_max:
+		return ev_port->stats.traffic.tx_nospc_inflight_max;
+
+	case tx_nospc_new_event_limit:
+		return ev_port->stats.traffic.tx_nospc_new_event_limit;
+
+	case tx_nospc_inflight_credits:
+		return ev_port->stats.traffic.tx_nospc_inflight_credits;
+
+	case is_configured: return ev_port->setup_done;
+
+	case is_load_balanced: return !ev_port->qm_port.is_directed;
+
+	case hw_id: return ev_port->qm_port.id;
+
+	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
+
+	case tx_fwd: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
+
+	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
+
+	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
+
+	case tx_sched_ordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case tx_sched_unordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case tx_sched_atomic:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case tx_sched_directed:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case tx_invalid: return ev_port->stats.tx_invalid;
+
+	case outstanding_releases: return ev_port->outstanding_releases;
+
+	case max_outstanding_releases:
+		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	case rx_sched_ordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case rx_sched_unordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case rx_sched_atomic:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case rx_sched_directed:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
+
+	default: return -1;
+	}
+}
+
+static uint64_t
+dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
+{
+	int port = 0;
+	uint64_t tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
+
+	return tally;
+}
+
+static uint64_t
+dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
+{
+	int port = 0;
+	uint64_t enq_ok_tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
+
+	return enq_ok_tally;
+}
+
+static uint64_t
+get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_queue *ev_queue =
+		&dlb2->ev_queues[obj_idx];
+
+	switch (type) {
+	case is_configured: return ev_queue->setup_done;
+
+	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
+
+	case hw_id: return ev_queue->qm_queue.id;
+
+	case num_links: return ev_queue->num_links;
+
+	case sched_type: return ev_queue->qm_queue.sched_type;
+
+	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
+
+	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
+
+	case depth_threshold: return ev_queue->depth_threshold;
+
+	case depth_le50_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_LE50);
+
+	case depth_gt50_le75_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT50_LE75);
+
+	case depth_gt75_le100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT75_LE100);
+
+	case depth_gt100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT100);
+
+	default: return -1;
+	}
+}
+
+int
+dlb2_xstats_init(struct dlb2_eventdev *dlb2)
+{
+	/*
+	 * define the stats names and types. Used to build up the device
+	 * xstats array
+	 * There are multiple set of stats:
+	 *   - device-level,
+	 *   - per-port,
+	 *   - per-qid,
+	 *
+	 * For each of these sets, we have three parallel arrays, one for the
+	 * names, the other for the stat type parameter to be passed in the fn
+	 * call to get that stat. The third array allows resetting or not.
+	 * All these arrays must be kept in sync
+	 */
+	static const char * const dev_stats[] = {
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"nb_events_limit",
+		"inflight_events",
+		"ldb_pool_size",
+		"dir_pool_size",
+	};
+	static const enum dlb2_xstats_type dev_types[] = {
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		nb_events_limit,
+		inflight_events,
+		ldb_pool_size,
+		dir_pool_size,
+	};
+	/* Note: generated device stats are not allowed to be reset. */
+	static const uint8_t dev_reset_allowed[] = {
+		0, /* rx_ok */
+		0, /* rx_drop */
+		0, /* rx_interrupt_wait */
+		0, /* rx_umonitor_umwait */
+		0, /* tx_ok */
+		0, /* total_polls */
+		0, /* zero_polls */
+		0, /* tx_nospc_ldb_hw_credits */
+		0, /* tx_nospc_dir_hw_credits */
+		0, /* tx_nospc_inflight_max */
+		0, /* tx_nospc_new_event_limit */
+		0, /* tx_nospc_inflight_credits */
+		0, /* nb_events_limit */
+		0, /* inflight_events */
+		0, /* ldb_pool_size */
+		0, /* dir_pool_size */
+	};
+	static const char * const port_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"tx_new",
+		"tx_fwd",
+		"tx_rel",
+		"tx_implicit_rel",
+		"tx_sched_ordered",
+		"tx_sched_unordered",
+		"tx_sched_atomic",
+		"tx_sched_directed",
+		"tx_invalid",
+		"outstanding_releases",
+		"max_outstanding_releases",
+		"rx_sched_ordered",
+		"rx_sched_unordered",
+		"rx_sched_atomic",
+		"rx_sched_directed",
+		"rx_sched_invalid"
+	};
+	static const enum dlb2_xstats_type port_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		tx_new,
+		tx_fwd,
+		tx_rel,
+		tx_implicit_rel,
+		tx_sched_ordered,
+		tx_sched_unordered,
+		tx_sched_atomic,
+		tx_sched_directed,
+		tx_invalid,
+		outstanding_releases,
+		max_outstanding_releases,
+		rx_sched_ordered,
+		rx_sched_unordered,
+		rx_sched_atomic,
+		rx_sched_directed,
+		rx_sched_invalid
+	};
+	static const uint8_t port_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		1, /* rx_ok */
+		1, /* rx_drop */
+		1, /* rx_interrupt_wait */
+		1, /* rx_umonitor_umwait */
+		1, /* tx_ok */
+		1, /* total_polls */
+		1, /* zero_polls */
+		1, /* tx_nospc_ldb_hw_credits */
+		1, /* tx_nospc_dir_hw_credits */
+		1, /* tx_nospc_inflight_max */
+		1, /* tx_nospc_new_event_limit */
+		1, /* tx_nospc_inflight_credits */
+		1, /* tx_new */
+		1, /* tx_fwd */
+		1, /* tx_rel */
+		1, /* tx_implicit_rel */
+		1, /* tx_sched_ordered */
+		1, /* tx_sched_unordered */
+		1, /* tx_sched_atomic */
+		1, /* tx_sched_directed */
+		1, /* tx_invalid */
+		0, /* outstanding_releases */
+		0, /* max_outstanding_releases */
+		1, /* rx_sched_ordered */
+		1, /* rx_sched_unordered */
+		1, /* rx_sched_atomic */
+		1, /* rx_sched_directed */
+		1  /* rx_sched_invalid */
+	};
+
+	/* QID specific stats */
+	static const char * const qid_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"num_links",
+		"sched_type",
+		"enq_ok",
+		"current_depth",
+		"depth_threshold",
+		"depth_le50_threshold",
+		"depth_gt50_le75_threshold",
+		"depth_gt75_le100_threshold",
+		"depth_gt100_threshold",
+	};
+	static const enum dlb2_xstats_type qid_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		num_links,
+		sched_type,
+		enq_ok,
+		current_depth,
+		depth_threshold,
+		depth_le50_threshold,
+		depth_gt50_le75_threshold,
+		depth_gt75_le100_threshold,
+		depth_gt100_threshold,
+	};
+	static const uint8_t qid_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		0, /* num_links */
+		0, /* sched_type */
+		1, /* enq_ok */
+		0, /* current_depth */
+		0, /* depth_threshold */
+		1, /* depth_le50_threshold */
+		1, /* depth_gt50_le75_threshold */
+		1, /* depth_gt75_le100_threshold */
+		1, /* depth_gt100_threshold */
+	};
+
+	/* ---- end of stat definitions ---- */
+
+	/* check sizes, since a missed comma can lead to strings being
+	 * joined by the compiler.
+	 */
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
+
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_reset_allowed));
+
+	/* other vars */
+	const unsigned int count = RTE_DIM(dev_stats) +
+			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
+			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
+	unsigned int i, port, qid, stat_id = 0;
+
+	dlb2->xstats = rte_zmalloc_socket(NULL,
+			sizeof(dlb2->xstats[0]) * count, 0,
+			dlb2->qm_instance.info.socket_id);
+	if (dlb2->xstats == NULL)
+		return -ENOMEM;
+
+#define sname dlb2->xstats[stat_id].name.name
+	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
+		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
+			.fn_id = DLB2_XSTATS_FN_DEV,
+			.stat = dev_types[i],
+			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
+			.reset_allowed = dev_reset_allowed[i],
+		};
+		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
+	}
+	dlb2->xstats_count_mode_dev = stat_id;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
+		dlb2->xstats_offset_for_port[port] = stat_id;
+
+		uint32_t count_offset = stat_id;
+
+		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_PORT,
+				.obj_idx = port,
+				.stat = port_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_PORT,
+				.reset_allowed = port_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "port_%u_%s",
+				 port, port_stats[i]);
+		}
+
+		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_port = stat_id - dlb2->xstats_count_mode_dev;
+
+	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
+		uint32_t count_offset = stat_id;
+
+		dlb2->xstats_offset_for_qid[qid] = stat_id;
+
+		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_QUEUE,
+				.obj_idx = qid,
+				.stat = qid_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
+				.reset_allowed = qid_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "qid_%u_%s",
+				 qid, qid_stats[i]);
+		}
+
+		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_queue = stat_id -
+		(dlb2->xstats_count_mode_dev + dlb2->xstats_count_mode_port);
+#undef sname
+
+	dlb2->xstats_count = stat_id;
+
+	return 0;
+}
+
+void
+dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
+{
+	rte_free(dlb2->xstats);
+	dlb2->xstats_count = 0;
+}
+
+int
+dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstats_names,
+		unsigned int *ids, unsigned int size)
+{
+	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+	uint32_t start_offset = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			break;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			break;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
+		break;
+	default:
+		return -EINVAL;
+	};
+
+	if (xstats_mode_count > size || ids == NULL || xstats_names == NULL)
+		return xstats_mode_count;
+
+	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
+		if (dlb2->xstats[i].mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != dlb2->xstats[i].obj_idx)
+			continue;
+
+		xstats_names[xidx] = dlb2->xstats[i].name;
+		if (ids)
+			ids[xidx] = start_offset + xidx;
+		xidx++;
+	}
+	return xidx;
+}
+
+static int
+dlb2_xstats_update(struct dlb2_eventdev *dlb2,
+		enum rte_event_dev_xstats_mode mode,
+		uint8_t queue_port_id, const unsigned int ids[],
+		uint64_t values[], unsigned int n, const uint32_t reset)
+{
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			goto invalid_value;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			goto invalid_value;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		break;
+	default:
+		goto invalid_value;
+	};
+
+	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
+		dlb2_xstats_fn fn;
+
+		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != xs->obj_idx)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			goto invalid_value;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+
+		if (values)
+			values[xidx] = val;
+
+		if (xs->reset_allowed && reset)
+			xs->reset_value += val;
+
+		xidx++;
+	}
+
+	return xidx;
+
+invalid_value:
+	return -EINVAL;
+}
+
+int
+dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	const uint32_t reset = 0;
+
+	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
+				  reset);
+}
+
+uint64_t
+dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+				 const char *name, unsigned int *id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	dlb2_xstats_fn fn;
+
+	for (i = 0; i < dlb2->xstats_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (strncmp(xs->name.name, name,
+			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
+			if (id != NULL)
+				*id = i;
+
+			switch (xs->fn_id) {
+			case DLB2_XSTATS_FN_DEV:
+				fn = get_dev_stat;
+				break;
+			case DLB2_XSTATS_FN_PORT:
+				fn = get_port_stat;
+				break;
+			case DLB2_XSTATS_FN_QUEUE:
+				fn = get_queue_stat;
+				break;
+			default:
+				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
+					  xs->fn_id);
+				return (uint64_t)-1;
+			}
+
+			return fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+		}
+	}
+	if (id != NULL)
+		*id = (uint32_t)-1;
+	return (uint64_t)-1;
+}
+
+static void
+dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
+			uint32_t num)
+{
+	uint32_t i;
+	dlb2_xstats_fn fn;
+
+	for (i = start; i < start + num; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (!xs->reset_allowed)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			return;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
+		xs->reset_value = val;
+	}
+}
+
+static int
+dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
+			const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_QUEUE,
+					queue_id, ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	if (ids == NULL)
+		dlb2_xstats_reset_range(dlb2,
+			dlb2->xstats_offset_for_qid[queue_id],
+			dlb2->xstats_count_per_qid[queue_id]);
+
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
+		       const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+	int offset = dlb2->xstats_offset_for_port[port_id];
+	int nb_stat = dlb2->xstats_count_per_port[port_id];
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_PORT, port_id,
+					ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
+		      uint32_t nb_ids)
+{
+	uint32_t i;
+
+	if (ids) {
+		for (i = 0; i < nb_ids; i++) {
+			uint32_t id = ids[i];
+
+			if (id >= dlb2->xstats_count_mode_dev)
+				return -EINVAL;
+			dlb2_xstats_reset_range(dlb2, id, 1);
+		}
+	} else {
+		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
+			dlb2_xstats_reset_range(dlb2, i, 1);
+	}
+
+	return 0;
+}
+
+int
+dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+			   enum rte_event_dev_xstats_mode mode,
+			   int16_t queue_port_id,
+			   const uint32_t ids[],
+			   uint32_t nb_ids)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	uint32_t i;
+
+	/* handle -1 for queue_port_id here, looping over all ports/queues */
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
+			return -EINVAL;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+				if (dlb2_xstats_reset_port(dlb2, i,
+							   ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
+			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
+						   ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
+				if (dlb2_xstats_reset_queue(dlb2, i,
+							    ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
+			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
+						    ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	};
+
+	return 0;
+}
+
+void
+dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	int i;
+
+	if (f == NULL) {
+		printf("Invalid file pointer\n");
+		return;
+	}
+
+	if (dev == NULL) {
+		fprintf(f, "Invalid event device\n");
+		return;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (dlb2 == NULL) {
+		fprintf(f, "DLB2 Event device cannot be dumped!\n");
+		return;
+	}
+
+	if (!dlb2->configured)
+		fprintf(f, "DLB2 Event device is not configured\n");
+
+	handle = &dlb2->qm_instance;
+
+	fprintf(f, "================\n");
+	fprintf(f, "DLB2 Device Dump\n");
+	fprintf(f, "================\n");
+
+	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
+		dlb2->umwait_allowed ? "yes" : "no");
+
+	/* Generic top level device information */
+
+	fprintf(f, "device is configured and run state =");
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		fprintf(f, "STOPPED\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
+		fprintf(f, "STOPPING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
+		fprintf(f, "STARTING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
+		fprintf(f, "STARTED\n");
+	else
+		fprintf(f, "UNEXPECTED\n");
+
+	fprintf(f, "domain ID=%u, socket_id=%u, evdev=%p\n",
+		handle->domain_id, handle->info.socket_id, dlb2->event_dev);
+
+	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
+		dlb2->num_dir_ports, dlb2->num_dir_queues);
+
+	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
+		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
+
+	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
+		handle->cfg.resources.num_atomic_inflights,
+		handle->cfg.resources.num_hist_list_entries);
+
+	fprintf(f, "results from most recent hw resource query:\n");
+
+	fprintf(f, "\tnum_sched_domains = %u\n",
+		dlb2->hw_rsrc_query_results.num_sched_domains);
+
+	fprintf(f, "\tnum_ldb_queues = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_queues);
+
+	fprintf(f, "\tnum_ldb_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_ports);
+
+	fprintf(f, "\tnum_dir_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_ports);
+
+	fprintf(f, "\tnum_atomic_inflights = %u\n",
+		dlb2->hw_rsrc_query_results.num_atomic_inflights);
+
+	fprintf(f, "\tnum_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.num_hist_list_entries);
+
+	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
+
+	fprintf(f, "\tnum_ldb_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_credits);
+
+	fprintf(f, "\tnum_dir_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_credits);
+
+	/* Port level information */
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
+		int j;
+
+		if (!p->enq_configured)
+			fprintf(f, "Port_%d is not configured\n", i);
+
+		fprintf(f, "Port_%d\n", i);
+		fprintf(f, "=======\n");
+
+		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
+			p->id, p->setup_done);
+
+		fprintf(f, "\tconfig state=%d, port state=%d\n",
+			p->qm_port.config_state, p->qm_port.state);
+
+		fprintf(f, "\tport is %s\n",
+			p->qm_port.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\toutstanding releases=%u\n",
+			p->outstanding_releases);
+
+		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
+			p->inflight_max, p->inflight_credits);
+
+		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
+			p->credit_update_quanta, p->implicit_release);
+
+		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (p->link[j].valid)
+				fprintf(f, "id=%u prio=%u ",
+					p->link[j].queue_id,
+					p->link[j].priority);
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
+
+		fprintf(f, "\tcached_ldb_credits=%u\n",
+			p->qm_port.cached_ldb_credits);
+
+		fprintf(f, "\tldb_credits = %u\n",
+			p->qm_port.ldb_credits);
+
+		fprintf(f, "\tcached_dir_credits = %u\n",
+			p->qm_port.cached_dir_credits);
+
+		fprintf(f, "\tdir_credits = %u\n",
+			p->qm_port.dir_credits);
+
+		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
+			p->qm_port.gen_bit,
+			p->qm_port.cq_idx,
+			p->qm_port.cq_depth);
+
+		fprintf(f, "\tinterrupt armed=%d\n",
+			p->qm_port.int_armed);
+
+		fprintf(f, "\tPort statistics\n");
+
+		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
+			p->stats.traffic.rx_ok);
+
+		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
+			p->stats.traffic.rx_drop);
+
+		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
+			p->stats.traffic.rx_interrupt_wait);
+
+		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
+			p->stats.traffic.rx_umonitor_umwait);
+
+		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
+			p->stats.traffic.tx_ok);
+
+		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
+			p->stats.traffic.total_polls);
+
+		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
+			p->stats.traffic.zero_polls);
+
+		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_ldb_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_dir_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_max);
+
+		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_new_event_limit);
+
+		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_credits);
+
+		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
+
+		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
+
+		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
+
+		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
+			p->stats.tx_implicit_rel);
+
+		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
+			p->stats.tx_invalid);
+
+		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
+			p->stats.rx_sched_invalid);
+	}
+
+	/* Queue level information */
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
+		int j, k;
+
+		if (!q->setup_done)
+			fprintf(f, "Queue_%d is not configured\n", i);
+
+		fprintf(f, "Queue_%d\n", i);
+		fprintf(f, "========\n");
+
+		fprintf(f, "\tevqueue_%u is set up\n", q->id);
+
+		fprintf(f, "\tqueue is %s\n",
+			q->qm_queue.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
+
+		for (j = 0; j < dlb2->num_ports; j++) {
+			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
+
+			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++) {
+				if (p->link[k].valid &&
+				    p->link[k].queue_id == q->id)
+					fprintf(f, "id=%u prio=%u ",
+						p->id, p->link[k].priority);
+			}
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\tcurrent depth: %u events\n",
+			dlb2_get_queue_depth(dlb2, q));
+
+		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
+			q->qm_queue.num_qid_inflights, q->qm_queue.sched_type);
+	}
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 28fef3f..dc16c96 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files('dlb2.c',
 		'dlb2_iface.c',
+		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c'
-- 
2.6.4

[dpdk-dev] [PATCH v7 10/23] event/dlb2: add infos get and configure

[Timothy McDaniel]

Add support for configuring the DLB2 hardware.
In particular, this patch configures the DLB2
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  116 +
 drivers/event/dlb2/dlb2.c                  |  313 +++
 drivers/event/dlb2/dlb2_iface.c            |    5 +
 drivers/event/dlb2/dlb2_iface.h            |    4 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 3237 ++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |   15 +
 drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
 7 files changed, 3734 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index e3091c9..5f6c486 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -32,9 +32,125 @@ detailed understanding of the hardware, but these details are important when
 writing high-performance code. This section describes the places where the
 eventdev API and DLB2 misalign.
 
+Scheduling Domain Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are 32 scheduling domainis the DLB2.
+When one is configured, it allocates load-balanced and
+directed queues, ports, credits, and other hardware resources. Some
+resource allocations are user-controlled -- the number of queues, for example
+-- and others, like credit pools (one directed and one load-balanced pool per
+scheduling domain), are not.
+
+The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
+setup argument and the per-port ``new_event_threshold`` argument apply as
+defined in the eventdev header file. The limit is applied to all enqueues,
+regardless of whether it will consume a directed or load-balanced credit.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Reconfiguration
+~~~~~~~~~~~~~~~
+
+The Eventdev API allows one to reconfigure a device, its ports, and its queues
+by first stopping the device, calling the configuration function(s), then
+restarting the device. The DLB2 does not support configuring an individual queue
+or port without first reconfiguring the entire device, however, so there are
+certain reconfiguration sequences that are valid in the eventdev API but not
+supported by the PMD.
+
+Specifically, the PMD supports the following configuration sequence:
+1. Configure and start the device
+2. Stop the device
+3. (Optional) Reconfigure the device
+4. (Optional) If step 3 is run:
+
+   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
+   b. The reconfigured port(s) lose their previous queue links.
+
+5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
+6. Restart the device. If the device is reconfigured in step 3 but one or more
+   of its ports or queues are not, the PMD will apply their previous
+   configuration (including port->queue links) at this time.
+
+The PMD does not support the following configuration sequences:
+1. Configure and start the device
+2. Stop the device
+3. Setup queue or setup port
+4. Start the device
+
+This sequence is not supported because the event device must be reconfigured
+before its ports or queues can be.
+
+Atomic Inflights Allocation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
+inflight event in a temporary buffer that is divided among load-balanced
+queues. If a queue's atomic buffer storage fills up, this can result in
+head-of-line-blocking. For example:
+
+- An LDB queue allocated N atomic buffer entries
+- All N entries are filled with events from flow X, which is pinned to CQ 0.
+
+Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
+scheduled. The likelihood of this case depends on the eventdev configuration,
+traffic behavior, event processing latency, potential for a worker to be
+interrupted or otherwise delayed, etc.
+
+By default, the PMD allocates 16 buffer entries for each load-balanced queue,
+which provides an even division across all 128 queues but potentially wastes
+buffer space (e.g. if not all queues are used, or aren't used for atomic
+scheduling).
+
+The PMD provides a dev arg to override the default per-queue allocation. To
+increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,atm_inflights=64
+
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 925824e..6798e66 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -85,6 +85,25 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
 	return 0;
 }
 
+static void
+dlb2_free_qe_mem(struct dlb2_port *qm_port)
+{
+	if (qm_port == NULL)
+		return;
+
+	rte_free(qm_port->qe4);
+	qm_port->qe4 = NULL;
+
+	rte_free(qm_port->int_arm_qe);
+	qm_port->int_arm_qe = NULL;
+
+	rte_free(qm_port->consume_qe);
+	qm_port->consume_qe = NULL;
+
+	rte_memzone_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz);
+	dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz = NULL;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -350,10 +369,304 @@ set_qid_depth_thresh(const char *key __rte_unused,
 }
 
 static void
+dlb2_eventdev_info_get(struct rte_eventdev *dev,
+		       struct rte_event_dev_info *dev_info)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret;
+
+	ret = dlb2_hw_query_resources(dlb2);
+	if (ret) {
+		const struct rte_eventdev_data *data = dev->data;
+
+		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+			     ret, data->dev_id);
+		/* fn is void, so fall through and return values set up in
+		 * probe
+		 */
+	}
+
+	/* Add num resources currently owned by this domain.
+	 * These would become available if the scheduling domain were reset due
+	 * to the application recalling eventdev_configure to *reconfigure* the
+	 * domain.
+	 */
+	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
+	evdev_dlb2_default_info.max_event_queues += dlb2->num_ldb_queues;
+	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
+
+	evdev_dlb2_default_info.max_event_queues =
+		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
+			RTE_EVENT_MAX_QUEUES_PER_DEV);
+
+	evdev_dlb2_default_info.max_num_events =
+		RTE_MIN(evdev_dlb2_default_info.max_num_events,
+			dlb2->max_num_events_override);
+
+	*dev_info = evdev_dlb2_default_info;
+}
+
+static int
+dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
+			    const struct dlb2_hw_rsrcs *resources_asked)
+{
+	int ret = 0;
+	struct dlb2_create_sched_domain_args *cfg;
+
+	if (resources_asked == NULL) {
+		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
+		ret = EINVAL;
+		goto error_exit;
+	}
+
+	/* Map generic qm resources to dlb2 resources */
+	cfg = &handle->cfg.resources;
+
+	/* DIR ports and queues */
+
+	cfg->num_dir_ports = resources_asked->num_dir_ports;
+
+	cfg->num_dir_credits = resources_asked->num_dir_credits;
+
+	/* LDB queues */
+
+	cfg->num_ldb_queues = resources_asked->num_ldb_queues;
+
+	/* LDB ports */
+
+	cfg->cos_strict = 0; /* Best effort */
+	cfg->num_cos_ldb_ports[0] = 0;
+	cfg->num_cos_ldb_ports[1] = 0;
+	cfg->num_cos_ldb_ports[2] = 0;
+	cfg->num_cos_ldb_ports[3] = 0;
+
+	switch (handle->cos_id) {
+	case DLB2_COS_0:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[0] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_1:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[1] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_2:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[2] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_3:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[3] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_DEFAULT:
+		/* all ldb ports are don't care ports from a cos perspective */
+		cfg->num_ldb_ports =
+			resources_asked->num_ldb_ports;
+		break;
+	}
+
+	cfg->num_ldb_credits =
+		resources_asked->num_ldb_credits;
+
+	cfg->num_atomic_inflights =
+		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
+		cfg->num_ldb_queues;
+
+	cfg->num_hist_list_entries = resources_asked->num_ldb_ports *
+		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d\n",
+		     cfg->num_ldb_queues,
+		     resources_asked->num_ldb_ports,
+		     cfg->num_dir_ports,
+		     cfg->num_atomic_inflights,
+		     cfg->num_hist_list_entries,
+		     cfg->num_ldb_credits,
+		     cfg->num_dir_credits);
+
+	/* Configure the QM */
+
+	ret = dlb2_iface_sched_domain_create(handle, cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: domain create failed, ret = %d, extra status: %s\n",
+			     ret,
+			     dlb2_error_strings[cfg->response.status]);
+
+		goto error_exit;
+	}
+
+	handle->domain_id = cfg->response.id;
+	handle->cfg.configured = true;
+
+error_exit:
+
+	return ret;
+}
+
+static void
+dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	enum dlb2_configuration_state config_state;
+	int i, j;
+
+	dlb2_iface_domain_reset(dlb2);
+
+	/* Free all dynamically allocated port memory */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
+
+	/* If reconfiguring, mark the device's queues and ports as "previously
+	 * configured." If the user doesn't reconfigure them, the PMD will
+	 * reapply their previous configuration when the device is started.
+	 */
+	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
+		DLB2_NOT_CONFIGURED;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		dlb2->ev_ports[i].qm_port.config_state = config_state;
+		/* Reset setup_done so ports can be reconfigured */
+		dlb2->ev_ports[i].setup_done = false;
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			dlb2->ev_ports[i].link[j].mapped = false;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++)
+		dlb2->ev_queues[i].qm_queue.config_state = config_state;
+
+	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
+		dlb2->ev_queues[i].setup_done = false;
+
+	dlb2->num_ports = 0;
+	dlb2->num_ldb_ports = 0;
+	dlb2->num_dir_ports = 0;
+	dlb2->num_queues = 0;
+	dlb2->num_ldb_queues = 0;
+	dlb2->num_dir_queues = 0;
+	dlb2->configured = false;
+}
+
+/* Note: 1 QM instance per QM device, QM instance/device == event device */
+static int
+dlb2_eventdev_configure(const struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
+	const struct rte_eventdev_data *data = dev->data;
+	const struct rte_event_dev_config *config = &data->dev_conf;
+	int ret;
+
+	/* If this eventdev is already configured, we must release the current
+	 * scheduling domain before attempting to configure a new one.
+	 */
+	if (dlb2->configured) {
+		dlb2_hw_reset_sched_domain(dev, true);
+
+		ret = dlb2_hw_query_resources(dlb2);
+		if (ret) {
+			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+				     ret, data->dev_id);
+			return ret;
+		}
+	}
+
+	if (config->nb_event_queues > rsrcs->num_queues) {
+		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_queues,
+			     rsrcs->num_queues);
+		return -EINVAL;
+	}
+	if (config->nb_event_ports > (rsrcs->num_ldb_ports
+			+ rsrcs->num_dir_ports)) {
+		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_ports,
+			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
+		return -EINVAL;
+	}
+	if (config->nb_events_limit > rsrcs->nb_events_limit) {
+		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_events_limit,
+			     rsrcs->nb_events_limit);
+		return -EINVAL;
+	}
+
+	if (config->event_dev_cfg & RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT)
+		dlb2->global_dequeue_wait = false;
+	else {
+		uint32_t timeout32;
+
+		dlb2->global_dequeue_wait = true;
+
+		/* note size mismatch of timeout vals in eventdev lib. */
+		timeout32 = config->dequeue_timeout_ns;
+
+		dlb2->global_dequeue_wait_ticks =
+			timeout32 * (rte_get_timer_hz() / 1E9);
+	}
+
+	/* Does this platform support umonitor/umwait? */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
+		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
+			DLB2_LOG_ERR("invalid value (%d) for RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
+				     RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
+			return -EINVAL;
+		}
+		dlb2->umwait_allowed = true;
+	}
+
+	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
+	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs->num_dir_ports;
+	/* 1 dir queue per dir port */
+	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports;
+
+	/* Scale down nb_events_limit by 4 for directed credits, since there
+	 * are 4x as many load-balanced credits.
+	 */
+	rsrcs->num_ldb_credits = 0;
+	rsrcs->num_dir_credits = 0;
+
+	if (rsrcs->num_ldb_queues)
+		rsrcs->num_ldb_credits = config->nb_events_limit;
+	if (rsrcs->num_dir_ports)
+		rsrcs->num_dir_credits = config->nb_events_limit / 4;
+	if (dlb2->num_dir_credits_override != -1)
+		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
+
+	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
+		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
+		return -ENODEV;
+	}
+
+	dlb2->new_event_limit = config->nb_events_limit;
+	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
+
+	/* Save number of ports/queues for this event dev */
+	dlb2->num_ports = config->nb_event_ports;
+	dlb2->num_queues = config->nb_event_queues;
+	dlb2->num_dir_ports = rsrcs->num_dir_ports;
+	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
+	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
+	dlb2->num_dir_queues = dlb2->num_dir_ports;
+	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
+	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
+	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
+	dlb2->max_dir_credits = rsrcs->num_dir_credits;
+
+	dlb2->configured = true;
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dev_infos_get    = dlb2_eventdev_info_get,
+		.dev_configure    = dlb2_eventdev_configure,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 0fc9991..a829b9b 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -26,3 +26,8 @@ int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
+
+int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				struct dlb2_create_sched_domain_args *args);
+
+void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4fb416e..6663dab 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -26,4 +26,8 @@ extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
 
+extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				 struct dlb2_create_sched_domain_args *args);
+
+extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 6de8b95..63a68a6 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -12,6 +12,27 @@
 #include "dlb2_regs.h"
 #include "dlb2_resource.h"
 
+#include "../../dlb2_priv.h"
+#include "../../dlb2_inline_fns.h"
+
+#define DLB2_DOM_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, domain_list, iter)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, func_list, iter)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, domain_list, it, it_tmp)
+
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, func_list, it, it_tmp)
+
 static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
 {
 	int i;
@@ -272,3 +293,3219 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
 
 	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
 }
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+	union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+	r0.field.count = domain->num_ldb_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+	r1.field.count = domain->num_dir_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+		       struct dlb2_function_resources *rsrcs,
+		       u32 domain_id,
+		       u32 cos_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	RTE_SET_USED(iter);
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+				   typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				   struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_ports,
+				   u32 cos_id,
+				   struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_ldb_port *port;
+
+		port = dlb2_get_next_ldb_port(hw, rsrcs,
+					      domain->id.phys_id, cos_id);
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_ports[cos_id],
+			      &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_ports[cos_id],
+			      &port->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_create_sched_domain_args *args,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      num,
+						      i,
+						      resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+					ret = __dlb2_attach_ldb_ports(hw,
+								      rsrcs,
+								      domain,
+								      1,
+								      cos_id,
+								      resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret < 0)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      1,
+						      j,
+						      resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 u32 num_ports,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+					   typeof(*port));
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_dir_pq_pairs, &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_dir_pq_pairs, &port->domain_list);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+					struct dlb2_hw_domain *domain,
+					u32 num_atomic_inflights,
+					struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+				     struct dlb2_hw_domain *domain,
+				     u32 num_hist_list_entries,
+				     struct dlb2_cmd_response *resp)
+{
+	struct dlb2_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb2_bitmap_find_set_bit_range(bitmap,
+						      num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+				  struct dlb2_function_resources *rsrcs,
+				  struct dlb2_hw_domain *domain,
+				  u32 num_queues,
+				  struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb2_ldb_queue *queue;
+
+		queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+					    typeof(*queue));
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_queues, &queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_queues, &queue->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+			     struct dlb2_function_resources *rsrcs,
+			     struct dlb2_hw_domain *domain,
+			     struct dlb2_create_sched_domain_args *args,
+			     struct dlb2_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb2_attach_ldb_queues(hw,
+				     rsrcs,
+				     domain,
+				     args->num_ldb_queues,
+				     resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_ports(hw,
+				    rsrcs,
+				    domain,
+				    args,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_ports(hw,
+				    rsrcs,
+				    domain,
+				    args->num_dir_ports,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_credits(rsrcs,
+				      domain,
+				      args->num_ldb_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_credits(rsrcs,
+				      domain,
+				      args->num_dir_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+						   domain,
+						   args->num_hist_list_entries,
+						   resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_atomic_inflights(rsrcs,
+					   domain,
+					   args->num_atomic_inflights,
+					   resp);
+	if (ret < 0)
+		return ret;
+
+	dlb2_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+				  struct dlb2_create_sched_domain_args *args,
+				  struct dlb2_cmd_response *resp)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb2_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+				  struct dlb2_create_sched_domain_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		    args->num_ldb_queues);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		    args->num_ldb_ports);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		    args->num_cos_ldb_ports[0]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		    args->cos_strict);
+	DLB2_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		    args->num_dir_ports);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		    args->num_atomic_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		    args->num_hist_list_entries);
+	DLB2_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		    args->num_ldb_credits);
+	DLB2_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		    args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ *	domain and its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+	if (ret)
+		return ret;
+
+	domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available domains\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (domain->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_domains contains configured domains.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	dlb2_init_domain_rsrc_lists(domain);
+
+	ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to verify args.\n",
+			    __func__);
+
+		return ret;
+	}
+
+	dlb2_list_del(&rsrcs->avail_domains, &domain->func_list);
+
+	dlb2_list_add(&rsrcs->used_domains, &domain->func_list);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+	DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+			     struct dlb2_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb2_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	int ret;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_disable(hw, port);
+
+		ret = dlb2_drain_dir_cq(hw, port);
+		if (ret < 0)
+			return ret;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_dir_pq_pair *queue)
+{
+	union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *queue)
+{
+	return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+				      struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		dlb2_movdir64b(pp_addr, hcw);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int ret, i;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			if (toggle_port)
+				dlb2_ldb_port_cq_disable(hw, port);
+
+			ret = dlb2_drain_ldb_cq(hw, port);
+			if (ret < 0)
+				return ret;
+
+			if (toggle_port)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+
+	return 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_atm_active r1;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_ldb_queue *queue)
+{
+	return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb2_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+					   struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to unmap domain queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = true;
+
+			dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+			   u32 id,
+			   bool vdev_req,
+			   unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iter1) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter2)
+			if (queue->id.virt_id == id)
+				return queue;
+	}
+
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue, iter1)
+		if (queue->id.virt_id == id)
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+						      u32 id,
+						      bool vdev_req,
+						      unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iteration;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	RTE_SET_USED(iteration);
+
+	if (id >= DLB2_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iteration)
+		if (domain->id.virt_id == id)
+			return domain;
+
+	return NULL;
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot,
+					   enum dlb2_qid_map_state new_state)
+{
+	enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb2_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB2_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB2_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB2_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB2_HW_DBG(hw,
+		    "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB2_HW_ERR(hw,
+		    "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+				enum dlb2_qid_map_state state,
+				int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+				      enum dlb2_qid_map_state state,
+				      struct dlb2_ldb_queue *queue,
+				      int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+					      struct dlb2_hw_domain *domain,
+					      struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain,
+					     struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+						struct dlb2_ldb_port *port,
+						int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 0;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 1;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+					struct dlb2_ldb_port *p,
+					struct dlb2_ldb_queue *q,
+					u8 priority)
+{
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_lsp_cq2qid0 r1;
+	union dlb2_atm_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix_00 r3;
+	union dlb2_lsp_qid2cqidix2_00 r4;
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+	    !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+	r0.field.v |= 1 << i;
+	r0.field.prio |= (priority & 0x7) << i * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+	else
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		r1.field.qid_p0 = q->id.phys_id;
+	if (i == 1 || i == 5)
+		r1.field.qid_p1 = q->id.phys_id;
+	if (i == 2 || i == 6)
+		r1.field.qid_p2 = q->id.phys_id;
+	if (i == 3 || i == 7)
+		r1.field.qid_p3 = q->id.phys_id;
+
+	if (i < 4)
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+	else
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r4.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+						  p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		r2.field.cq_p0 |= 1 << i;
+		r3.field.cq_p0 |= 1 << i;
+		r4.field.cq_p0 |= 1 << i;
+		break;
+
+	case 1:
+		r2.field.cq_p1 |= 1 << i;
+		r3.field.cq_p1 |= 1 << i;
+		r4.field.cq_p1 |= 1 << i;
+		break;
+
+	case 2:
+		r2.field.cq_p2 |= 1 << i;
+		r3.field.cq_p2 |= 1 << i;
+		r4.field.cq_p2 |= 1 << i;
+		break;
+
+	case 3:
+		r2.field.cq_p3 |= 1 << i;
+		r3.field.cq_p3 |= 1 << i;
+		r4.field.cq_p3 |= 1 << i;
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r3.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		    r4.val);
+
+	dlb2_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB2_QUEUE_MAPPED;
+
+	return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	/* Set the atomic scheduling haswork bit */
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.rlist_haswork_v = r0.field.count > 0;
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      u8 slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.rlist_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.nalb_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+					      struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+	r0.field.limit = queue->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+						struct dlb2_ldb_queue *queue)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+						struct dlb2_hw_domain *domain,
+						struct dlb2_ldb_port *port,
+						struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_lsp_qid_ldb_infl_cnt r0;
+	enum dlb2_qid_map_state state;
+	int slot, ret, i;
+	u8 prio;
+	RTE_SET_USED(iter);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: non-zero QID inflight count\n",
+			    __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			state = DLB2_QUEUE_MAPPED;
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+					 struct dlb2_ldb_port *port,
+					 struct dlb2_ldb_queue *queue,
+					 u8 priority)
+{
+	union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	int domain_id, slot, ret;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+		DLB2_HW_ERR(hw,
+			    "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		if (port->enabled)
+			dlb2_ldb_port_cq_enable(hw, port);
+
+		dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain,
+					struct dlb2_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		union dlb2_lsp_qid_ldb_infl_cnt r0;
+		struct dlb2_ldb_queue *queue;
+		int qid;
+
+		if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: unable to find queue %d\n",
+				    __func__, qid);
+			continue;
+		}
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count)
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb2_ldb_port_cq_disable(hw, port);
+
+		dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count) {
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+
+			dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port,
+				   struct dlb2_ldb_queue *queue)
+{
+	enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_atm_qid2cqidix_00 r1;
+	union dlb2_lsp_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix2_00 r3;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB2_QUEUE_MAPPED;
+	in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: QID %d isn't mapped\n",
+			    __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+	r0.field.v &= ~(1 << i);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		r1.field.cq_p0 &= ~(1 << i);
+		r2.field.cq_p0 &= ~(1 << i);
+		r3.field.cq_p0 &= ~(1 << i);
+		break;
+
+	case 1:
+		r1.field.cq_p1 &= ~(1 << i);
+		r2.field.cq_p1 &= ~(1 << i);
+		r3.field.cq_p1 &= ~(1 << i);
+		break;
+
+	case 2:
+		r1.field.cq_p2 &= ~(1 << i);
+		r2.field.cq_p2 &= ~(1 << i);
+		r3.field.cq_p2 &= ~(1 << i);
+		break;
+
+	case 3:
+		r1.field.cq_p3 &= ~(1 << i);
+		r2.field.cq_p3 &= ~(1 << i);
+		r3.field.cq_p3 &= ~(1 << i);
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+		    r1.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+		    r3.val);
+
+	dlb2_flush_csr(hw);
+
+	unmapped = DLB2_QUEUE_UNMAPPED;
+
+	return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_ldb_port *port,
+				 struct dlb2_ldb_queue *queue,
+				 u8 prio)
+{
+	if (domain->started)
+		return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   int slot)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb2_ldb_port_qid_map *map;
+		struct dlb2_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain,
+					  struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (r0.field.count > 0)
+		return false;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = false;
+
+			dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_dir_vpp_v r1;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		unsigned int offs;
+		u32 virt_id;
+
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+	}
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_ldb_vpp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			unsigned int offs;
+			u32 virt_id;
+
+			if (hw->virt_mode == DLB2_VIRT_SRIOV)
+				virt_id = port->id.virt_id;
+			else
+				virt_id = port->id.phys_id;
+
+			offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+			DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+	union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+				    r0.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+				    r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+	union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+			    r0.val);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+			    r1.val);
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+		union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+		union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+				    r1.val);
+
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID_V(idx),
+				    r2.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID2QID(idx),
+				    r3.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID_V(idx),
+				    r1.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID2QID(idx),
+				    r2.val);
+		}
+	}
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+					       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_sn_chk_enbl r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.en = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			int i;
+
+			for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+				if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					    __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		port->enabled = false;
+
+		dlb2_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	union dlb2_sys_dir_pp_v r1;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+			    r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_ldb_pp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_ldb_queue *queue;
+	int i;
+	RTE_SET_USED(iter);
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty ldb queue %d\n",
+				    __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port, iter) {
+			if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb2_ldb_cq_token_count(hw, ldb_port)) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to empty ldb port %d\n",
+					    __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir queue %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb2_dir_cq_token_count(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir port %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						   struct dlb2_ldb_port *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP2PP(offs),
+			    DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP_V(offs),
+			    DLB2_SYS_VF_LDB_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    DLB2_SYS_LDB_PP_V_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_DSBL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_BASE_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID0(port->id.phys_id),
+		    DLB2_LSP_CQ2QID0_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID1(port->id.phys_id),
+		    DLB2_LSP_CQ2QID1_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+		    DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			__dlb2_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+				       struct dlb2_dir_pq_pair *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_DSBL_RST);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_FMT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP2PP(offs),
+			    DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP_V(offs),
+			    DLB2_SYS_VF_DIR_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		__dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+			    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_ITS(queue_id),
+			    DLB2_SYS_LDB_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN(queue_id),
+			    DLB2_CHP_ORD_QID_SN_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+			    DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_V(queue_id),
+			    DLB2_SYS_LDB_QID_V_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_CFG_V(queue_id),
+			    DLB2_SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB2_CSR_WR(hw,
+				    offs[queue->sn_group],
+				    DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX2(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX2_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_ATM_QID2CQIDIX(queue_id, i),
+				    DLB2_ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb2_domain_reset_dir_port_registers(hw, domain);
+
+	dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_dir_pq_pair *tmp_dir_port;
+	struct dlb2_ldb_queue *tmp_ldb_queue;
+	struct dlb2_ldb_port *tmp_ldb_port;
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_list_head *list;
+	int ret, i;
+	RTE_SET_USED(tmp_dir_port);
+	RTE_SET_USED(tmp_ldb_queue);
+	RTE_SET_USED(tmp_ldb_port);
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list = &domain->used_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb2_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		dlb2_list_del(&domain->used_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list = &domain->avail_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		ldb_queue->owned = false;
+
+		dlb2_list_del(&domain->avail_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		list = &domain->used_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB2_QUEUE_UNMAPPED;
+
+			dlb2_list_del(&domain->used_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list = &domain->avail_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			ldb_port->owned = false;
+
+			dlb2_list_del(&domain->avail_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list = &domain->used_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		dlb2_list_del(&domain->used_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list = &domain->avail_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				    domain->hist_list_entry_base,
+				    domain->total_hist_list_entries);
+	if (ret) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			    __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	dlb2_list_del(&rsrcs->used_domains, &domain->func_list);
+	dlb2_list_add(&rsrcs->avail_domains, &domain->func_list);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain,
+					    struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_ldb_port *port;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (!dlb2_list_empty(&domain->used_ldb_ports[i]))
+			break;
+	}
+
+	if (i == DLB2_NUM_COS_DOMAINS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: No configured LDB ports\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb2_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+	 * prior to this in dlb2_domain_drain_mapped_queues().
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings != 0 ||
+		    dlb2_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ *	hardware resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain  == NULL || !domain->configured)
+		return -EINVAL;
+
+	/* Disable VPPs */
+	if (vdev_req) {
+		dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+		dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+	}
+
+	/* Disable CQ interrupts */
+	dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+	dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+	dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb2_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_mapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	dlb2_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb2_domain_disable_dir_cqs(hw, domain);
+
+	/* Disable PPs */
+	dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+	ret = dlb2_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Reset the QID and port state. */
+	dlb2_domain_reset_registers(hw, domain);
+
+	/* Hardware reset complete. Reset the domain's software state */
+	ret = dlb2_domain_reset_software_state(hw, domain);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 210b3186..285ad2c 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -598,3 +598,18 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
 
 	return 0;
 }
+
+int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+			    struct dlb2_create_sched_domain_args *args,
+			    struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_sched_domain(hw, args, resp, NOT_VF_REQ,
+					   PF_ID_ZERO);
+}
+
+int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 7d64309..9310f72 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -108,15 +108,59 @@ dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
 	return 0;
 }
 
+static int
+dlb2_pf_sched_domain_create(struct dlb2_hw_dev *handle,
+			    struct dlb2_create_sched_domain_args *arg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (dlb2_dev->domain_reset_failed) {
+		response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = dlb2_pf_create_sched_domain(&dlb2_dev->hw, arg, &response);
+	if (ret)
+		goto done;
+
+done:
+
+	arg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static void
+dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret;
+
+	dlb2_dev = (struct dlb2_dev *)dlb2->qm_instance.pf_dev;
+	ret = dlb2_pf_reset_domain(&dlb2_dev->hw, dlb2->qm_instance.domain_id);
+	if (ret)
+		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
 	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
 	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_domain_reset = dlb2_pf_domain_reset;
 	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
 	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v7 11/23] event/dlb2: add queue and port default conf

[Timothy McDaniel]

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 6798e66..ef1c000 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -661,12 +661,42 @@ dlb2_eventdev_configure(const struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
+				    uint8_t port_id,
+				    struct rte_event_port_conf *port_conf)
+{
+	RTE_SET_USED(port_id);
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	port_conf->new_event_threshold = dlb2->new_event_limit;
+	port_conf->dequeue_depth = 32;
+	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
+	port_conf->event_port_cfg = 0;
+}
+
+static void
+dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
+				     uint8_t queue_id,
+				     struct rte_event_queue_conf *queue_conf)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(queue_id);
+
+	queue_conf->nb_atomic_flows = 1024;
+	queue_conf->nb_atomic_order_sequences = 64;
+	queue_conf->event_queue_cfg = 0;
+	queue_conf->priority = 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v7 12/23] event/dlb2: add queue setup

[Timothy McDaniel]

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  73 +++--
 drivers/event/dlb2/dlb2.c                  | 312 +++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |  12 +
 drivers/event/dlb2/dlb2_iface.h            |  12 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 464 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  82 +++++
 7 files changed, 926 insertions(+), 39 deletions(-)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 5f6c486..b154cac 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -47,45 +47,40 @@ setup argument and the per-port ``new_event_threshold`` argument apply as
 defined in the eventdev header file. The limit is applied to all enqueues,
 regardless of whether it will consume a directed or load-balanced credit.
 
-Load-balanced and Directed Ports
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
-does not have the same concept, but it has a similar one: ports and queues that
-are singly-linked (i.e. linked to a single queue or port, respectively).
-
-The ``rte_event_dev_info_get()`` function reports the number of available
-event ports and queues (among other things). For the DLB2 PMD, max_event_ports
-and max_event_queues report the number of available load-balanced ports and
-queues, and max_single_link_event_port_queue_pairs reports the number of
-available directed ports and queues.
-
-When a scheduling domain is created in ``rte_event_dev_configure()``, the user
-specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
-control the total number of ports (load-balanced and directed) and the number
-of directed ports. Hence, the number of requested load-balanced ports is
-``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
-specifies the total number of queues (load-balanced and directed). The number
-of directed queues comes from ``nb_single_link_event_port_queues``, since
-directed ports and queues come in pairs.
-
-When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
-whether it should be configured as a directed (the flag is set) or a
-load-balanced (the flag is unset) port. Similarly, the
-``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
-whether it is a directed or load-balanced queue.
-
-Load-balanced ports can only be linked to load-balanced queues, and directed
-ports can only be linked to directed queues. Furthermore, directed ports can
-only be linked to a single directed queue (and vice versa), and that link
-cannot change after the eventdev is started.
-
-The eventdev API does not have a directed scheduling type. To support directed
-traffic, the dlb PMD detects when an event is being sent to a directed queue
-and overrides its scheduling type. Note that the originally selected scheduling
-type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
-will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
-port.
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
 
 Flow ID
 ~~~~~~~
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index ef1c000..8c1e06d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -688,6 +688,317 @@ dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
 	queue_conf->priority = 0;
 }
 
+static int32_t
+dlb2_get_sn_allocation(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_allocation_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_set_sn_allocation(struct dlb2_eventdev *dlb2, int group, int num)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_set_sn_allocation_args cfg;
+	int ret;
+
+	cfg.num = num;
+	cfg.group = group;
+
+	ret = dlb2_iface_set_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: set_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int32_t
+dlb2_get_sn_occupancy(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_occupancy_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_occupancy(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_occupancy ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+/* Query the current sequence number allocations and, if they conflict with the
+ * requested LDB queue configuration, attempt to re-allocate sequence numbers.
+ * This is best-effort; if it fails, the PMD will attempt to configure the
+ * load-balanced queue and return an error.
+ */
+static void
+dlb2_program_sn_allocation(struct dlb2_eventdev *dlb2,
+			   const struct rte_event_queue_conf *queue_conf)
+{
+	int grp_occupancy[DLB2_NUM_SN_GROUPS];
+	int grp_alloc[DLB2_NUM_SN_GROUPS];
+	int i, sequence_numbers;
+
+	sequence_numbers = (int)queue_conf->nb_atomic_order_sequences;
+
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		int total_slots;
+
+		grp_alloc[i] = dlb2_get_sn_allocation(dlb2, i);
+		if (grp_alloc[i] < 0)
+			return;
+
+		total_slots = DLB2_MAX_LDB_SN_ALLOC / grp_alloc[i];
+
+		grp_occupancy[i] = dlb2_get_sn_occupancy(dlb2, i);
+		if (grp_occupancy[i] < 0)
+			return;
+
+		/* DLB has at least one available slot for the requested
+		 * sequence numbers, so no further configuration required.
+		 */
+		if (grp_alloc[i] == sequence_numbers &&
+		    grp_occupancy[i] < total_slots)
+			return;
+	}
+
+	/* None of the sequence number groups are configured for the requested
+	 * sequence numbers, so we have to reconfigure one of them. This is
+	 * only possible if a group is not in use.
+	 */
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		if (grp_occupancy[i] == 0)
+			break;
+	}
+
+	if (i == DLB2_NUM_SN_GROUPS) {
+		printf("[%s()] No groups with %d sequence_numbers are available or have free slots\n",
+		       __func__, sequence_numbers);
+		return;
+	}
+
+	/* Attempt to configure slot i with the requested number of sequence
+	 * numbers. Ignore the return value -- if this fails, the error will be
+	 * caught during subsequent queue configuration.
+	 */
+	dlb2_set_sn_allocation(dlb2, i, sequence_numbers);
+}
+
+static int32_t
+dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 const struct rte_event_queue_conf *evq_conf)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_queue *queue = &ev_queue->qm_queue;
+	struct dlb2_create_ldb_queue_args cfg;
+	int32_t ret;
+	uint32_t qm_qid;
+	int sched_type = -1;
+
+	if (evq_conf == NULL)
+		return -EINVAL;
+
+	if (evq_conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_ALL_TYPES) {
+		if (evq_conf->nb_atomic_order_sequences != 0)
+			sched_type = RTE_SCHED_TYPE_ORDERED;
+		else
+			sched_type = RTE_SCHED_TYPE_PARALLEL;
+	} else
+		sched_type = evq_conf->schedule_type;
+
+	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
+	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
+	cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences;
+
+	if (sched_type != RTE_SCHED_TYPE_ORDERED) {
+		cfg.num_sequence_numbers = 0;
+		cfg.num_qid_inflights = 2048;
+	}
+
+	/* App should set this to the number of hardware flows they want, not
+	 * the overall number of flows they're going to use. E.g. if app is
+	 * using 64 flows and sets compression to 64, best-case they'll get
+	 * 64 unique hashed flows in hardware.
+	 */
+	switch (evq_conf->nb_atomic_flows) {
+	/* Valid DLB2 compression levels */
+	case 64:
+	case 128:
+	case 256:
+	case 512:
+	case (1 * 1024): /* 1K */
+	case (2 * 1024): /* 2K */
+	case (4 * 1024): /* 4K */
+	case (64 * 1024): /* 64K */
+		cfg.lock_id_comp_level = evq_conf->nb_atomic_flows;
+		break;
+	default:
+		/* Invalid compression level */
+		cfg.lock_id_comp_level = 0; /* no compression */
+	}
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create LB event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	qm_qid = cfg.response.id;
+
+	/* Save off queue config for debug, resource lookups, and reconfig */
+	queue->num_qid_inflights = cfg.num_qid_inflights;
+	queue->num_atm_inflights = cfg.num_atomic_inflights;
+
+	queue->sched_type = sched_type;
+	queue->config_state = DLB2_CONFIGURED;
+
+	DLB2_LOG_DBG("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n",
+		     qm_qid,
+		     cfg.num_atomic_inflights,
+		     cfg.num_sequence_numbers,
+		     cfg.num_qid_inflights);
+
+	return qm_qid;
+}
+
+static int
+dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int32_t qm_qid;
+
+	if (queue_conf->nb_atomic_order_sequences)
+		dlb2_program_sn_allocation(dlb2, queue_conf);
+
+	qm_qid = dlb2_hw_create_ldb_queue(dlb2, ev_queue, queue_conf);
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the load-balanced queue\n");
+
+		return qm_qid;
+	}
+
+	dlb2->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int dlb2_num_dir_queues_setup(struct dlb2_eventdev *dlb2)
+{
+	int i, num = 0;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].setup_done &&
+		    dlb2->ev_queues[i].qm_queue.is_directed)
+			num++;
+	}
+
+	return num;
+}
+
+static void
+dlb2_queue_link_teardown(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue)
+{
+	struct dlb2_eventdev_port *ev_port;
+	int i, j;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (!ev_port->link[j].valid ||
+			    ev_port->link[j].queue_id != ev_queue->id)
+				continue;
+
+			ev_port->link[j].valid = false;
+			ev_port->num_links--;
+		}
+	}
+
+	ev_queue->num_links = 0;
+}
+
+static int
+dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
+			  uint8_t ev_qid,
+			  const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_queue *ev_queue;
+	int ret;
+
+	if (queue_conf == NULL)
+		return -EINVAL;
+
+	if (ev_qid >= dlb2->num_queues)
+		return -EINVAL;
+
+	ev_queue = &dlb2->ev_queues[ev_qid];
+
+	ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg &
+		RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+	ev_queue->id = ev_qid;
+	ev_queue->conf = *queue_conf;
+
+	if (!ev_queue->qm_queue.is_directed) {
+		ret = dlb2_eventdev_ldb_queue_setup(dev, ev_queue, queue_conf);
+	} else {
+		/* The directed queue isn't setup until link time, at which
+		 * point we know its directed port ID. Directed queue setup
+		 * will only fail if this queue is already setup or there are
+		 * no directed queues left to configure.
+		 */
+		ret = 0;
+
+		ev_queue->qm_queue.config_state = DLB2_NOT_CONFIGURED;
+
+		if (ev_queue->setup_done ||
+		    dlb2_num_dir_queues_setup(dlb2) == dlb2->num_dir_queues)
+			ret = -EINVAL;
+	}
+
+	/* Tear down pre-existing port->queue links */
+	if (!ret && dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_queue_link_teardown(dlb2, ev_queue);
+
+	if (!ret)
+		ev_queue->setup_done = true;
+
+	return ret;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -696,6 +1007,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a829b9b..4c07574 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -31,3 +31,15 @@ int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				struct dlb2_create_sched_domain_args *args);
 
 void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_ldb_queue_args *cfg);
+
+int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_sn_allocation_args *args);
+
+int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_set_sn_allocation_args *args);
+
+int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_occupancy_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 6663dab..4c88fe0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -30,4 +30,16 @@ extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				 struct dlb2_create_sched_domain_args *args);
 
 extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+extern int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_queue_args *cfg);
+
+extern int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_allocation_args *args);
+
+extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_set_sn_allocation_args *args);
+
+extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				  struct dlb2_get_sn_occupancy_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 63a68a6..e6ea0d7 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3509,3 +3509,467 @@ unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
 
 	return num;
 }
+
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_ldb_queue *queue,
+				     struct dlb2_create_ldb_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+	union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+	union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+	union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+	union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+	union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+	union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+	union dlb2_sys_ldb_qid_its r7 = { {0} };
+	union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+	union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+	union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+	union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+	union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+	union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+	struct dlb2_sn_group *sn_group;
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r3.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	r4.field.limit = args->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+	r5.field.limit = queue->aqed_limit;
+
+	if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+		r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+		    r5.val);
+
+	switch (args->lock_id_comp_level) {
+	case 64:
+		r6.field.compress_code = 1;
+		break;
+	case 128:
+		r6.field.compress_code = 2;
+		break;
+	case 256:
+		r6.field.compress_code = 3;
+		break;
+	case 512:
+		r6.field.compress_code = 4;
+		break;
+	case 1024:
+		r6.field.compress_code = 5;
+		break;
+	case 2048:
+		r6.field.compress_code = 6;
+		break;
+	case 4096:
+		r6.field.compress_code = 7;
+		break;
+	case 0:
+	case 65536:
+		r6.field.compress_code = 0;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+		    r6.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r7.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+		    r7.val);
+
+	r8.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+		    r8.val);
+
+	r9.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+		    r9.val);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	r10.field.qid_fid_limit = 512;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+		    r10.val);
+
+	/* Configure SNs */
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	r11.field.mode = sn_group->mode;
+	r11.field.slot = queue->sn_slot;
+	r11.field.grp  = sn_group->id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+	r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+	r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		r0.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+		r1.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+		r2.field.vqid = queue->id.virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+			    r2.val);
+	}
+
+	r13.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+				  struct dlb2_ldb_queue *queue,
+				  struct dlb2_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb2_sn_group_full(group)) {
+			slot = dlb2_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no sequence number slots available\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+				struct dlb2_hw_domain *domain,
+				struct dlb2_ldb_queue *queue,
+				struct dlb2_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_ldb_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (dlb2_list_empty(&domain->avail_ldb_queues)) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb2_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_ldb_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		    args->num_sequence_numbers);
+	DLB2_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		    args->num_qid_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		    args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+
+	dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			    __func__, __LINE__);
+		return ret;
+	}
+
+	dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_queues, &queue->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_queues, &queue->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+						unsigned int group_id,
+						unsigned long val)
+{
+	DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+	DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+	DLB2_HW_DBG(hw, "\tValue:    %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val)
+{
+	u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+	union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+	struct dlb2_sn_group *group;
+	int mode;
+
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	group = &hw->rsrcs.sn_groups[group_id];
+
+	/*
+	 * Once the first load-balanced queue using an SN group is configured,
+	 * the group cannot be changed.
+	 */
+	if (group->slot_use_bitmap != 0)
+		return -EPERM;
+
+	for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+		if (val == valid_allocations[mode])
+			break;
+
+	if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+		return -EINVAL;
+
+	group->mode = mode;
+	group->sequence_numbers_per_queue = val;
+
+	r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+	r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+	DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+	dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 285ad2c..14bb04b 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -613,3 +613,13 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
 {
 	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_ldb_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 9310f72..a6cd178 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -150,6 +150,84 @@ dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
 		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
 }
 
+static int
+dlb2_pf_ldb_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_ldb_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_ldb_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw_dev *handle,
+			 struct dlb2_get_sn_occupancy_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_number_occupancy(&dlb2_dev->hw,
+						       args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_numbers(&dlb2_dev->hw, args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_set_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_set_group_sequence_numbers(&dlb2_dev->hw, args->group,
+					      args->num);
+
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -161,6 +239,10 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
+	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
+	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
+	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v7 13/23] event/dlb2: add port setup

[Timothy McDaniel]

Configure the load balanced (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  75 +++
 drivers/event/dlb2/dlb2.c                  | 498 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   8 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 922 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  28 +
 drivers/event/dlb2/pf/dlb2_pf.c            | 180 ++++++
 7 files changed, 1719 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index b154cac..bbd6ac8 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -82,6 +82,81 @@ The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
 the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
 load-balanced queues can use the full 16-bit flow ID range.
 
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 8c1e06d..7898e42 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -999,6 +999,503 @@ dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
 	return ret;
 }
 
+static int
+dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_cq_pop_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_cq_pop_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL)	{
+		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->consume_qe = qe;
+
+	qe->qe_valid = 0;
+	qe->qe_frag = 0;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
+	 * and so on.
+	 */
+	qe->tokens = 0;	/* set at run time */
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_enqueue_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_enqueue_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL) {
+		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->int_arm_qe = qe;
+
+	/* V2 - INT ARM is CQ_TOKEN + FRAG */
+	qe->qe_valid = 0;
+	qe->qe_frag = 1;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
+{
+	int ret, sz;
+
+	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct dlb2_enqueue_qe);
+
+	qm_port->qe4 = rte_zmalloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
+
+	if (qm_port->qe4 == NULL) {
+		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	ret = dlb2_init_consume_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	return 0;
+
+error_exit:
+
+	dlb2_free_qe_mem(qm_port);
+
+	return ret;
+}
+
+static int
+dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_ldb_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_CQ_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	cfg.cq_history_list_size = DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	if (handle->cos_id == DLB2_COS_DEFAULT)
+		cfg.cos_id = 0;
+	else
+		cfg.cos_id = handle->cos_id;
+
+	cfg.cos_strict = 0;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* If there are no directed ports, the kernel driver will ignore this
+	 * port's directed credit settings. Don't use enqueue_depth if it would
+	 * require more directed credits than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2; /* back ptr */
+	/*
+	 * Allocate and init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_ldb_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(qm_port->cfg.ldb));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     qm_port->ldb_credits,
+		     qm_port->dir_credits);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
+
+	return ret;
+}
+
+static void
+dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port)
+{
+	struct dlb2_eventdev_queue *ev_queue;
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (!ev_port->link[i].valid)
+			continue;
+
+		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
+
+		ev_port->link[i].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+}
+
+static int
+dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (dlb2 == NULL || handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* Directed queues are configured at link time. */
+	cfg.queue_id = -1;
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* Don't use enqueue_depth if it would require more directed credits
+	 * than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2;  /* back ptr */
+
+	/*
+	 * Init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE to be aligned
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_dir_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = cfg.cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(qm_port->cfg.dir));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     dir_credit_high_watermark,
+		     ldb_credit_high_watermark);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_setup(struct rte_eventdev *dev,
+			 uint8_t ev_port_id,
+			 const struct rte_event_port_conf *port_conf)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_eventdev_port *ev_port;
+	int ret;
+
+	if (dev == NULL || port_conf == NULL) {
+		DLB2_LOG_ERR("Null parameter\n");
+		return -EINVAL;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
+		return -EINVAL;
+
+	if (port_conf->dequeue_depth >
+		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
+	    port_conf->enqueue_depth >
+		evdev_dlb2_default_info.max_event_port_enqueue_depth)
+		return -EINVAL;
+
+	ev_port = &dlb2->ev_ports[ev_port_id];
+	/* configured? */
+	if (ev_port->setup_done) {
+		DLB2_LOG_ERR("evport %d is already configured\n", ev_port_id);
+		return -EINVAL;
+	}
+
+	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
+		RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (!ev_port->qm_port.is_directed) {
+		ret = dlb2_hw_create_ldb_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the lB port ve portId=%d\n",
+				     ev_port_id);
+
+			return ret;
+		}
+	} else {
+		ret = dlb2_hw_create_dir_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the DIR port\n");
+			return ret;
+		}
+	}
+
+	/* Save off port config for reconfig */
+	ev_port->conf = *port_conf;
+
+	ev_port->id = ev_port_id;
+	ev_port->enq_configured = true;
+	ev_port->setup_done = true;
+	ev_port->inflight_max = port_conf->new_event_threshold;
+	ev_port->implicit_release = !(port_conf->event_port_cfg &
+		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
+	ev_port->outstanding_releases = 0;
+	ev_port->inflight_credits = 0;
+	ev_port->credit_update_quanta = RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
+	ev_port->dlb2 = dlb2; /* reverse link */
+
+	/* Tear down pre-existing port->queue links */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
+
+	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1009,6 +1506,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
+		.port_setup       = dlb2_eventdev_port_setup,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 4c07574..01818c7 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -43,3 +43,11 @@ int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				   struct dlb2_get_sn_occupancy_args *args);
+
+int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_dir_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4c88fe0..a6b923a 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -42,4 +42,12 @@ extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				  struct dlb2_get_sn_occupancy_args *args);
+
+extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_ldb_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_dir_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index e6ea0d7..a633d3e 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3973,3 +3973,925 @@ int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_ldb_port *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_pp2vas r0 = { {0} };
+	union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+		union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_ldb_port *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_ldb_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+	union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+	union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+	union dlb2_chp_hist_list_lim r5 = { {0} };
+	union dlb2_chp_hist_list_base r6 = { {0} };
+	union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+	union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+	union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+	union dlb2_sys_ldb_cq_at r10 = { {0} };
+	union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+	union dlb2_chp_ldb_cq2vas r12 = { {0} };
+	union dlb2_lsp_cq2priov r13 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r14.field.token_count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    r14.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	r5.field.limit = port->hist_list_entry_limit - 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+	r6.field.base = port->hist_list_entry_base;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	r7.field.limit = args->cq_history_list_size;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+	r8.field.push_ptr = r6.field.base;
+	r8.field.generation = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    r8.val);
+
+	r9.field.pop_ptr = r6.field.base;
+	r9.field.generation = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+	/* Disable the port's QID mappings */
+	r13.field.v = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+	return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_ldb_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb2_ldb_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+	if (ret < 0)
+		return ret;
+
+	dlb2_ldb_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_ldb_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		    args->cq_history_list_size);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+	DLB2_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		    args->cos_strict);
+}
+
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_ldb_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+		resp->status = DLB2_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		if (dlb2_list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	} else {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			if (!dlb2_list_empty(&domain->avail_ldb_ports[i]))
+				break;
+		}
+
+		if (i == DLB2_NUM_COS_DOMAINS) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ *	its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	int ret, cos_id, i;
+
+	dlb2_log_create_ldb_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->cos_strict) {
+		cos_id = args->cos_id;
+
+		port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+					  typeof(*port));
+	} else {
+		int idx;
+
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+			port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+						  typeof(*port));
+			if (port)
+				break;
+		}
+
+		cos_id = idx;
+	}
+
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (port->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_ldb_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_ports[cos_id], &port->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_ports[cos_id], &port->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_dir_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+}
+
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+			    bool vdev_req,
+			    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+
+	return NULL;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_dir_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the queue is already configured, validate
+	 * the queue ID, its domain, and whether the queue is configured.
+	 */
+	if (args->queue_id != -1) {
+		struct dlb2_dir_pq_pair *queue;
+
+		queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+						    vdev_req,
+						    domain);
+
+		if (queue == NULL || queue->domain_id.phys_id !=
+				domain->id.phys_id ||
+				!queue->queue_configured) {
+			resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the port's queue is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->queue_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_dir_pq_pair *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_dir_pp2vas r0 = { {0} };
+	union dlb2_sys_dir_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+		union dlb2_sys_dir_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_dir_pq_pair *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_dir_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+	union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+	union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+	union dlb2_sys_dir_cq_fmt r9 = { {0} };
+	union dlb2_sys_dir_cq_at r10 = { {0} };
+	union dlb2_sys_dir_cq_pasid r11 = { {0} };
+	union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r13.field.count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    r13.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.disable_wb_opt = 0;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	r9.field.keep_pf_ppid = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+	return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_dir_pq_pair *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_dir_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb2_dir_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+
+	if (ret < 0)
+		return ret;
+
+	dlb2_dir_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ *	and queue. The port/queue pair have the same ID and name.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *port;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (args->queue_id != -1)
+		port = dlb2_get_domain_used_dir_pq(args->queue_id,
+						   vdev_req,
+						   domain);
+	else
+		port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					  typeof(*port));
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_dir_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs, &port->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs, &port->domain_list);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 14bb04b..2a089f6 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -623,3 +623,31 @@ dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_ldb_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
+
+int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_dir_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a6cd178..2a82b8f 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -228,6 +228,184 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static void *
+dlb2_alloc_coherent_aligned(const struct rte_memzone **mz, uintptr_t *phys,
+			    size_t size, int align)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t core_id = rte_lcore_id();
+	unsigned int socket_id;
+
+	snprintf(mz_name, sizeof(mz_name) - 1, "event_dlb2_pf_%lx",
+		 (unsigned long)rte_get_timer_cycles());
+	if (core_id == (unsigned int)LCORE_ID_ANY)
+		core_id = rte_get_main_lcore();
+	socket_id = rte_lcore_to_socket_id(core_id);
+	*mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
+					 RTE_MEMZONE_IOVA_CONTIG, align);
+	if (*mz == NULL) {
+		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu bytes - %s\n",
+			     size, rte_strerror(rte_errno));
+		*phys = 0;
+		return NULL;
+	}
+	*phys = (*mz)->iova;
+	return (*mz)->addr;
+}
+
+static int
+dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_ldb_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret, cq_alloc_depth;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = false;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	cq_alloc_depth = RTE_MAX(cfg->cq_depth, DLB2_MIN_HARDWARE_CQ_DEPTH);
+	alloc_sz = cq_alloc_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].cq_base = (void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+	return ret;
+}
+
+static int
+dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_dir_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = true;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	alloc_sz = cfg->cq_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
+		(void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -240,6 +418,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v7 14/23] event/dlb2: add port link

[Timothy McDaniel]

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 302 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 633 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
 6 files changed, 1007 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7898e42..5068d6d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1496,6 +1496,307 @@ dlb2_eventdev_port_setup(struct rte_eventdev *dev,
 	return 0;
 }
 
+static int16_t
+dlb2_hw_map_ldb_qid_to_port(struct dlb2_hw_dev *handle,
+			    uint32_t qm_port_id,
+			    uint16_t qm_qid,
+			    uint8_t priority)
+{
+	struct dlb2_map_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	/* Build message */
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+	cfg.priority = EV_TO_DLB2_PRIO(priority);
+
+	ret = dlb2_iface_map_qid(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: map qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		DLB2_LOG_ERR("dlb2: grp=%d, qm_port=%d, qm_qid=%d prio=%d\n",
+			     handle->domain_id, cfg.port_id,
+			     cfg.qid,
+			     cfg.priority);
+	} else {
+		DLB2_LOG_DBG("dlb2: mapped queue %d to qm_port %d\n",
+			     qm_qid, qm_port_id);
+	}
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port,
+			  struct dlb2_eventdev_queue *ev_queue,
+			  uint8_t priority)
+{
+	int first_avail = -1;
+	int ret, i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid) {
+			if (ev_port->link[i].queue_id == ev_queue->id &&
+			    ev_port->link[i].priority == priority) {
+				if (ev_port->link[i].mapped)
+					return 0; /* already mapped */
+				first_avail = i;
+			}
+		} else if (first_avail == -1)
+			first_avail = i;
+	}
+	if (first_avail == -1) {
+		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
+			     ev_port->qm_port.id);
+		return -EINVAL;
+	}
+
+	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
+					  ev_port->qm_port.id,
+					  ev_queue->qm_queue.id,
+					  priority);
+
+	if (!ret)
+		ev_port->link[first_avail].mapped = true;
+
+	return ret;
+}
+
+static int32_t
+dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 int32_t qm_port_id)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_queue_args cfg;
+	int32_t ret;
+
+	/* The directed port is always configured before its queue */
+	cfg.port_id = qm_port_id;
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_dir_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	int32_t qm_qid;
+
+	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port->qm_port.id);
+
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the DIR queue\n");
+		return qm_qid;
+	}
+
+	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int
+dlb2_do_port_link(struct rte_eventdev *dev,
+		  struct dlb2_eventdev_queue *ev_queue,
+		  struct dlb2_eventdev_port *ev_port,
+		  uint8_t prio)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int err;
+
+	/* Don't link until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	if (ev_queue->qm_queue.is_directed)
+		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue, ev_port);
+	else
+		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue, prio);
+
+	if (err) {
+		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
+			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
+			     ev_queue->id, ev_port->id);
+
+		rte_errno = err;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
+			uint8_t queue_id,
+			bool link_exists,
+			int index)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	bool port_is_dir, queue_is_dir;
+
+	if (queue_id > dlb2->num_queues) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	ev_queue = &dlb2->ev_queues[queue_id];
+
+	if (!ev_queue->setup_done &&
+	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	port_is_dir = ev_port->qm_port.is_directed;
+	queue_is_dir = ev_queue->qm_queue.is_directed;
+
+	if (port_is_dir != queue_is_dir) {
+		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
+			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
+			     port_is_dir ? "DIR" : "LDB", ev_port->id);
+
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if there is space for the requested link */
+	if (!link_exists && index == -1) {
+		DLB2_LOG_ERR("no space for new link\n");
+		rte_errno = -ENOSPC;
+		return -1;
+	}
+
+	/* Check if the directed port is already linked */
+	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if the directed queue is already linked */
+	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
+			     ev_queue->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
+			const uint8_t queues[], const uint8_t priorities[],
+			uint16_t nb_links)
+
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i, j;
+
+	RTE_SET_USED(dev);
+
+	if (ev_port == NULL) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!ev_port->setup_done &&
+	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
+	 * queues pointer.
+	 */
+	if (nb_links == 0) {
+		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
+		     nb_links,
+		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
+		     ev_port->id);
+
+	for (i = 0; i < nb_links; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		uint8_t queue_id, prio;
+		bool found = false;
+		int index = -1;
+
+		queue_id = queues[i];
+		prio = priorities[i];
+
+		/* Check if the link already exists. */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].valid) {
+				if (ev_port->link[j].queue_id == queue_id) {
+					found = true;
+					index = j;
+					break;
+				}
+			} else if (index == -1) {
+				index = j;
+			}
+
+		/* could not link */
+		if (index == -1)
+			break;
+
+		/* Check if already linked at the requested priority */
+		if (found && ev_port->link[j].priority == prio)
+			continue;
+
+		if (dlb2_validate_port_link(ev_port, queue_id, found, index))
+			break; /* return index of offending queue */
+
+		ev_queue = &dlb2->ev_queues[queue_id];
+
+		if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+			break; /* return index of offending queue */
+
+		ev_queue->num_links++;
+
+		ev_port->link[index].queue_id = queue_id;
+		ev_port->link[index].priority = prio;
+		ev_port->link[index].valid = true;
+		/* Entry already exists?  If so, then must be prio change */
+		if (!found)
+			ev_port->num_links++;
+	}
+	return i;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1507,6 +1808,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_link        = dlb2_eventdev_port_link,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 01818c7..7d1ba73 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -51,3 +51,9 @@ int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 				  struct dlb2_create_dir_port_args *cfg,
 				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_dir_queue_args *cfg);
+
+int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+			  struct dlb2_map_qid_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index a6b923a..4eddd14 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -50,4 +50,10 @@ extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 struct dlb2_create_dir_port_args *cfg,
 					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+					struct dlb2_create_dir_queue_args *cfg);
+
+extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+				 struct dlb2_map_qid_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index a633d3e..14ad262 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -4895,3 +4895,636 @@ int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_dir_pq_pair *queue,
+				     struct dlb2_create_dir_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_dir_vasqid_v r0 = { {0} };
+	union dlb2_sys_dir_qid_its r1 = { {0} };
+	union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+	union dlb2_sys_dir_qid_v r5 = { {0} };
+
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r0.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r1.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+		    r1.val);
+
+	r2.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+		    r2.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		r3.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+		r4.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+	}
+
+	r5.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+	queue->queue_configured = true;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_dir_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_dir_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = dlb2_get_domain_used_dir_pq(args->port_id,
+						   vdev_req,
+						   domain);
+
+		if (port == NULL || port->domain_id.phys_id !=
+				domain->id.phys_id || !port->port_configured) {
+			resp->status = DLB2_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the queue's port is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->port_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->port_id != -1)
+		queue = dlb2_get_domain_used_dir_pq(args->port_id,
+						    vdev_req,
+						    domain);
+	else
+		queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					   typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &queue->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs,
+			      &queue->domain_list);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot,
+					      struct dlb2_map_qid_args *args)
+{
+	union dlb2_lsp_cq2priov r0;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+	r0.field.v |= 1 << slot;
+	r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+	dlb2_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+					      struct dlb2_ldb_queue *queue,
+					      struct dlb2_cmd_response *resp)
+{
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB2_QUEUE_UNMAPPED;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+			  bool vdev_req,
+			  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter)
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+			      bool vdev_req,
+			      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+
+		DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+	}
+
+	return NULL;
+}
+
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+				    u32 domain_id,
+				    struct dlb2_map_qid_args *args,
+				    struct dlb2_cmd_response *resp,
+				    bool vdev_req,
+				    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	struct dlb2_ldb_queue *queue;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB2_QID_PRIORITIES) {
+		resp->status = DLB2_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_map_qid_args *args,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	DLB2_HW_DBG(hw, "\tPriority:  %d\n",
+		    args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	int ret, i, id;
+	u8 prio;
+
+	dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_map_qid_args(hw,
+				       domain_id,
+				       args,
+				       resp,
+				       vdev_req,
+				       vdev_id);
+	if (ret)
+		return ret;
+
+	prio = args->priority;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	if (port->num_pending_removals)
+		dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].pending_priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+		if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb2_qid_map_state st;
+
+			if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+				DLB2_HW_ERR(hw,
+					    "[%s():%d] Internal error: port slot tracking failed\n",
+					    __func__, __LINE__);
+				return -EFAULT;
+			}
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb2_port_slot_state_transition(hw, port, queue,
+							      i, st);
+			if (ret)
+				return ret;
+
+			DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 2a089f6..a010e25 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -651,3 +651,13 @@ dlb2_pf_create_dir_port(struct dlb2_hw *hw,
 				       NOT_VF_REQ,
 				       PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_dir_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 2a82b8f..37be4b1 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -406,6 +406,54 @@ dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_dir_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_dir_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_dir_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
+		struct dlb2_map_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_map_qid(&dlb2_dev->hw,
+			      handle->domain_id,
+			      cfg,
+			      &response,
+			      false,
+			      0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -419,7 +467,9 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
 	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
+	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v7 15/23] event/dlb2: add port unlink and port unlinks in progress

[Timothy McDaniel]

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 163 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 283 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  51 ++++++
 5 files changed, 509 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 5068d6d..c221a61 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1797,6 +1797,166 @@ dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
 	return i;
 }
 
+static int16_t
+dlb2_hw_unmap_ldb_qid_from_port(struct dlb2_hw_dev *handle,
+				uint32_t qm_port_id,
+				uint16_t qm_qid)
+{
+	struct dlb2_unmap_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+
+	ret = dlb2_iface_unmap_qid(handle, &cfg);
+	if (ret < 0)
+		DLB2_LOG_ERR("dlb2: unmap qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_detach_ldb(struct dlb2_eventdev *dlb2,
+			    struct dlb2_eventdev_port *ev_port,
+			    struct dlb2_eventdev_queue *ev_queue)
+{
+	int ret, i;
+
+	/* Don't unlink until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid &&
+		    ev_port->link[i].queue_id == ev_queue->id)
+			break; /* found */
+	}
+
+	/* This is expected with eventdev API!
+	 * It blindly attemmpts to unmap all queues.
+	 */
+	if (i == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_LOG_DBG("dlb2: ignoring LB QID %d not mapped for qm_port %d.\n",
+			     ev_queue->qm_queue.id,
+			     ev_port->qm_port.id);
+		return 0;
+	}
+
+	ret = dlb2_hw_unmap_ldb_qid_from_port(&dlb2->qm_instance,
+					      ev_port->qm_port.id,
+					      ev_queue->qm_queue.id);
+	if (!ret)
+		ev_port->link[i].mapped = false;
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
+			  uint8_t queues[], uint16_t nb_unlinks)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (queues == NULL || nb_unlinks == 0) {
+		DLB2_LOG_DBG("dlb2: queues is NULL or nb_unlinks is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	/* FIXME: How to handle unlink on a directed port? */
+	if (ev_port->qm_port.is_directed) {
+		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
+			     ev_port->id);
+		rte_errno = 0;
+		return nb_unlinks; /* as if success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	for (i = 0; i < nb_unlinks; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		int ret, j;
+
+		if (queues[i] >= dlb2->num_queues) {
+			DLB2_LOG_ERR("dlb2: invalid queue id %d\n", queues[i]);
+			rte_errno = -EINVAL;
+			return i; /* return index of offending queue */
+		}
+
+		ev_queue = &dlb2->ev_queues[queues[i]];
+
+		/* Does a link exist? */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].queue_id == queues[i] &&
+			    ev_port->link[j].valid)
+				break;
+
+		if (j == DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+			continue;
+
+		ret = dlb2_event_queue_detach_ldb(dlb2, ev_port, ev_queue);
+		if (ret) {
+			DLB2_LOG_ERR("unlink err=%d for port %d queue %d\n",
+				     ret, ev_port->id, queues[i]);
+			rte_errno = -ENOENT;
+			return i; /* return index of offending queue */
+		}
+
+		ev_port->link[j].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+
+	return nb_unlinks;
+}
+
+static int
+dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
+				       void *event_port)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	struct dlb2_pending_port_unmaps_args cfg;
+	int ret;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	cfg.port_id = ev_port->qm_port.id;
+	dlb2 = ev_port->dlb2;
+	handle = &dlb2->qm_instance;
+	ret = dlb2_iface_pending_port_unmaps(handle, &cfg);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: num_unlinks_in_progress ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1809,6 +1969,9 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
 		.port_link        = dlb2_eventdev_port_link,
+		.port_unlink      = dlb2_eventdev_port_unlink,
+		.port_unlinks_in_progress =
+				    dlb2_eventdev_port_unlinks_in_progress,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 7d1ba73..bd241f3 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -57,3 +57,9 @@ int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 			  struct dlb2_map_qid_args *cfg);
+
+int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+			    struct dlb2_unmap_qid_args *cfg);
+
+int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4eddd14..e892036 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -56,4 +56,10 @@ extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 				 struct dlb2_map_qid_args *cfg);
+
+extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+				   struct dlb2_unmap_qid_args *cfg);
+
+extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 14ad262..4a3d96d 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5528,3 +5528,286 @@ int dlb2_hw_map_qid(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_unmap_qid_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+		DLB2_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			    hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+				      u32 domain_id,
+				      struct dlb2_unmap_qid_args *args,
+				      struct dlb2_cmd_response *resp,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	struct dlb2_ldb_port *port;
+	int slot;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			    __func__, args->qid);
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		return 0;
+
+	resp->status = DLB2_ST_INVALID_QID;
+	return -EINVAL;
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	bool unmap_complete;
+	int i, ret, id;
+
+	dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_unmap_qid_args(hw,
+					 domain_id,
+					 args,
+					 resp,
+					 vdev_req,
+					 vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+		st = DLB2_QUEUE_UNMAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		st = DLB2_QUEUE_UNMAP_IN_PROG;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB2_QUEUE_MAPPED;
+	if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: no available CQ slots\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asynchronous procedure. It requires
+	 * stopping the DLB2 from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb2_ldb_port_cq_disable(hw, port);
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete && !os_worker_active(hw))
+		os_schedule_work(hw);
+
+unmap_qid_done:
+	resp->status = 0;
+
+	return 0;
+}
+
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+				  struct dlb2_pending_port_unmaps_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+
+	dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 37be4b1..33ea73e 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -454,6 +454,55 @@ dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw_dev *handle,
+		  struct dlb2_unmap_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_unmap_qid(&dlb2_dev->hw,
+				handle->domain_id,
+				cfg,
+				&response,
+				false,
+				0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
+			    struct dlb2_pending_port_unmaps_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_pending_port_unmaps(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -470,6 +519,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
+	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v7 16/23] event/dlb2: add eventdev start

[Timothy McDaniel]

Add support for the eventdev start entry point.
We delay initializing some resources until
eventdev start, since the number of linked queues can be
used to determine if we are dealing with a ldb or dir resource.
If this is a device restart, then the previous configuration
will be reapplied.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 133 +++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   3 +
 drivers/event/dlb2/dlb2_iface.h            |   3 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 123 ++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +++
 drivers/event/dlb2/pf/dlb2_pf.c            |  25 ++++++
 6 files changed, 297 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index c221a61..051836d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1957,6 +1957,138 @@ dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
 	return cfg.response.id;
 }
 
+static int
+dlb2_eventdev_reapply_configuration(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret, i;
+
+	/* If an event queue or port was previously configured, but hasn't been
+	 * reconfigured, reapply its original configuration.
+	 */
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+
+		ev_queue = &dlb2->ev_queues[i];
+
+		if (ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_queue_setup(dev, i, &ev_queue->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure queue %d", i);
+			return ret;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+
+		if (ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_port_setup(dev, i, &ev_port->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure ev_port %d",
+				     i);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_apply_port_links(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int i;
+
+	/* Perform requested port->queue links */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+		int j;
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			struct dlb2_eventdev_queue *ev_queue;
+			uint8_t prio, queue_id;
+
+			if (!ev_port->link[j].valid)
+				continue;
+
+			prio = ev_port->link[j].priority;
+			queue_id = ev_port->link[j].queue_id;
+
+			if (dlb2_validate_port_link(ev_port, queue_id, true, j))
+				return -EINVAL;
+
+			ev_queue = &dlb2->ev_queues[queue_id];
+
+			if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_start(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_start_domain_args cfg;
+	int ret, i;
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_ERR("bad state %d for dev_start\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return -EINVAL;
+	}
+	dlb2->run_state = DLB2_RUN_STATE_STARTING;
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	/* If the device was configured more than once, some event ports and/or
+	 * queues may need to be reconfigured.
+	 */
+	ret = dlb2_eventdev_reapply_configuration(dev);
+	if (ret)
+		return ret;
+
+	/* The DLB PMD delays port links until the device is started. */
+	ret = dlb2_eventdev_apply_port_links(dev);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		if (!dlb2->ev_ports[i].setup_done) {
+			DLB2_LOG_ERR("dlb2: port %d not setup", i);
+			return -ESTALE;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0) {
+			DLB2_LOG_ERR("dlb2: queue %d is not linked", i);
+			return -ENOLINK;
+		}
+	}
+
+	ret = dlb2_iface_sched_domain_start(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: sched_domain_start ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STARTED;
+	DLB2_LOG_DBG("dlb2: sched_domain_start completed OK\n");
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1964,6 +2096,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.dev_start        = dlb2_eventdev_start,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index bd241f3..a86191d 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -63,3 +63,6 @@ int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				     struct dlb2_start_domain_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index e892036..7ead374 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -62,4 +62,7 @@ extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				struct dlb2_start_domain_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 4a3d96d..c14a2f3 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5811,3 +5811,126 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 struct dlb2_cmd_response *resp,
+					 bool vdev_req,
+					 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_start_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 start domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_hw_start_domain() - Lock the domain configuration
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @arg: User-provided arguments (unused, here for ioctl callback template).
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int
+dlb2_hw_start_domain(struct dlb2_hw *hw,
+		     u32 domain_id,
+		     __attribute((unused)) struct dlb2_start_domain_args *arg,
+		     struct dlb2_cmd_response *resp,
+		     bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_queue;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+	RTE_SET_USED(arg);
+	RTE_SET_USED(iter);
+
+	dlb2_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb2_verify_start_domain_args(hw,
+					    domain_id,
+					    resp,
+					    vdev_req,
+					    vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB2 will drop all
+	 * incoming traffic to those queues.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, ldb_queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r0.val);
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+	}
+
+	dlb2_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index a010e25..aee8336 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -661,3 +661,13 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_start_domain(struct dlb2_hw *hw,
+		     u32 id,
+		     struct dlb2_start_domain_args *args,
+		     struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_start_domain(hw, id, args, resp, NOT_VF_REQ,
+				    PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 33ea73e..fff2e57 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -503,6 +503,30 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
 
 	return ret;
 }
+
+static int
+dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
+			   struct dlb2_start_domain_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_start_domain(&dlb2_dev->hw,
+				   handle->domain_id,
+				   cfg,
+				   &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -520,6 +544,7 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v7 17/23] event/dlb2: add enqueue and its burst variants

[Timothy McDaniel]

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst | 118 +++++++++
 drivers/event/dlb2/dlb2.c     | 578 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 696 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index bbd6ac8..aa8bf01 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -163,6 +163,124 @@ Flow ID
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Hardware Credits
+~~~~~~~~~~~~~~~~
+
+DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
+event storage, with each unit of storage represented by a credit. A port spends
+a credit to enqueue an event, and hardware refills the ports with credits as the
+events are scheduled to ports. Refills come from credit pools, and each port is
+a member of a load-balanced credit pool and a directed credit pool. The
+load-balanced credits are used to enqueue to load-balanced queues, and directed
+credits are used for directed queues.
+
+A DLB2 eventdev contains one load-balanced and one directed credit pool. These
+pools' sizes are controlled by the nb_events_limit field in struct
+rte_event_dev_config. The load-balanced pool is sized to contain
+nb_events_limit credits, and the directed pool is sized to contain
+nb_events_limit/4 credits. The directed pool size can be overridden with the
+num_dir_credits vdev argument, like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,num_dir_credits=<value>
+
+This can be used if the default allocation is too low or too high for the
+specific application needs. The PMD also supports a vdev arg that limits the
+max_num_events reported by rte_event_dev_info_get():
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,max_num_events=<value>
+
+By default, max_num_events is reported as the total available load-balanced
+credits. If multiple DLB2-based applications are being used, it may be desirable
+to control how many load-balanced credits each application uses, particularly
+when application(s) are written to configure nb_events_limit equal to the
+reported max_num_events.
+
+Each port is a member of both credit pools. A port's credit allocation is
+defined by its low watermark, high watermark, and refill quanta. These three
+parameters are calculated by the dlb PMD like so:
+
+- The load-balanced high watermark is set to the port's enqueue_depth.
+  The directed high watermark is set to the minimum of the enqueue_depth and
+  the directed pool size divided by the total number of ports.
+- The refill quanta is set to half the high watermark.
+- The low watermark is set to the minimum of 16 and the refill quanta.
+
+When the eventdev is started, each port is pre-allocated a high watermark's
+worth of credits. For example, if an eventdev contains four ports with enqueue
+depths of 32 and a load-balanced credit pool size of 4096, each port will start
+with 32 load-balanced credits, and there will be 3968 credits available to
+replenish the ports. Thus, a single port is not capable of enqueueing up to the
+nb_events_limit (without any events being dequeued), since the other ports are
+retaining their initial credit allocation; in short, all ports must enqueue in
+order to reach the limit.
+
+If a port attempts to enqueue and has no credits available, the enqueue
+operation will fail and the application must retry the enqueue. Credits are
+replenished asynchronously by the DLB2 hardware.
+
+Software Credits
+~~~~~~~~~~~~~~~~
+
+The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
+credit scheme on top of the hardware credit scheme in order to comply with
+the per-port backpressure described in the eventdev API.
+
+The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
+credit when it enqueues to a queue, and credits are later replenished after the
+events are dequeued and released.
+
+In the software credit scheme, a credit is consumed when a new (.op =
+RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
+replenished when the event is released from the system (either explicitly with
+RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
+
+In this model, an event is "in the system" from its first enqueue into eventdev
+until it is last dequeued. If the event goes through multiple event queues, it
+is still considered "in the system" while a worker thread is processing it.
+
+A port will fail to enqueue if the number of events in the system exceeds its
+``new_event_threshold`` (specified at port setup time). A port will also fail
+to enqueue if it lacks enough hardware credits to enqueue; load-balanced
+credits are used to enqueue to a load-balanced queue, and directed credits are
+used to enqueue to a directed queue.
+
+The out-of-credit situations are typically transient, and an eventdev
+application using the DLB2 ought to retry its enqueues if they fail.
+If enqueue fails, DLB2 PMD sets rte_errno as follows:
+
+- -ENOSPC: Credit exhaustion (either hardware or software)
+- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
+
+Depending on the pipeline the application has constructed, it's possible to
+enter a credit deadlock scenario wherein the worker thread lacks the credit
+to enqueue an event, and it must dequeue an event before it can recover the
+credit. If the worker thread retries its enqueue indefinitely, it will not
+make forward progress. Such deadlock is possible if the application has event
+"loops", in which an event in dequeued from queue A and later enqueued back to
+queue A.
+
+Due to this, workers should stop retrying after a time, release the events it
+is attempting to enqueue, and dequeue more events. It is important that the
+worker release the events and don't simply set them aside to retry the enqueue
+again later, because the port has limited history list size (by default, twice
+the port's dequeue_depth).
+
+Priority
+~~~~~~~~
+
+The DLB2 supports event priority and per-port queue service priority, as
+described in the eventdev header file. The DLB2 does not support 'global' event
+queue priority established at queue creation time.
+
+DLB2 supports 8 event and queue service priority levels. For both priority
+types, the PMD uses the upper three bits of the priority field to determine the
+DLB2 priority, discarding the 5 least significant bits. The 5 least significant
+event priority bits are not preserved when an event is enqueued.
+
 Reconfiguration
 ~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 051836d..7d2cf76 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2089,6 +2089,578 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
 	return 0;
 }
 
+static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
+	{
+		/* Load-balanced cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
+	},
+	{
+		/* Directed cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
+	},
+};
+
+static inline uint32_t
+dlb2_port_credits_get(struct dlb2_port *qm_port,
+		      enum dlb2_hw_queue_types type)
+{
+	uint32_t credits = *qm_port->credit_pool[type];
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	if (unlikely(credits < batch_size))
+		batch_size = credits;
+
+	if (likely(credits &&
+		   __atomic_compare_exchange_n(
+			qm_port->credit_pool[type],
+			&credits, credits - batch_size, false,
+			__ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
+		return batch_size;
+	else
+		return 0;
+}
+
+static inline void
+dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port)
+{
+	uint16_t quanta = ev_port->credit_update_quanta;
+
+	if (ev_port->inflight_credits >= quanta * 2) {
+		/* Replenish credits, saving one quanta for enqueues */
+		uint16_t val = ev_port->inflight_credits - quanta;
+
+		__atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits -= val;
+	}
+}
+
+static inline int
+dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
+						__ATOMIC_SEQ_CST);
+	const int num = 1;
+
+	if (unlikely(ev_port->inflight_max < sw_inflights)) {
+		DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
+		rte_errno = -ENOSPC;
+		return 1;
+	}
+
+	if (ev_port->inflight_credits < num) {
+		/* check if event enqueue brings ev_port over max threshold */
+		uint32_t credit_update_quanta = ev_port->credit_update_quanta;
+
+		if (sw_inflights + credit_update_quanta >
+				dlb2->new_event_limit) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_new_event_limit,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+
+		__atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
+				   __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits += (credit_update_quanta);
+
+		if (ev_port->inflight_credits < num) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_inflight_credits,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_ldb_credits == 0)) {
+		qm_port->cached_ldb_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_LDB_QUEUE);
+		if (unlikely(qm_port->cached_ldb_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
+			1);
+			DLB2_LOG_DBG("ldb credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_dir_credits == 0)) {
+		qm_port->cached_dir_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_DIR_QUEUE);
+		if (unlikely(qm_port->cached_dir_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
+			1);
+			DLB2_LOG_DBG("dir credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static __rte_always_inline void
+dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
+	      struct process_local_port_data *port_data)
+{
+	dlb2_movdir64b(port_data->pp_addr, qe4);
+}
+
+static inline int
+dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_cq_pop_qe *qe;
+
+	RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
+
+	qe = qm_port->consume_qe;
+
+	qe->tokens = num - 1;
+
+	/* No store fence needed since no pointer is being sent, and CQ token
+	 * pops can be safely reordered with other HCWs.
+	 */
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	dlb2_movntdq_single(port_data->pp_addr, qe);
+
+	DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
+
+	qm_port->owed_tokens = 0;
+
+	return 0;
+}
+
+static inline void
+dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
+		   bool do_sfence,
+		   struct process_local_port_data *port_data)
+{
+	/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
+	 * application writes complete before enqueueing the QE.
+	 */
+	if (do_sfence)
+		rte_wmb();
+
+	dlb2_pp_write(qm_port->qe4, port_data);
+}
+
+static inline void
+dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
+{
+	struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
+	int num = qm_port->owed_tokens;
+
+	qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
+	qe[idx].tokens = num - 1;
+
+	qm_port->owed_tokens = 0;
+}
+
+static inline void
+dlb2_event_build_hcws(struct dlb2_port *qm_port,
+		      const struct rte_event ev[],
+		      int num,
+		      uint8_t *sched_type,
+		      uint8_t *queue_id)
+{
+	struct dlb2_enqueue_qe *qe;
+	uint16_t sched_word[4];
+	__m128i sse_qe[2];
+	int i;
+
+	qe = qm_port->qe4;
+
+	sse_qe[0] = _mm_setzero_si128();
+	sse_qe[1] = _mm_setzero_si128();
+
+	switch (num) {
+	case 4:
+		/* Construct the metadata portion of two HCWs in one 128b SSE
+		 * register. HCW metadata is constructed in the SSE registers
+		 * like so:
+		 * sse_qe[0][63:0]:   qe[0]'s metadata
+		 * sse_qe[0][127:64]: qe[1]'s metadata
+		 * sse_qe[1][63:0]:   qe[2]'s metadata
+		 * sse_qe[1][127:64]: qe[3]'s metadata
+		 */
+
+		/* Convert the event operation into a command byte and store it
+		 * in the metadata:
+		 * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
+		 * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
+		 * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
+		 * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
+		 */
+#define DLB2_QE_CMD_BYTE 7
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[0].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[1].op],
+				DLB2_QE_CMD_BYTE + 8);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[2].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[3].op],
+				DLB2_QE_CMD_BYTE + 8);
+
+		/* Store priority, scheduling type, and queue ID in the sched
+		 * word array because these values are re-used when the
+		 * destination is a directed queue.
+		 */
+		sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
+				sched_type[0] << 8 |
+				queue_id[0];
+		sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
+				sched_type[1] << 8 |
+				queue_id[1];
+		sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
+				sched_type[2] << 8 |
+				queue_id[2];
+		sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
+				sched_type[3] << 8 |
+				queue_id[3];
+
+		/* Store the event priority, scheduling type, and queue ID in
+		 * the metadata:
+		 * sse_qe[0][31:16] = sched_word[0]
+		 * sse_qe[0][95:80] = sched_word[1]
+		 * sse_qe[1][31:16] = sched_word[2]
+		 * sse_qe[1][95:80] = sched_word[3]
+		 */
+#define DLB2_QE_QID_SCHED_WORD 1
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[0],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[1],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[2],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[3],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+
+		/* If the destination is a load-balanced queue, store the lock
+		 * ID. If it is a directed queue, DLB places this field in
+		 * bytes 10-11 of the received QE, so we format it accordingly:
+		 * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
+		 * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
+		 * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
+		 * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
+		 */
+#define DLB2_QE_LOCK_ID_WORD 2
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[0] == DLB2_SCHED_DIRECTED) ?
+					sched_word[0] : ev[0].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[1] == DLB2_SCHED_DIRECTED) ?
+					sched_word[1] : ev[1].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[2] == DLB2_SCHED_DIRECTED) ?
+					sched_word[2] : ev[2].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[3] == DLB2_SCHED_DIRECTED) ?
+					sched_word[3] : ev[3].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+
+		/* Store the event type and sub event type in the metadata:
+		 * sse_qe[0][15:0]  = flow_id[0]
+		 * sse_qe[0][79:64] = flow_id[1]
+		 * sse_qe[1][15:0]  = flow_id[2]
+		 * sse_qe[1][79:64] = flow_id[3]
+		 */
+#define DLB2_QE_EV_TYPE_WORD 0
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[0].sub_event_type << 8 |
+						ev[0].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[1].sub_event_type << 8 |
+						ev[1].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[2].sub_event_type << 8 |
+						ev[2].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[3].sub_event_type << 8 |
+						ev[3].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+
+		/* Store the metadata to memory (use the double-precision
+		 * _mm_storeh_pd because there is no integer function for
+		 * storing the upper 64b):
+		 * qe[0] metadata = sse_qe[0][63:0]
+		 * qe[1] metadata = sse_qe[0][127:64]
+		 * qe[2] metadata = sse_qe[1][63:0]
+		 * qe[3] metadata = sse_qe[1][127:64]
+		 */
+		_mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
+		_mm_storeh_pd((double *)&qe[1].u.opaque_data,
+			      (__m128d)sse_qe[0]);
+		_mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
+		_mm_storeh_pd((double *)&qe[3].u.opaque_data,
+			      (__m128d)sse_qe[1]);
+
+		qe[0].data = ev[0].u64;
+		qe[1].data = ev[1].u64;
+		qe[2].data = ev[2].u64;
+		qe[3].data = ev[3].u64;
+
+		break;
+	case 3:
+	case 2:
+	case 1:
+		/* At least one QE will be valid, so only zero out three */
+		qe[1].cmd_byte = 0;
+		qe[2].cmd_byte = 0;
+		qe[3].cmd_byte = 0;
+
+		for (i = 0; i < num; i++) {
+			qe[i].cmd_byte =
+				cmd_byte_map[qm_port->is_directed][ev[i].op];
+			qe[i].sched_type = sched_type[i];
+			qe[i].data = ev[i].u64;
+			qe[i].qid = queue_id[i];
+			qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
+			qe[i].lock_id = ev[i].flow_id;
+			if (sched_type[i] == DLB2_SCHED_DIRECTED) {
+				struct dlb2_msg_info *info =
+					(struct dlb2_msg_info *)&qe[i].lock_id;
+
+				info->qid = queue_id[i];
+				info->sched_type = DLB2_SCHED_DIRECTED;
+				info->priority = qe[i].priority;
+			}
+			qe[i].u.event_type.major = ev[i].event_type;
+			qe[i].u.event_type.sub = ev[i].sub_event_type;
+		}
+		break;
+	}
+}
+
+static inline int
+dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
+			struct dlb2_port *qm_port,
+			const struct rte_event ev[],
+			uint8_t *sched_type,
+			uint8_t *queue_id)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	uint16_t *cached_credits = NULL;
+	struct dlb2_queue *qm_queue;
+
+	ev_queue = &dlb2->ev_queues[ev->queue_id];
+	qm_queue = &ev_queue->qm_queue;
+	*queue_id = qm_queue->id;
+
+	/* Ignore sched_type and hardware credits on release events */
+	if (ev->op == RTE_EVENT_OP_RELEASE)
+		goto op_check;
+
+	if (!qm_queue->is_directed) {
+		/* Load balanced destination queue */
+
+		if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_ldb_credits;
+
+		switch (ev->sched_type) {
+		case RTE_SCHED_TYPE_ORDERED:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
+			if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
+				DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
+					     *queue_id);
+				rte_errno = -EINVAL;
+				return 1;
+			}
+			*sched_type = DLB2_SCHED_ORDERED;
+			break;
+		case RTE_SCHED_TYPE_ATOMIC:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
+			*sched_type = DLB2_SCHED_ATOMIC;
+			break;
+		case RTE_SCHED_TYPE_PARALLEL:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
+			if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
+				*sched_type = DLB2_SCHED_ORDERED;
+			else
+				*sched_type = DLB2_SCHED_UNORDERED;
+			break;
+		default:
+			DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
+			DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
+			rte_errno = -EINVAL;
+			return 1;
+		}
+	} else {
+		/* Directed destination queue */
+
+		if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_dir_credits;
+
+		DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
+
+		*sched_type = DLB2_SCHED_DIRECTED;
+	}
+
+op_check:
+	switch (ev->op) {
+	case RTE_EVENT_OP_NEW:
+		/* Check that a sw credit is available */
+		if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		ev_port->inflight_credits--;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_FORWARD:
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_RELEASE:
+		ev_port->inflight_credits++;
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+
+		/* Replenish s/w credits if enough are cached */
+		dlb2_replenish_sw_credits(dlb2, ev_port);
+		break;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+	if (ev->op != RTE_EVENT_OP_RELEASE) {
+		DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
+		DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
+	}
+#endif
+
+	return 0;
+}
+
+static inline uint16_t
+dlb2_event_enqueue_burst(void *event_port,
+			 const struct rte_event events[],
+			 uint16_t num)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct process_local_port_data *port_data;
+	int i, cnt;
+
+	RTE_ASSERT(ev_port->enq_configured);
+	RTE_ASSERT(events != NULL);
+
+	cnt = 0;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
+		int j = 0;
+
+		for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
+			const struct rte_event *ev = &events[i + j];
+
+			if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
+						    &sched_types[j],
+						    &queue_ids[j]))
+				break;
+		}
+
+		if (j == 0)
+			break;
+
+		dlb2_event_build_hcws(qm_port, &events[i], j,
+				      sched_types, queue_ids);
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+
+		if (j < DLB2_NUM_QES_PER_CACHE_LINE)
+			break;
+	}
+
+	return cnt;
+}
+
+static inline uint16_t
+dlb2_event_enqueue(void *event_port,
+		   const struct rte_event events[])
+{
+	return dlb2_event_enqueue_burst(event_port, events, 1);
+}
+
+static uint16_t
+dlb2_event_enqueue_new_burst(void *event_port,
+			     const struct rte_event events[],
+			     uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
+static uint16_t
+dlb2_event_enqueue_forward_burst(void *event_port,
+				 const struct rte_event events[],
+				 uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2112,7 +2684,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
 	};
 
+	/* Expose PMD's eventdev interface */
+
 	dev->dev_ops = &dlb2_eventdev_entry_ops;
+	dev->enqueue = dlb2_event_enqueue;
+	dev->enqueue_burst = dlb2_event_enqueue_burst;
+	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
+	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v7 18/23] event/dlb2: add dequeue and its burst variants

[Timothy McDaniel]

Add support for dequeue, dequeue_burst, ...

DLB2 does not currently support interrupts, but instead use
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB2 supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst |  21 ++
 drivers/event/dlb2/dlb2.c     | 757 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 778 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index aa8bf01..b9f57fd 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -314,6 +314,27 @@ The PMD does not support the following configuration sequences:
 This sequence is not supported because the event device must be reconfigured
 before its ports or queues can be.
 
+Deferred Scheduling
+~~~~~~~~~~~~~~~~~~~
+
+The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
+dequeued event before returning from rte_event_dequeue_burst(). This frees the
+corresponding entries in the CQ, which enables the DLB2 to schedule more events
+to it.
+
+To support applications seeking finer-grained scheduling control -- for example
+deferring scheduling to get the best possible priority scheduling and
+load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
+the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
+call. This mode only applies to load-balanced event ports with dequeue depth of
+1.
+
+To enable deferred scheduling, use the defer_sched vdev argument like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,defer_sched=on
+
 Atomic Inflights Allocation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7d2cf76..b9082bc 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -26,6 +26,7 @@
 #include <rte_log.h>
 #include <rte_malloc.h>
 #include <rte_mbuf.h>
+#include <rte_power_intrinsics.h>
 #include <rte_prefetch.h>
 #include <rte_ring.h>
 #include <rte_string_fns.h>
@@ -2661,9 +2662,756 @@ dlb2_event_enqueue_forward_burst(void *event_port,
 	return dlb2_event_enqueue_burst(event_port, events, num);
 }
 
+static inline void
+dlb2_port_credits_inc(struct dlb2_port *qm_port, int num)
+{
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	/* increment port credits, and return to pool if exceeds threshold */
+	if (!qm_port->is_directed) {
+		qm_port->cached_ldb_credits += num;
+		if (qm_port->cached_ldb_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_LDB_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_ldb_credits -= batch_size;
+		}
+	} else {
+		qm_port->cached_dir_credits += num;
+		if (qm_port->cached_dir_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_DIR_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_dir_credits -= batch_size;
+		}
+	}
+}
+
+static inline int
+dlb2_dequeue_wait(struct dlb2_eventdev *dlb2,
+		  struct dlb2_eventdev_port *ev_port,
+		  struct dlb2_port *qm_port,
+		  uint64_t timeout,
+		  uint64_t start_ticks)
+{
+	struct process_local_port_data *port_data;
+	uint64_t elapsed_ticks;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	elapsed_ticks = rte_get_timer_cycles() - start_ticks;
+
+	/* Wait/poll time expired */
+	if (elapsed_ticks >= timeout) {
+		return 1;
+	} else if (dlb2->umwait_allowed) {
+		volatile struct dlb2_dequeue_qe *cq_base;
+		union {
+			uint64_t raw_qe[2];
+			struct dlb2_dequeue_qe qe;
+		} qe_mask;
+		uint64_t expected_value;
+		volatile uint64_t *monitor_addr;
+
+		qe_mask.qe.cq_gen = 1; /* set mask */
+
+		cq_base = port_data->cq_base;
+		monitor_addr = (volatile uint64_t *)(volatile void *)
+			&cq_base[qm_port->cq_idx];
+		monitor_addr++; /* cq_gen bit is in second 64bit location */
+
+		if (qm_port->gen_bit)
+			expected_value = qe_mask.raw_qe[1];
+		else
+			expected_value = 0;
+
+		rte_power_monitor(monitor_addr, expected_value,
+				  qe_mask.raw_qe[1], timeout + start_ticks,
+				  sizeof(uint64_t));
+
+		DLB2_INC_STAT(ev_port->stats.traffic.rx_umonitor_umwait, 1);
+	} else {
+		uint64_t poll_interval = RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL;
+		uint64_t curr_ticks = rte_get_timer_cycles();
+		uint64_t init_ticks = curr_ticks;
+
+		while ((curr_ticks - start_ticks < timeout) &&
+		       (curr_ticks - init_ticks < poll_interval))
+			curr_ticks = rte_get_timer_cycles();
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
+			 struct dlb2_port *qm_port,
+			 struct rte_event *events,
+			 struct dlb2_dequeue_qe *qes,
+			 int cnt)
+{
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	int i, num, evq_id;
+
+	for (i = 0, num = 0; i < cnt; i++) {
+		struct dlb2_dequeue_qe *qe = &qes[i];
+		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
+			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+			[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+		};
+
+		/* Fill in event information.
+		 * Note that flow_id must be embedded in the data by
+		 * the app, such as the mbuf RSS hash field if the data
+		 * buffer is a mbuf.
+		 */
+		if (unlikely(qe->error)) {
+			DLB2_LOG_ERR("QE error bit ON\n");
+			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
+			dlb2_consume_qe_immediate(qm_port, 1);
+			continue; /* Ignore */
+		}
+
+		events[num].u64 = qe->data;
+		events[num].flow_id = qe->flow_id;
+		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
+		events[num].event_type = qe->u.event_type.major;
+		events[num].sub_event_type = qe->u.event_type.sub;
+		events[num].sched_type = sched_type_map[qe->sched_type];
+		events[num].impl_opaque = qe->qid_depth;
+
+		/* qid not preserved for directed queues */
+		if (qm_port->is_directed)
+			evq_id = ev_port->link[0].queue_id;
+		else
+			evq_id = qid_mappings[qe->qid];
+
+		events[num].queue_id = evq_id;
+		DLB2_INC_STAT(
+			ev_port->stats.queue[evq_id].qid_depth[qe->qid_depth],
+			1);
+		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type], 1);
+		num++;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num);
+
+	return num;
+}
+
+static inline int
+dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
+			      struct dlb2_port *qm_port,
+			      struct rte_event *events,
+			      struct dlb2_dequeue_qe *qes)
+{
+	int sched_type_map[] = {
+		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+	};
+	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	__m128i sse_evt[2];
+
+	/* In the unlikely case that any of the QE error bits are set, process
+	 * them one at a time.
+	 */
+	if (unlikely(qes[0].error || qes[1].error ||
+		     qes[2].error || qes[3].error))
+		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
+						 qes, num_events);
+
+	events[0].u64 = qes[0].data;
+	events[1].u64 = qes[1].data;
+	events[2].u64 = qes[2].data;
+	events[3].u64 = qes[3].data;
+
+	/* Construct the metadata portion of two struct rte_events
+	 * in one 128b SSE register. Event metadata is constructed in the SSE
+	 * registers like so:
+	 * sse_evt[0][63:0]:   event[0]'s metadata
+	 * sse_evt[0][127:64]: event[1]'s metadata
+	 * sse_evt[1][63:0]:   event[2]'s metadata
+	 * sse_evt[1][127:64]: event[3]'s metadata
+	 */
+	sse_evt[0] = _mm_setzero_si128();
+	sse_evt[1] = _mm_setzero_si128();
+
+	/* Convert the hardware queue ID to an event queue ID and store it in
+	 * the metadata:
+	 * sse_evt[0][47:40]   = qid_mappings[qes[0].qid]
+	 * sse_evt[0][111:104] = qid_mappings[qes[1].qid]
+	 * sse_evt[1][47:40]   = qid_mappings[qes[2].qid]
+	 * sse_evt[1][111:104] = qid_mappings[qes[3].qid]
+	 */
+#define DLB_EVENT_QUEUE_ID_BYTE 5
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[0].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[1].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[2].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[3].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+
+	/* Convert the hardware priority to an event priority and store it in
+	 * the metadata, while also returning the queue depth status
+	 * value captured by the hardware, storing it in impl_opaque, which can
+	 * be read by the application but not modified
+	 * sse_evt[0][55:48]   = DLB2_TO_EV_PRIO(qes[0].priority)
+	 * sse_evt[0][63:56]   = qes[0].qid_depth
+	 * sse_evt[0][119:112] = DLB2_TO_EV_PRIO(qes[1].priority)
+	 * sse_evt[0][127:120] = qes[1].qid_depth
+	 * sse_evt[1][55:48]   = DLB2_TO_EV_PRIO(qes[2].priority)
+	 * sse_evt[1][63:56]   = qes[2].qid_depth
+	 * sse_evt[1][119:112] = DLB2_TO_EV_PRIO(qes[3].priority)
+	 * sse_evt[1][127:120] = qes[3].qid_depth
+	 */
+#define DLB_EVENT_PRIO_IMPL_OPAQUE_WORD 3
+#define DLB_BYTE_SHIFT 8
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[0].priority) |
+			(qes[0].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[1].priority) |
+			(qes[1].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[2].priority) |
+			(qes[2].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[3].priority) |
+			(qes[3].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+
+	/* Write the event type, sub event type, and flow_id to the event
+	 * metadata.
+	 * sse_evt[0][31:0]   = qes[0].flow_id |
+	 *			qes[0].u.event_type.major << 28 |
+	 *			qes[0].u.event_type.sub << 20;
+	 * sse_evt[0][95:64]  = qes[1].flow_id |
+	 *			qes[1].u.event_type.major << 28 |
+	 *			qes[1].u.event_type.sub << 20;
+	 * sse_evt[1][31:0]   = qes[2].flow_id |
+	 *			qes[2].u.event_type.major << 28 |
+	 *			qes[2].u.event_type.sub << 20;
+	 * sse_evt[1][95:64]  = qes[3].flow_id |
+	 *			qes[3].u.event_type.major << 28 |
+	 *			qes[3].u.event_type.sub << 20;
+	 */
+#define DLB_EVENT_EV_TYPE_DW 0
+#define DLB_EVENT_EV_TYPE_SHIFT 28
+#define DLB_EVENT_SUB_EV_TYPE_SHIFT 20
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[0].flow_id |
+			qes[0].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[0].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[1].flow_id |
+			qes[1].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[1].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[2].flow_id |
+			qes[2].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[2].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[3].flow_id |
+			qes[3].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT  |
+			qes[3].u.event_type.sub << DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+
+	/* Write the sched type to the event metadata. 'op' and 'rsvd' are not
+	 * set:
+	 * sse_evt[0][39:32]  = sched_type_map[qes[0].sched_type] << 6
+	 * sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6
+	 * sse_evt[1][39:32]  = sched_type_map[qes[2].sched_type] << 6
+	 * sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6
+	 */
+#define DLB_EVENT_SCHED_TYPE_BYTE 4
+#define DLB_EVENT_SCHED_TYPE_SHIFT 6
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[0].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[1].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[2].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[3].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+
+	/* Store the metadata to the event (use the double-precision
+	 * _mm_storeh_pd because there is no integer function for storing the
+	 * upper 64b):
+	 * events[0].event = sse_evt[0][63:0]
+	 * events[1].event = sse_evt[0][127:64]
+	 * events[2].event = sse_evt[1][63:0]
+	 * events[3].event = sse_evt[1][127:64]
+	 */
+	_mm_storel_epi64((__m128i *)&events[0].event, sse_evt[0]);
+	_mm_storeh_pd((double *)&events[1].event, (__m128d) sse_evt[0]);
+	_mm_storel_epi64((__m128i *)&events[2].event, sse_evt[1]);
+	_mm_storeh_pd((double *)&events[3].event, (__m128d) sse_evt[1]);
+
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[0].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[1].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[2].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[3].sched_type], 1);
+
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[0].queue_id].
+			qid_depth[qes[0].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[1].queue_id].
+			qid_depth[qes[1].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[2].queue_id].
+			qid_depth[qes[2].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[3].queue_id].
+			qid_depth[qes[3].qid_depth],
+		1);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num_events);
+
+	return num_events;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe_sparse(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe)
+{
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	uint8_t xor_mask[2] = {0x0F, 0x00};
+	const uint8_t and_mask = 0x0F;
+	__m128i *qes = (__m128i *)qe;
+	uint8_t gen_bits, gen_bit;
+	uintptr_t addr[4];
+	uint16_t idx;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+
+	idx = qm_port->cq_idx;
+
+	/* Load the next 4 QEs */
+	addr[0] = (uintptr_t)&cq_addr[idx];
+	addr[1] = (uintptr_t)&cq_addr[(idx +  4) & qm_port->cq_depth_mask];
+	addr[2] = (uintptr_t)&cq_addr[(idx +  8) & qm_port->cq_depth_mask];
+	addr[3] = (uintptr_t)&cq_addr[(idx + 12) & qm_port->cq_depth_mask];
+
+	/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */
+	rte_prefetch0(&cq_addr[(idx + 16) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 20) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 24) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 28) & qm_port->cq_depth_mask]);
+
+	/* Correct the xor_mask for wrap-around QEs */
+	gen_bit = qm_port->gen_bit;
+	xor_mask[gen_bit] ^= !!((idx +  4) > qm_port->cq_depth_mask) << 1;
+	xor_mask[gen_bit] ^= !!((idx +  8) > qm_port->cq_depth_mask) << 2;
+	xor_mask[gen_bit] ^= !!((idx + 12) > qm_port->cq_depth_mask) << 3;
+
+	/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is
+	 * valid, then QEs[0:N-1] are too.
+	 */
+	qes[3] = _mm_load_si128((__m128i *)(void *)addr[3]);
+	rte_compiler_barrier();
+	qes[2] = _mm_load_si128((__m128i *)(void *)addr[2]);
+	rte_compiler_barrier();
+	qes[1] = _mm_load_si128((__m128i *)(void *)addr[1]);
+	rte_compiler_barrier();
+	qes[0] = _mm_load_si128((__m128i *)(void *)addr[0]);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[gen_bit];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask;
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline void
+dlb2_inc_cq_idx(struct dlb2_port *qm_port, int cnt)
+{
+	uint16_t idx = qm_port->cq_idx_unmasked + cnt;
+
+	qm_port->cq_idx_unmasked = idx;
+	qm_port->cq_idx = idx & qm_port->cq_depth_mask;
+	qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) & 0x1;
+}
+
+static int
+dlb2_event_release(struct dlb2_eventdev *dlb2,
+		   uint8_t port_id,
+		   int n)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_eventdev_port *ev_port;
+	struct dlb2_port *qm_port;
+	int i, cnt;
+
+	if (port_id > dlb2->num_ports) {
+		DLB2_LOG_ERR("Invalid port id %d in dlb2-event_release\n",
+			     port_id);
+		rte_errno = -EINVAL;
+		return rte_errno;
+	}
+
+	ev_port = &dlb2->ev_ports[port_id];
+	qm_port = &ev_port->qm_port;
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	cnt = 0;
+
+	if (qm_port->is_directed) {
+		cnt = n;
+		goto sw_credit_update;
+	}
+
+	for (i = 0; i < n; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		int j;
+
+		/* Zero-out QEs */
+		qm_port->qe4[0].cmd_byte = 0;
+		qm_port->qe4[1].cmd_byte = 0;
+		qm_port->qe4[2].cmd_byte = 0;
+		qm_port->qe4[3].cmd_byte = 0;
+
+		for (j = 0; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++)
+			qm_port->qe4[j].cmd_byte = DLB2_COMP_CMD_BYTE;
+
+		qm_port->issued_releases += j;
+
+		if (j == 0)
+			break;
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+	}
+
+sw_credit_update:
+	/* each release returns one credit */
+	if (!ev_port->outstanding_releases) {
+		DLB2_LOG_ERR("Unrecoverable application error. Outstanding releases underflowed.\n");
+		rte_errno = -ENOTRECOVERABLE;
+		return rte_errno;
+	}
+
+	ev_port->outstanding_releases -= cnt;
+	ev_port->inflight_credits += cnt;
+
+	/* Replenish s/w credits if enough releases are performed */
+	dlb2_replenish_sw_credits(dlb2, ev_port);
+	return 0;
+}
+
+static inline int16_t
+dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
+		       struct dlb2_eventdev_port *ev_port,
+		       struct rte_event *events,
+		       uint16_t max_num,
+		       uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe_sparse(qm_port, qes);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail << 2);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							      qm_port,
+							      &events[num],
+							      &qes[0]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							 qm_port,
+							 &events[num],
+							 &qes[0],
+							 num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe,
+	     uint8_t *offset)
+{
+	uint8_t xor_mask[2][4] = { {0x0F, 0x0E, 0x0C, 0x08},
+				   {0x00, 0x01, 0x03, 0x07} };
+	uint8_t and_mask[4] = {0x0F, 0x0E, 0x0C, 0x08};
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	__m128i *qes = (__m128i *)qe;
+	uint64_t *cache_line_base;
+	uint8_t gen_bits;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	cq_addr = &cq_addr[qm_port->cq_idx];
+
+	cache_line_base = (void *)(((uintptr_t)cq_addr) & ~0x3F);
+	*offset = ((uintptr_t)cq_addr & 0x30) >> 4;
+
+	/* Load the next CQ cache line from memory. Pack these reads as tight
+	 * as possible to reduce the chance that DLB invalidates the line while
+	 * the CPU is reading it. Read the cache line backwards to ensure that
+	 * if QE[N] (N > 0) is valid, then QEs[0:N-1] are too.
+	 *
+	 * (Valid QEs start at &qe[offset])
+	 */
+	qes[3] = _mm_load_si128((__m128i *)&cache_line_base[6]);
+	qes[2] = _mm_load_si128((__m128i *)&cache_line_base[4]);
+	qes[1] = _mm_load_si128((__m128i *)&cache_line_base[2]);
+	qes[0] = _mm_load_si128((__m128i *)&cache_line_base[0]);
+
+	/* Evict the cache line ASAP */
+	dlb2_cldemote(cache_line_base);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[qm_port->gen_bit][*offset];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask[*offset];
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline int16_t
+dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
+		struct dlb2_eventdev_port *ev_port,
+		struct rte_event *events,
+		uint16_t max_num,
+		uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t offset;
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe(qm_port, qes, &offset);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							     qm_port,
+							     &events[num],
+							     &qes[offset]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							qm_port,
+							&events[num],
+							&qes[offset],
+							num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static uint16_t
+dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
+			 uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue(void *event_port, struct rte_event *ev, uint64_t wait)
+{
+	return dlb2_event_dequeue_burst(event_port, ev, 1, wait);
+}
+
+static uint16_t
+dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
+				uint16_t num, uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
+			  uint64_t wait)
+{
+	return dlb2_event_dequeue_burst_sparse(event_port, ev, 1, wait);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
+	struct dlb2_eventdev *dlb2;
+
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
@@ -2691,6 +3439,15 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	dev->enqueue_burst = dlb2_event_enqueue_burst;
 	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
 	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
+
+	dlb2 = dev->data->dev_private;
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE) {
+		dev->dequeue = dlb2_event_dequeue_sparse;
+		dev->dequeue_burst = dlb2_event_dequeue_burst_sparse;
+	} else {
+		dev->dequeue = dlb2_event_dequeue;
+		dev->dequeue_burst = dlb2_event_dequeue_burst;
+	}
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v7 19/23] event/dlb2: add eventdev stop and close

[Timothy McDaniel]

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 257 +++++++++++++++++++++++++++--
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c |  91 ++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  52 ++++++
 5 files changed, 396 insertions(+), 16 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index b9082bc..82108b6 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -71,21 +71,6 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
-/*
- * DUMMY - added so that xstats path will compile/link.
- * Will be replaced by real version in a subsequent
- * patch.
- */
-uint32_t
-dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
-		     struct dlb2_eventdev_queue *queue)
-{
-	RTE_SET_USED(dlb2);
-	RTE_SET_USED(queue);
-
-	return 0;
-}
-
 static void
 dlb2_free_qe_mem(struct dlb2_port *qm_port)
 {
@@ -1878,7 +1863,6 @@ dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
 		return 0; /* Ignore and return success */
 	}
 
-	/* FIXME: How to handle unlink on a directed port? */
 	if (ev_port->qm_port.is_directed) {
 		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
 			     ev_port->id);
@@ -3408,6 +3392,245 @@ dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
 }
 
 static void
+dlb2_flush_port(struct rte_eventdev *dev, int port_id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	eventdev_stop_flush_t flush;
+	struct rte_event ev;
+	uint8_t dev_id;
+	void *arg;
+	int i;
+
+	flush = dev->dev_ops->dev_stop_flush;
+	dev_id = dev->data->dev_id;
+	arg = dev->data->dev_stop_flush_arg;
+
+	while (rte_event_dequeue_burst(dev_id, port_id, &ev, 1, 0)) {
+		if (flush)
+			flush(dev_id, ev, arg);
+
+		if (dlb2->ev_ports[port_id].qm_port.is_directed)
+			continue;
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+	}
+
+	/* Enqueue any additional outstanding releases */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = dlb2->ev_ports[port_id].outstanding_releases; i > 0; i--)
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+}
+
+static uint32_t
+dlb2_get_ldb_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_ldb_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_ldb_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_ldb_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static uint32_t
+dlb2_get_dir_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_dir_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_dir_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_dir_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	if (queue->qm_queue.is_directed)
+		return dlb2_get_dir_queue_depth(dlb2, queue);
+	else
+		return dlb2_get_ldb_queue_depth(dlb2, queue);
+}
+
+static bool
+dlb2_queue_is_empty(struct dlb2_eventdev *dlb2,
+		    struct dlb2_eventdev_queue *queue)
+{
+	return dlb2_get_queue_depth(dlb2, queue) == 0;
+}
+
+static bool
+dlb2_linked_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0)
+			continue;
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static bool
+dlb2_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static void
+dlb2_drain(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_port *ev_port = NULL;
+	uint8_t dev_id;
+	int i;
+
+	dev_id = dev->data->dev_id;
+
+	while (!dlb2_linked_queues_empty(dlb2)) {
+		/* Flush all the ev_ports, which will drain all their connected
+		 * queues.
+		 */
+		for (i = 0; i < dlb2->num_ports; i++)
+			dlb2_flush_port(dev, i);
+	}
+
+	/* The queues are empty, but there may be events left in the ports. */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_flush_port(dev, i);
+
+	/* If the domain's queues are empty, we're done. */
+	if (dlb2_queues_empty(dlb2))
+		return;
+
+	/* Else, there must be at least one unlinked load-balanced queue.
+	 * Select a load-balanced port with which to drain the unlinked
+	 * queue(s).
+	 */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		if (!ev_port->qm_port.is_directed)
+			break;
+	}
+
+	if (i == dlb2->num_ports) {
+		DLB2_LOG_ERR("internal error: no LDB ev_ports\n");
+		return;
+	}
+
+	rte_errno = 0;
+	rte_event_port_unlink(dev_id, ev_port->id, NULL, 0);
+
+	if (rte_errno) {
+		DLB2_LOG_ERR("internal error: failed to unlink ev_port %d\n",
+			     ev_port->id);
+		return;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		uint8_t qid, prio;
+		int ret;
+
+		if (dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			continue;
+
+		qid = i;
+		prio = 0;
+
+		/* Link the ev_port to the queue */
+		ret = rte_event_port_link(dev_id, ev_port->id, &qid, &prio, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to link ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+
+		/* Flush the queue */
+		while (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			dlb2_flush_port(dev, ev_port->id);
+
+		/* Drain any extant events in the ev_port. */
+		dlb2_flush_port(dev, ev_port->id);
+
+		/* Unlink the ev_port from the queue */
+		ret = rte_event_port_unlink(dev_id, ev_port->id, &qid, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to unlink ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+	}
+}
+
+static void
+dlb2_eventdev_stop(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_DBG("Internal error: already stopped\n");
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	} else if (dlb2->run_state != DLB2_RUN_STATE_STARTED) {
+		DLB2_LOG_ERR("Internal error: bad state %d for dev_stop\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPING;
+
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	dlb2_drain(dev);
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPED;
+}
+
+static int
+dlb2_eventdev_close(struct rte_eventdev *dev)
+{
+	dlb2_hw_reset_sched_domain(dev, false);
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3417,6 +3640,8 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.dev_start        = dlb2_eventdev_start,
+		.dev_stop         = dlb2_eventdev_stop,
+		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a86191d..5471dd8 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -66,3 +66,9 @@ int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				     struct dlb2_start_domain_args *cfg);
+
+int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 7ead374..b508eb0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -65,4 +65,10 @@ extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				struct dlb2_start_domain_args *cfg);
+
+extern int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+extern int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index c14a2f3..ae5ef2f 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5934,3 +5934,94 @@ dlb2_hw_start_domain(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_dir_queue_depth(hw, queue);
+
+	return 0;
+}
+
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+
+	dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index fff2e57..632c4e0 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -527,6 +527,56 @@ dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_ldb_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_ldb_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_dir_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret = 0;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_dir_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -544,6 +594,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth;
+	dlb2_iface_get_dir_queue_depth = dlb2_pf_get_dir_queue_depth;
 	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v7 20/23] event/dlb2: add PMD's token pop public interface

[Timothy McDaniel]

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/api/doxy-api-index.md         |  1 +
 drivers/event/dlb2/dlb2.c         | 53 ++++++++++++++++++++++++----
 drivers/event/dlb2/dlb2_priv.h    |  3 ++
 drivers/event/dlb2/meson.build    |  5 ++-
 drivers/event/dlb2/rte_pmd_dlb2.c | 39 +++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2.h | 72 +++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/version.map    |  6 ++++
 7 files changed, 172 insertions(+), 7 deletions(-)
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index a9c12d1..00f13e2 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -52,6 +52,7 @@ The public API headers are grouped by topics:
   [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
   [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
+  [dlb2]	       (@ref rte_pmd_dlb2.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 82108b6..7a22231 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1197,7 +1197,7 @@ dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
 	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
 
 	qm_port->dequeue_depth = dequeue_depth;
-
+	qm_port->token_pop_thresh = dequeue_depth;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -1365,6 +1365,8 @@ dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
 
 	qm_port->dequeue_depth = dequeue_depth;
 
+	/* Directed ports are auto-pop, by default. */
+	qm_port->token_pop_mode = AUTO_POP;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -2612,6 +2614,14 @@ dlb2_event_enqueue_burst(void *event_port,
 		dlb2_event_build_hcws(qm_port, &events[i], j,
 				      sched_types, queue_ids);
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
@@ -2620,6 +2630,11 @@ dlb2_event_enqueue_burst(void *event_port,
 			break;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
 	return cnt;
 }
 
@@ -3097,11 +3112,25 @@ dlb2_event_release(struct dlb2_eventdev *dlb2,
 		if (j == 0)
 			break;
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
 sw_credit_update:
 	/* each release returns one credit */
 	if (!ev_port->outstanding_releases) {
@@ -3185,8 +3214,8 @@ dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3312,8 +3341,8 @@ dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3328,6 +3357,7 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 			 uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3343,6 +3373,9 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3362,6 +3395,7 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 				uint16_t num, uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3377,6 +3411,9 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3681,7 +3718,7 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    struct dlb2_devargs *dlb2_args)
 {
 	struct dlb2_eventdev *dlb2;
-	int err;
+	int err, i;
 
 	dlb2 = dev->data->dev_private;
 
@@ -3731,6 +3768,10 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Initialize each port's token pop mode */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
+		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 61567a6..b73cf3f 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -12,6 +12,7 @@
 #include <rte_config.h>
 #include "dlb2_user.h"
 #include "dlb2_log.h"
+#include "rte_pmd_dlb2.h"
 
 #ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
 #define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
@@ -290,6 +291,7 @@ struct dlb2_port {
 	bool gen_bit;
 	uint16_t dir_credits;
 	uint32_t dequeue_depth;
+	enum dlb2_token_pop_mode token_pop_mode;
 	union dlb2_port_config cfg;
 	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
 	uint16_t cached_ldb_credits;
@@ -298,6 +300,7 @@ struct dlb2_port {
 	bool int_armed;
 	uint16_t owed_tokens;
 	int16_t issued_releases;
+	int16_t token_pop_thresh;
 	int cq_depth;
 	uint16_t cq_idx;
 	uint16_t cq_idx_unmasked;
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index dc16c96..b734ff6 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,3 +1,4 @@
+
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
@@ -12,7 +13,9 @@ sources = files('dlb2.c',
 		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
-		'pf/base/dlb2_resource.c'
+		'pf/base/dlb2_resource.c',
+		'rte_pmd_dlb2.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
+install_headers('rte_pmd_dlb2.h')
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
new file mode 100644
index 0000000..b09b585
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "rte_pmd_dlb2.h"
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode)
+{
+	struct dlb2_eventdev *dlb2;
+	struct rte_eventdev *dev;
+
+	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+	dev = &rte_eventdevs[dev_id];
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (mode >= NUM_TOKEN_POP_MODES)
+		return -EINVAL;
+
+	/* The event device must be configured, but not yet started */
+	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
+		return -EINVAL;
+
+	/* The token pop mode must be set before configuring the port */
+	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
+		return -EINVAL;
+
+	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
new file mode 100644
index 0000000..74399db
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+/*!
+ *  @file      rte_pmd_dlb2.h
+ *
+ *  @brief     DLB PMD-specific functions
+ */
+
+#ifndef _RTE_PMD_DLB2_H_
+#define _RTE_PMD_DLB2_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Selects the token pop mode for a DLB2 port.
+ */
+enum dlb2_token_pop_mode {
+	/* Pop the CQ tokens immediately after dequeueing. */
+	AUTO_POP,
+	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
+	 * Supported on load-balanced ports only.
+	 */
+	DELAYED_POP,
+	/* Pop the CQ tokens during next dequeue operation. */
+	DEFERRED_POP,
+
+	/* NUM_TOKEN_POP_MODES must be last */
+	NUM_TOKEN_POP_MODES
+};
+
+/*!
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Configure the token pop mode for a DLB2 port. By default, all ports use
+ * AUTO_POP. This function must be called before calling rte_event_port_setup()
+ * for the port, but after calling rte_event_dev_configure().
+ *
+ * @param dev_id
+ *    The identifier of the event device.
+ * @param port_id
+ *    The identifier of the event port.
+ * @param mode
+ *    The token pop mode.
+ *
+ * @return
+ * - 0: Success
+ * - EINVAL: Invalid dev_id, port_id, or mode
+ * - EINVAL: The DLB2 is not configured, is already running, or the port is
+ *   already setup
+ */
+
+__rte_experimental
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_DLB2_H_ */
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
index 4a76d1d..b1e4dff 100644
--- a/drivers/event/dlb2/version.map
+++ b/drivers/event/dlb2/version.map
@@ -1,3 +1,9 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmd_dlb2_set_token_pop_mode;
+};
-- 
2.6.4

[dpdk-dev] [PATCH v7 21/23] event/dlb2: add PMD self-tests

[Timothy McDaniel]

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 app/test/test_eventdev.c           |    7 +
 drivers/event/dlb2/dlb2.c          |    1 +
 drivers/event/dlb2/dlb2_selftest.c | 1524 ++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build     |    3 +-
 4 files changed, 1534 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c

diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
index 62019c1..bcfaa53 100644
--- a/app/test/test_eventdev.c
+++ b/app/test/test_eventdev.c
@@ -1030,6 +1030,12 @@ test_eventdev_selftest_dpaa2(void)
 	return test_eventdev_selftest_impl("event_dpaa2", "");
 }
 
+static int
+test_eventdev_selftest_dlb2(void)
+{
+	return test_eventdev_selftest_impl("dlb2_event", "");
+}
+
 REGISTER_TEST_COMMAND(eventdev_common_autotest, test_eventdev_common);
 REGISTER_TEST_COMMAND(eventdev_selftest_sw, test_eventdev_selftest_sw);
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
@@ -1037,3 +1043,4 @@ REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
 		test_eventdev_selftest_octeontx2);
 REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2, test_eventdev_selftest_dpaa2);
+REGISTER_TEST_COMMAND(eventdev_selftest_dlb2, test_eventdev_selftest_dlb2);
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7a22231..99a22e6 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3692,6 +3692,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
 		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+		.dev_selftest     = test_dlb2_eventdev,
 	};
 
 	/* Expose PMD's eventdev interface */
diff --git a/drivers/event/dlb2/dlb2_selftest.c b/drivers/event/dlb2/dlb2_selftest.c
new file mode 100644
index 0000000..e300067
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_selftest.c
@@ -0,0 +1,1524 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <unistd.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_eventdev.h>
+#include <rte_pause.h>
+
+#include "dlb2_priv.h"
+#include "rte_pmd_dlb2.h"
+
+#define MAX_PORTS 32
+#define MAX_QIDS 32
+#define DEFAULT_NUM_SEQ_NUMS 64
+
+static struct rte_mempool *eventdev_func_mempool;
+static int evdev;
+
+struct test {
+	struct rte_mempool *mbuf_pool;
+	int nb_qids;
+};
+
+/* initialization and config */
+static inline int
+init(struct test *t, int nb_queues, int nb_ports)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	memset(t, 0, sizeof(*t));
+
+	t->mbuf_pool = eventdev_func_mempool;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = nb_queues;
+	config.nb_event_ports = nb_ports;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0)
+		printf("%d: Error configuring device\n", __LINE__);
+
+	return ret;
+}
+
+static inline int
+create_ports(int num_ports)
+{
+	int i;
+
+	if (num_ports > MAX_PORTS)
+		return -1;
+
+	for (i = 0; i < num_ports; i++) {
+		struct rte_event_port_conf conf;
+
+		if (rte_event_port_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default port conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		if (rte_event_port_setup(evdev, i, &conf) < 0) {
+			printf("%d: Error setting up port %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+create_lb_qids(struct test *t, int num_qids, uint32_t flags)
+{
+	int i;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		struct rte_event_queue_conf conf;
+
+		if (rte_event_queue_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default queue conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		conf.schedule_type = flags;
+
+		if (conf.schedule_type == RTE_SCHED_TYPE_PARALLEL)
+			conf.nb_atomic_order_sequences = 0;
+		else
+			conf.nb_atomic_order_sequences = DEFAULT_NUM_SEQ_NUMS;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+static inline int
+create_atomic_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ATOMIC);
+}
+
+/* destruction */
+static inline void
+cleanup(void)
+{
+	int ret = 0;
+
+	rte_event_dev_stop(evdev);
+	ret = rte_event_dev_close(evdev);
+
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+};
+
+static inline int
+enqueue_timeout(uint8_t port_id, struct rte_event *ev, uint64_t tmo_us)
+{
+	const uint64_t start = rte_get_timer_cycles();
+	const uint64_t ticks = (tmo_us * rte_get_timer_hz()) / 1E6;
+
+	while ((rte_get_timer_cycles() - start) < ticks) {
+		if (rte_event_enqueue_burst(evdev, port_id, ev, 1) == 1)
+			return 0;
+
+		if (rte_errno != -ENOSPC) {
+			printf("enqueue_burst returned rte_errno %d\n",
+			       rte_errno);
+			return -1;
+		}
+	}
+	printf("%s time out\n", __func__);
+	return -1;
+}
+
+static void
+flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused)
+{
+	rte_pktmbuf_free(event.mbuf);
+}
+
+static int
+test_stop_flush(struct test *t) /* test to check we can properly flush events */
+{
+	struct rte_event ev;
+	uint32_t dequeue_depth;
+	unsigned int i, count;
+	uint8_t queue_id;
+
+	ev.op = RTE_EVENT_OP_NEW;
+
+	if (init(t, 2, 1) < 0 ||
+	    create_ports(1) < 0 ||
+	    create_atomic_qids(t, 2) < 0) {
+		printf("%d: Error initializing device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_port_link(evdev, 0, NULL, NULL, 0) != 2) {
+		printf("%d: Error linking queues to the port\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: Error with start call\n", __LINE__);
+		goto err;
+	}
+
+	/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked
+	 * queue.
+	 */
+	queue_id = 1;
+
+	if (rte_event_port_unlink(evdev, 0, &queue_id, 1) != 1) {
+		printf("%d: Error unlinking queue 1 from port\n", __LINE__);
+		goto err;
+	}
+
+	count = rte_mempool_avail_count(t->mbuf_pool);
+
+	if (rte_event_port_attr_get(evdev,
+				    0,
+				    RTE_EVENT_PORT_ATTR_DEQ_DEPTH,
+				    &dequeue_depth)) {
+		printf("%d: Error retrieveing dequeue depth\n", __LINE__);
+		goto err;
+	}
+
+	/* Send QEs to queue 0 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 0;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Send QEs to queue 1 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 1;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Now the DLB is scheduling events from the port to the IQ, and at
+	 * least one event should be remaining in each queue.
+	 */
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, flush, NULL)) {
+		printf("%d: Error installing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+
+	if (count != rte_mempool_avail_count(t->mbuf_pool)) {
+		printf("%d: Error executing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, NULL, NULL)) {
+		printf("%d: Error uninstalling the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	return 0;
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_single_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Create a directed port */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) == 0) {
+		printf("%d: port 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Create a load-balanced port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a directed queue */
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed queue */
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) == 0) {
+		printf("%d: queue 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a load-balanced queue */
+	queue_conf.event_queue_cfg = 0;
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to link directed and load-balanced resources */
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 0 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 1 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Link ports to queues */
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+#define NUM_LDB_PORTS 64
+#define NUM_LDB_QUEUES 32
+
+static int
+test_info_get(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	config.nb_event_ports = info.max_event_ports;
+	config.nb_event_queues = NUM_LDB_QUEUES + info.max_event_ports / 2;
+	config.nb_single_link_event_port_queues = info.max_event_ports / 2;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		goto err;
+	}
+
+	/* The DLB2 PMD only reports load-balanced ports and queues in its
+	 * info_get function. Confirm that these values don't include the
+	 * directed port or queue counts.
+	 */
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret) {
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+		return -1;
+	}
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+static int
+test_reconfiguration_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 2 LDB queues */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+			printf("%d: P0->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P1->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+			printf("%d: P1->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Configure device with 2 DIR ports and 2 DIR queues */
+	config.nb_single_link_event_port_queues = 2;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q1 */
+	queue_id = 1;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: P0->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Link P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: P1->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_queues = 5;
+	config.nb_event_ports = 5;
+	config.nb_single_link_event_port_queues = 1;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < config.nb_event_queues - 1; i++) {
+		port_conf.event_port_cfg = 0;
+		queue_conf.event_queue_cfg = 0;
+
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: P%d->Q%d link expected to succeed\n",
+			       __LINE__, i, i);
+			goto err;
+		}
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+		printf("%d: port %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+		printf("%d: queue %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	queue_id = i;
+
+	if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+		printf("%d: P%d->Q%d link expected to succeed\n",
+		       __LINE__, i, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_ports += 1;
+
+	/* Reconfigure device with 1 more load-balanced port */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Configure the new port */
+	if (rte_event_port_setup(evdev, config.nb_event_ports - 1,
+				 &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_load_balanced_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_directed_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 DIR port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (ev.queue_id != 0) {
+		printf("%d: invalid dequeued event queue ID (%d)\n",
+		       __LINE__, ev.queue_id);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_deferred_sched(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	const int num_events = 128;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 1 queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 1, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.dequeue_depth = 1;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.schedule_type = RTE_SCHED_TYPE_PARALLEL;
+	queue_conf.nb_atomic_order_sequences = 0;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 128 NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_PARALLEL;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue one event from port 0 */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue (and release) all other events from port 1. Deferred
+	 * scheduling ensures no other events are scheduled to port 0 without a
+	 * subsequent rte_event_dequeue_burst() call.
+	 */
+	for (i = 0; i < num_events - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 1, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		if (rte_event_enqueue_burst(evdev, 1, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_delayed_pop(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	int ret, i, num_events;
+	struct rte_event ev;
+	uint8_t queue_id;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DELAYED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	num_events = 2 * port_conf.dequeue_depth;
+
+	/* Enqueue 2 * dequeue_depth NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue dequeue_depth events but only release dequeue_depth - 2.
+	 * Delayed pop won't perform the pop and no more events will be
+	 * scheduled.
+	 */
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = 0; i < port_conf.dequeue_depth - 2; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	ret = rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout);
+	if (ret != 0) {
+		printf("%d: event dequeue expected to fail (ret = %d)\n",
+		       __LINE__, ret);
+		goto err;
+	}
+
+	/* Release one more event. This will trigger the token pop, and
+	 * dequeue_depth - 1 more events will be scheduled to the device.
+	 */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: RELEASE enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 0) {
+		printf("%d: event dequeue expected to fail\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+do_selftest(void)
+{
+	struct test t;
+	int ret;
+
+	/* Only create mbuf pool once, reuse for each test run */
+	if (!eventdev_func_mempool) {
+		eventdev_func_mempool =
+			rte_pktmbuf_pool_create("EVENTDEV_DLB2_ST_POOL",
+						(1 << 12), /* 4k buffers */
+						32 /*MBUF_CACHE_SIZE*/,
+						0,
+						512, /* use very small mbufs */
+						rte_socket_id());
+		if (!eventdev_func_mempool) {
+			printf("ERROR creating mempool\n");
+			goto test_fail;
+		}
+	}
+	t.mbuf_pool = eventdev_func_mempool;
+
+	printf("*** Running Stop Flush test...\n");
+	ret = test_stop_flush(&t);
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Single Link test...\n");
+	ret = test_single_link();
+	if (ret != 0) {
+		printf("ERROR - Single Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Info Get test...\n");
+	ret = test_info_get();
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Reconfiguration Link test...\n");
+	ret = test_reconfiguration_link();
+	if (ret != 0) {
+		printf("ERROR - Reconfiguration Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Load-Balanced Traffic test...\n");
+	ret = test_load_balanced_traffic();
+	if (ret != 0) {
+		printf("ERROR - Load-Balanced Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Directed Traffic test...\n");
+	ret = test_directed_traffic();
+	if (ret != 0) {
+		printf("ERROR - Directed Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Deferred Scheduling test...\n");
+	ret = test_deferred_sched();
+	if (ret != 0) {
+		printf("ERROR - Deferred Scheduling test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Delayed Pop test...\n");
+	ret = test_delayed_pop();
+	if (ret != 0) {
+		printf("ERROR - Delayed Pop test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	return 0;
+
+test_fail:
+	return -1;
+}
+
+int
+test_dlb2_eventdev(void)
+{
+	const char *dlb2_eventdev_name = "dlb2_event";
+	uint8_t num_evdevs = rte_event_dev_count();
+	int i, ret = 0;
+	int found = 0, skipped = 0, passed = 0, failed = 0;
+	struct rte_event_dev_info info;
+
+	for (i = 0; found + skipped < num_evdevs && i < RTE_EVENT_MAX_DEVS;
+	     i++) {
+		ret = rte_event_dev_info_get(i, &info);
+		if (ret < 0)
+			continue;
+
+		/* skip non-dlb2 event devices */
+		if (strncmp(info.driver_name, dlb2_eventdev_name,
+			    sizeof(*info.driver_name)) != 0) {
+			skipped++;
+			continue;
+		}
+
+		evdev = rte_event_dev_get_dev_id(info.driver_name);
+		if (evdev < 0) {
+			printf("Could not get dev_id for eventdev with name %s, i=%d\n",
+			       info.driver_name, i);
+			skipped++;
+			continue;
+		}
+		found++;
+		printf("Running selftest on eventdev %s\n", info.driver_name);
+		ret = do_selftest();
+		if (ret == 0) {
+			passed++;
+			printf("Selftest passed for eventdev %s\n",
+			       info.driver_name);
+		} else {
+			failed++;
+			printf("Selftest failed for eventdev %s, err=%d\n",
+			       info.driver_name, ret);
+		}
+	}
+
+	printf("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n",
+	       found, skipped, passed, failed);
+	return ret;
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index b734ff6..340bc57 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -14,7 +14,8 @@ sources = files('dlb2.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c',
-		'rte_pmd_dlb2.c'
+		'rte_pmd_dlb2.c',
+		'dlb2_selftest.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v7 22/23] event/dlb2: add queue and port release

[Timothy McDaniel]

DLB does not support reconfiguring individual queues
or ports on the fly. The entire device must be reconfigured.
Previously allocated port QE and memzone memory
is freed in this patch.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 99a22e6..d9b13a1 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3668,6 +3668,28 @@ dlb2_eventdev_close(struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(id);
+
+	/* This function intentionally left blank. */
+}
+
+static void
+dlb2_eventdev_port_release(void *port)
+{
+	struct dlb2_eventdev_port *ev_port = port;
+	struct dlb2_port *qm_port;
+
+	if (ev_port) {
+		qm_port = &ev_port->qm_port;
+		if (qm_port->config_state == DLB2_CONFIGURED)
+			dlb2_free_qe_mem(qm_port);
+	}
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3681,8 +3703,10 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
+		.queue_release    = dlb2_eventdev_queue_release,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_release     = dlb2_eventdev_port_release,
 		.port_link        = dlb2_eventdev_port_link,
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
-- 
2.6.4

[dpdk-dev] [PATCH v7 23/23] event/dlb2: add timeout ticks entry point

[Timothy McDaniel]

Adds the timeout ticks conversion function.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index d9b13a1..d12cbf7 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3689,6 +3689,18 @@ dlb2_eventdev_port_release(void *port)
 	}
 }
 
+static int
+dlb2_eventdev_timeout_ticks(struct rte_eventdev *dev, uint64_t ns,
+			    uint64_t *timeout_ticks)
+{
+	RTE_SET_USED(dev);
+	uint64_t cycles_per_ns = rte_get_timer_hz() / 1E9;
+
+	*timeout_ticks = ns * cycles_per_ns;
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -3711,6 +3723,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
 				    dlb2_eventdev_port_unlinks_in_progress,
+		.timeout_ticks    = dlb2_eventdev_timeout_ticks,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v8 00/23] Add DLB2 PMD

[Timothy McDaniel]

The following patch series adds support for a new eventdev PMD. The DLB2
PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
hardware.
The DLB2 is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB2 hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB2 ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a
directed queue. Additionally, even though LDB ports may link multiple
queues,
the number of queues that may be linked is limited by hardware.

While reviewing the code, please be aware that this PMD has full
control over the DLB2 hardware. Intel will be extending the DLB2 PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.
Note that the DLB2 hardware is a successor of the DLB hardware, and
as such is structured similarly, both in terms of code layout and
implementation.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Changes since V7:
================
- fix CENTOS build error: use __m128i instead of __v2di with
  _mm_stream_si128
- deleted unused dlb2_umwait and dlb2_umonitor functions

Changes since V6:
=================
- removed unused function, fixing build error
- fixed typo in port_setup commit message
- this patch series is based on dpdk-next-eventdev

Changes since V5:
================
- convert to use rte_power_monitor patches
- replace __builtin_ia32_movntdq() with _mm_stream_si128()
- remove unused functions in dlb_selftest.c
- workaround for RHEL 7 gcc brace bug

Changes since V4:
================
- moved introduction of dlb in relnotes_20_11 to first patch in series
- fixed underlines in dlb.rst that were too short
- note that the code still uses its private byte-encoded versions of
  umonitor/umwait, rather than the new functions in the power
  patch that are built on top of those intrinsics. This is intentional.

Changes since V3:
================
- updated MAINTAINERS file to alphabetically insert DLB
- don't create RTE_ symbols in pmd
- converted to use version.map scheme
- converted to use .._master_lcore instead of .._main_lcore
- this patch set is based on dpdk-next-eventdev

Changes since V2:
================
- fixed meson conditional build. Moved test into driver’s meson.build
  file instead of event/meson.build
- documentation is populated as associated code is introduced
- add log_register in add dynamic logging patch
- rename RTE_xxx symbol(s) as DLB2_xxx
- replaced function ptr enqueue_four with direct call to movdir64b
- remove unused port_pages
- broke up probe patch into 3 smaller patches for easier review
- changed param order of movdir64b/movntdq to match intrinsics
- added self to MAINTAINERS files
- squashed announcement of availability into last patch in series
- correct spelling errors and delete repeated words
- DPDK_21.0 -> DPDK 21 in map file
- add experimental banner to public structs and APIs

Changes since V1:
=================
- implement changes requested in code reviews by Gage Eads and Mike
  Chen
- fix a memzone leak
- convert to use eal rte-cpuflags patch from Liang Ma

Known Issues:
- the documentation contained in dlb2.rst is not complete, and
  should be updated to include command line parameters (class of service,
  etc ...).

Depends-on: patch-82202 ("eventdev: increase MAX QUEUES PER DEV to 255")

Timothy McDaniel (23):
  event/dlb2: add documentation and meson build infrastructure
  event/dlb2: add dynamic logging
  event/dlb2: add private data structures and constants
  event/dlb2: add definitions shared with LKM or shared code
  event/dlb2: add inline functions
  event/dlb2: add eventdev probe
  event/dlb2: add flexible interface
  event/dlb2: add probe-time hardware init
  event/dlb2: add xstats
  event/dlb2: add infos get and configure
  event/dlb2: add queue and port default conf
  event/dlb2: add queue setup
  event/dlb2: add port setup
  event/dlb2: add port link
  event/dlb2: add port unlink and port unlinks in progress
  event/dlb2: add eventdev start
  event/dlb2: add enqueue and its burst variants
  event/dlb2: add dequeue and its burst variants
  event/dlb2: add eventdev stop and close
  event/dlb2: add PMD's token pop public interface
  event/dlb2: add PMD self-tests
  event/dlb2: add queue and port release
  event/dlb2: add timeout ticks entry point

 MAINTAINERS                                    |    6 +-
 app/test/test_eventdev.c                       |    7 +
 config/rte_config.h                            |    7 +
 doc/api/doxy-api-index.md                      |    1 +
 doc/guides/eventdevs/dlb2.rst                  |  365 ++
 doc/guides/eventdevs/index.rst                 |    1 +
 doc/guides/rel_notes/release_20_11.rst         |    5 +
 drivers/event/dlb2/dlb2.c                      | 3947 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c                |   74 +
 drivers/event/dlb2/dlb2_iface.h                |   74 +
 drivers/event/dlb2/dlb2_inline_fns.h           |   42 +
 drivers/event/dlb2/dlb2_log.h                  |   25 +
 drivers/event/dlb2/dlb2_priv.h                 |  578 +++
 drivers/event/dlb2/dlb2_selftest.c             | 1524 ++++++
 drivers/event/dlb2/dlb2_user.h                 |  679 +++
 drivers/event/dlb2/dlb2_xstats.c               | 1240 +++++
 drivers/event/dlb2/meson.build                 |   22 +
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 +++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 +
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c     | 6027 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  673 +++
 drivers/event/dlb2/pf/dlb2_main.h              |   97 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  728 +++
 drivers/event/dlb2/rte_pmd_dlb2.c              |   39 +
 drivers/event/dlb2/rte_pmd_dlb2.h              |   72 +
 drivers/event/dlb2/version.map                 |    9 +
 drivers/event/meson.build                      |    3 +-
 33 files changed, 22478 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
 create mode 100644 drivers/event/dlb2/dlb2_log.h
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c
 create mode 100644 drivers/event/dlb2/dlb2_user.h
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
 create mode 100644 drivers/event/dlb2/version.map

-- 
2.6.4

[dpdk-dev] [PATCH v8 01/23] event/dlb2: add documentation and meson build infrastructure

[Timothy McDaniel]

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux 64 bit X86 platforms at this time.

Adds announcement of availabililty for the new driver
for Intel Dynamic Load Balancer 2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 MAINTAINERS                            |  6 ++++-
 config/rte_config.h                    |  7 ++++++
 doc/guides/eventdevs/dlb2.rst          | 40 ++++++++++++++++++++++++++++++++++
 doc/guides/eventdevs/index.rst         |  1 +
 doc/guides/rel_notes/release_20_11.rst |  5 +++++
 drivers/event/dlb2/meson.build         | 13 +++++++++++
 drivers/event/dlb2/version.map         |  3 +++
 drivers/event/meson.build              |  3 ++-
 8 files changed, 76 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index a3d1927..727451b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1162,6 +1162,11 @@ Cavium OCTEON TX timvf
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 F: drivers/event/octeontx/timvf_*
 
+Intel DLB2
+M: Timothy McDaniel <timothy.mcdaniel@intel.com>
+F: drivers/event/dlb2/
+F: doc/guides/eventdevs/dlb2.rst
+
 Marvell OCTEON TX2
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 M: Jerin Jacob <jerinj@marvell.com>
@@ -1198,7 +1203,6 @@ M: Peter Mccarthy <peter.mccarthy@intel.com>
 F: drivers/event/opdl/
 F: doc/guides/eventdevs/opdl.rst
 
-
 Rawdev Drivers
 --------------
 
diff --git a/config/rte_config.h b/config/rte_config.h
index b78c6aa..64aa333 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -135,4 +135,11 @@
 /* QEDE PMD defines */
 #define RTE_LIBRTE_QEDE_FW ""
 
+/* DLB2 defines */
+#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
+#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
+#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
+#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
+
 #endif /* _RTE_CONFIG_H_ */
diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
new file mode 100644
index 0000000..e3091c9
--- /dev/null
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -0,0 +1,40 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Driver for the Intel® Dynamic Load Balancer (DLB2)
+==================================================
+
+The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
+
+Prerequisites
+-------------
+
+Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
+the basic DPDK environment.
+
+Configuration
+-------------
+
+The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
+device access. To use this operation mode, the PCIe PF device must be bound
+to a DPDK-compatible VFIO driver, such as vfio-pci.
+
+Eventdev API Notes
+------------------
+
+The DLB2 provides the functions of a DPDK event device; specifically, it
+supports atomic, ordered, and parallel scheduling events from queues to ports.
+However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
+features are abstracted by the PMD such as directed ports.
+
+In general the dlb PMD is designed for ease-of-use and does not require a
+detailed understanding of the hardware, but these details are important when
+writing high-performance code. This section describes the places where the
+eventdev API and DLB2 misalign.
+
+Flow ID
+~~~~~~~
+
+The flow ID field is preserved in the event when it is scheduled in the
+DLB2.
+
diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
index bb66a5e..738788d 100644
--- a/doc/guides/eventdevs/index.rst
+++ b/doc/guides/eventdevs/index.rst
@@ -11,6 +11,7 @@ application through the eventdev API.
     :maxdepth: 2
     :numbered:
 
+    dlb2
     dpaa
     dpaa2
     dsw
diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
index d8ac359..dbce3b1 100644
--- a/doc/guides/rel_notes/release_20_11.rst
+++ b/doc/guides/rel_notes/release_20_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
+
+  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
+  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
+
 * **Added write combining store APIs.**
 
   Added ``rte_write32_wc`` and ``rte_write32_wc_relaxed`` APIs
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
new file mode 100644
index 0000000..5324043
--- /dev/null
+++ b/drivers/event/dlb2/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2020 Intel Corporation
+
+if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
+        build = false
+        reason = 'only supported on ARCH_X86_64 Linux'
+        subdir_done()
+endif
+
+sources = files(
+)
+
+deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
new file mode 100644
index 0000000..4a76d1d
--- /dev/null
+++ b/drivers/event/dlb2/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index a7dac99..c433c10 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -5,7 +5,8 @@ if is_windows
 	subdir_done()
 endif
 
-drivers = ['dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw', 'dsw']
+drivers = ['dlb2', 'dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw',
+	   'dsw']
 if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
-- 
2.6.4

[dpdk-dev] [PATCH v8 02/23] event/dlb2: add dynamic logging

[Timothy McDaniel]

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c      |  7 +++++++
 drivers/event/dlb2/dlb2_log.h  | 25 +++++++++++++++++++++++++
 drivers/event/dlb2/meson.build |  3 +--
 3 files changed, 33 insertions(+), 2 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_log.h

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
new file mode 100644
index 0000000..2082dd6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2.c
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <rte_log.h>
+
+RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
new file mode 100644
index 0000000..dc1481e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_log.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_EVDEV_LOG_H_
+#define _DLB2_EVDEV_LOG_H_
+
+extern int eventdev_dlb2_log_level;
+
+/* Dynamic logging */
+#define DLB2_LOG_IMPL(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
+		__func__, ##args)
+
+#define DLB2_LOG_INFO(fmt, args...) \
+	DLB2_LOG_IMPL(INFO, fmt, ## args)
+
+#define DLB2_LOG_ERR(fmt, args...) \
+	DLB2_LOG_IMPL(ERR, fmt, ## args)
+
+/* remove debug logs at compile time unless actually debugging */
+#define DLB2_LOG_DBG(fmt, args...) \
+	RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
+
+#endif /* _DLB2_EVDEV_LOG_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 5324043..435c1ee 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,7 +7,6 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files(
-)
+sources = files('dlb2.c')
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v8 03/23] event/dlb2: add private data structures and constants

[Timothy McDaniel]

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_priv.h | 575 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 575 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_priv.h

diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
new file mode 100644
index 0000000..61567a6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -0,0 +1,575 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_PRIV_H_
+#define _DLB2_PRIV_H_
+
+#include <emmintrin.h>
+#include <stdbool.h>
+
+#include <rte_eventdev.h>
+#include <rte_config.h>
+#include "dlb2_user.h"
+#include "dlb2_log.h"
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
+#else
+#define DLB2_INC_STAT(_stat, _incr_val)
+#endif
+
+#define EVDEV_DLB2_NAME_PMD dlb2_event
+
+/*  command line arg strings */
+#define NUMA_NODE_ARG "numa_node"
+#define DLB2_MAX_NUM_EVENTS "max_num_events"
+#define DLB2_NUM_DIR_CREDITS "num_dir_credits"
+#define DEV_ID_ARG "dev_id"
+#define DLB2_DEFER_SCHED_ARG "defer_sched"
+#define DLB2_QID_DEPTH_THRESH_ARG "qid_depth_thresh"
+#define DLB2_COS_ARG "cos"
+
+/* Begin HW related defines and structs */
+
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_VFS 16
+#define DLB2_MAX_NUM_LDB_QUEUES 32
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS 64
+#define DLB2_MAX_NUM_DIR_QUEUES 64
+#define DLB2_MAX_NUM_FLOWS (64 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDITS (8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS (2 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_DIR_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQOS_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_MAX_DEVICE_PATH 32
+#define DLB2_MIN_DEQUEUE_TIMEOUT_NS 1
+/* Note: "- 1" here to support the timeout range check in eventdev_autotest */
+#define DLB2_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1)
+#define DLB2_SW_CREDIT_BATCH_SZ 32
+#define DLB2_NUM_SN_GROUPS 2
+#define DLB2_MAX_LDB_SN_ALLOC 1024
+#define DLB2_MAX_QUEUE_DEPTH_THRESHOLD 8191
+
+/* 2048 total hist list entries and 64 total ldb ports, which
+ * makes for 2048/64 == 32 hist list entries per port. However, CQ
+ * depth must be a power of 2 and must also be >= HIST LIST entries.
+ * As a result we just limit the maximum dequeue depth to 32.
+ */
+#define DLB2_MIN_CQ_DEPTH 1
+#define DLB2_MAX_CQ_DEPTH 32
+#define DLB2_MIN_HARDWARE_CQ_DEPTH 8
+#define DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \
+	DLB2_MAX_CQ_DEPTH
+
+/*
+ * Static per queue/port provisioning values
+ */
+#define DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 64
+
+#define CQ_BASE(is_dir) ((is_dir) ? DLB2_DIR_CQ_BASE : DLB2_LDB_CQ_BASE)
+#define CQ_SIZE(is_dir) ((is_dir) ? DLB2_DIR_CQ_MAX_SIZE : \
+				    DLB2_LDB_CQ_MAX_SIZE)
+#define PP_BASE(is_dir) ((is_dir) ? DLB2_DIR_PP_BASE : DLB2_LDB_PP_BASE)
+
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+
+#define DLB2_NUM_QES_PER_CACHE_LINE 4
+
+#define DLB2_MAX_ENQUEUE_DEPTH 64
+#define DLB2_MIN_ENQUEUE_DEPTH 4
+
+#define DLB2_NAME_SIZE 64
+
+#define DLB2_1K 1024
+#define DLB2_2K (2 * DLB2_1K)
+#define DLB2_4K (4 * DLB2_1K)
+#define DLB2_16K (16 * DLB2_1K)
+#define DLB2_32K (32 * DLB2_1K)
+#define DLB2_1MB (DLB2_1K * DLB2_1K)
+#define DLB2_16MB (16 * DLB2_1MB)
+
+/* Use the upper 3 bits of the event priority to select the DLB2 priority */
+#define EV_TO_DLB2_PRIO(x) ((x) >> 5)
+#define DLB2_TO_EV_PRIO(x) ((x) << 5)
+
+enum dlb2_hw_port_types {
+	DLB2_LDB_PORT,
+	DLB2_DIR_PORT,
+	DLB2_NUM_PORT_TYPES /* Must be last */
+};
+
+enum dlb2_hw_queue_types {
+	DLB2_LDB_QUEUE,
+	DLB2_DIR_QUEUE,
+	DLB2_NUM_QUEUE_TYPES /* Must be last */
+};
+
+#define PORT_TYPE(p) ((p)->is_directed ? DLB2_DIR_PORT : DLB2_LDB_PORT)
+
+/* Do not change - must match hardware! */
+enum dlb2_hw_sched_type {
+	DLB2_SCHED_ATOMIC = 0,
+	DLB2_SCHED_UNORDERED,
+	DLB2_SCHED_ORDERED,
+	DLB2_SCHED_DIRECTED,
+	/* DLB2_NUM_HW_SCHED_TYPES must be last */
+	DLB2_NUM_HW_SCHED_TYPES
+};
+
+struct dlb2_hw_rsrcs {
+	int32_t nb_events_limit;
+	uint32_t num_queues;		/* Total queues (lb + dir) */
+	uint32_t num_ldb_queues;	/* Number of available ldb queues */
+	uint32_t num_ldb_ports;         /* Number of load balanced ports */
+	uint32_t num_dir_ports;         /* Number of directed ports */
+	uint32_t num_ldb_credits;       /* Number of load balanced credits */
+	uint32_t num_dir_credits;       /* Number of directed credits */
+	uint32_t reorder_window_size;   /* Size of reorder window */
+};
+
+struct dlb2_hw_resource_info {
+	/**> Max resources that can be provided */
+	struct dlb2_hw_rsrcs hw_rsrc_max;
+	int num_sched_domains;
+	uint32_t socket_id;
+};
+
+enum dlb2_enqueue_type {
+	/**>
+	 * New : Used to inject a new packet into the QM.
+	 */
+	DLB2_ENQ_NEW,
+	/**>
+	 * Forward : Enqueues a packet, and
+	 *  - if atomic: release any lock it holds in the QM
+	 *  - if ordered: release the packet for egress re-ordering
+	 */
+	DLB2_ENQ_FWD,
+	/**>
+	 * Enqueue Drop : Release an inflight packet. Must be called with
+	 * event == NULL. Used to drop a packet.
+	 *
+	 * Note that all packets dequeued from a load-balanced port must be
+	 * released, either with DLB2_ENQ_DROP or DLB2_ENQ_FWD.
+	 */
+	DLB2_ENQ_DROP,
+
+	/* marker for array sizing etc. */
+	_DLB2_NB_ENQ_TYPES
+};
+
+/* hw-specific format - do not change */
+
+struct dlb2_event_type {
+	uint8_t major:4;
+	uint8_t unused:4;
+	uint8_t sub;
+};
+
+union dlb2_opaque_data {
+	uint16_t opaque_data;
+	struct dlb2_event_type event_type;
+};
+
+struct dlb2_msg_info {
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+};
+
+#define DLB2_NEW_CMD_BYTE 0x08
+#define DLB2_FWD_CMD_BYTE 0x0A
+#define DLB2_COMP_CMD_BYTE 0x02
+#define DLB2_POP_CMD_BYTE 0x01
+#define DLB2_NOOP_CMD_BYTE 0x00
+
+/* hw-specific format - do not change */
+struct dlb2_enqueue_qe {
+	uint64_t data;
+	/* Word 3 */
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	/* Word 4 */
+	uint16_t lock_id;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_cq_pop_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t tokens:10;
+	uint16_t rsvd2:6;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_dequeue_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t flow_id:16; /* was pp_id in v1 */
+	uint8_t debug;
+	uint8_t cq_gen:1;
+	uint8_t qid_depth:2; /* 2 bits in v2 */
+	uint8_t rsvd1:2;
+	uint8_t error:1;
+	uint8_t rsvd2:2;
+};
+
+union dlb2_port_config {
+	struct dlb2_create_ldb_port_args ldb;
+	struct dlb2_create_dir_port_args dir;
+};
+
+enum dlb2_port_state {
+	PORT_CLOSED,
+	PORT_STARTED,
+	PORT_STOPPED
+};
+
+enum dlb2_configuration_state {
+	/* The resource has not been configured */
+	DLB2_NOT_CONFIGURED,
+	/* The resource was configured, but the device was stopped */
+	DLB2_PREV_CONFIGURED,
+	/* The resource is currently configured */
+	DLB2_CONFIGURED
+};
+
+struct dlb2_port {
+	uint32_t id;
+	bool is_directed;
+	bool gen_bit;
+	uint16_t dir_credits;
+	uint32_t dequeue_depth;
+	union dlb2_port_config cfg;
+	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
+	uint16_t cached_ldb_credits;
+	uint16_t ldb_credits;
+	uint16_t cached_dir_credits;
+	bool int_armed;
+	uint16_t owed_tokens;
+	int16_t issued_releases;
+	int cq_depth;
+	uint16_t cq_idx;
+	uint16_t cq_idx_unmasked;
+	uint16_t cq_depth_mask;
+	uint16_t gen_bit_shift;
+	enum dlb2_port_state state;
+	enum dlb2_configuration_state config_state;
+	int num_mapped_qids;
+	uint8_t *qid_mappings;
+	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
+	struct dlb2_enqueue_qe *int_arm_qe;
+	struct dlb2_cq_pop_qe *consume_qe;
+	struct dlb2_eventdev *dlb2; /* back ptr */
+	struct dlb2_eventdev_port *ev_port; /* back ptr */
+};
+
+/* Per-process per-port mmio and memory pointers */
+struct process_local_port_data {
+	uint64_t *pp_addr;
+	struct dlb2_dequeue_qe *cq_base;
+	const struct rte_memzone *mz;
+	bool mmaped;
+};
+
+struct dlb2_eventdev;
+
+struct dlb2_config {
+	int configured;
+	int reserved;
+	uint32_t num_ldb_credits;
+	uint32_t num_dir_credits;
+	struct dlb2_create_sched_domain_args resources;
+};
+
+enum dlb2_cos {
+	DLB2_COS_DEFAULT = -1,
+	DLB2_COS_0 = 0,
+	DLB2_COS_1,
+	DLB2_COS_2,
+	DLB2_COS_3
+};
+
+struct dlb2_hw_dev {
+	struct dlb2_config cfg;
+	struct dlb2_hw_resource_info info;
+	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
+	uint32_t domain_id;
+	enum dlb2_cos cos_id;
+	rte_spinlock_t resource_lock; /* for MP support */
+} __rte_cache_aligned;
+
+/* End HW related defines and structs */
+
+/* Begin DLB2 PMD Eventdev related defines and structs */
+
+#define DLB2_MAX_NUM_QUEUES \
+	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
+
+#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS + DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
+
+/** Structure to hold the queue to port link establishment attributes */
+
+struct dlb2_event_queue_link {
+	uint8_t queue_id;
+	uint8_t priority;
+	bool mapped;
+	bool valid;
+};
+
+struct dlb2_traffic_stats {
+	uint64_t rx_ok;
+	uint64_t rx_drop;
+	uint64_t rx_interrupt_wait;
+	uint64_t rx_umonitor_umwait;
+	uint64_t tx_ok;
+	uint64_t total_polls;
+	uint64_t zero_polls;
+	uint64_t tx_nospc_ldb_hw_credits;
+	uint64_t tx_nospc_dir_hw_credits;
+	uint64_t tx_nospc_inflight_max;
+	uint64_t tx_nospc_new_event_limit;
+	uint64_t tx_nospc_inflight_credits;
+};
+
+/* DLB2 HW sets the 2bit qid_depth in rx QEs based on the programmable depth
+ * threshold. The global default value in config/common_base (or rte_config.h)
+ * can be overridden on a per-qid basis using a vdev command line parameter.
+ * 3: depth > threshold
+ * 2: threshold >= depth > 3/4 threshold
+ * 1: 3/4 threshold >= depth > 1/2 threshold
+ * 0: depth <= 1/2 threshold.
+ */
+#define DLB2_QID_DEPTH_LE50 0
+#define DLB2_QID_DEPTH_GT50_LE75 1
+#define DLB2_QID_DEPTH_GT75_LE100 2
+#define DLB2_QID_DEPTH_GT100 3
+#define DLB2_NUM_QID_DEPTH_STAT_VALS 4 /* 2 bits */
+
+struct dlb2_queue_stats {
+	uint64_t enq_ok;
+	uint64_t qid_depth[DLB2_NUM_QID_DEPTH_STAT_VALS];
+};
+
+struct dlb2_port_stats {
+	struct dlb2_traffic_stats traffic;
+	uint64_t tx_op_cnt[4]; /* indexed by rte_event.op */
+	uint64_t tx_implicit_rel;
+	uint64_t tx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t tx_invalid;
+	uint64_t rx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t rx_sched_invalid;
+	struct dlb2_queue_stats queue[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_eventdev_port {
+	struct dlb2_port qm_port; /* hw specific data structure */
+	struct rte_event_port_conf conf; /* user-supplied configuration */
+	uint16_t inflight_credits; /* num credits this port has right now */
+	uint16_t credit_update_quanta;
+	struct dlb2_eventdev *dlb2; /* backlink optimization */
+	struct dlb2_port_stats stats __rte_cache_aligned;
+	struct dlb2_event_queue_link link[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	int num_links;
+	uint32_t id; /* port id */
+	/* num releases yet to be completed on this port.
+	 * Only applies to load-balanced ports.
+	 */
+	uint16_t outstanding_releases;
+	uint16_t inflight_max; /* app requested max inflights for this port */
+	/* setup_done is set when the event port is setup */
+	bool setup_done;
+	/* enq_configured is set when the qm port is created */
+	bool enq_configured;
+	uint8_t implicit_release; /* release events before dequeueing */
+}  __rte_cache_aligned;
+
+struct dlb2_queue {
+	uint32_t num_qid_inflights; /* User config */
+	uint32_t num_atm_inflights; /* User config */
+	enum dlb2_configuration_state config_state;
+	int sched_type; /* LB queue only */
+	uint32_t id;
+	bool is_directed;
+};
+
+struct dlb2_eventdev_queue {
+	struct dlb2_queue qm_queue;
+	struct rte_event_queue_conf conf; /* User config */
+	int depth_threshold; /* use default if 0 */
+	uint32_t id;
+	bool setup_done;
+	uint8_t num_links;
+};
+
+enum dlb2_run_state {
+	DLB2_RUN_STATE_STOPPED = 0,
+	DLB2_RUN_STATE_STOPPING,
+	DLB2_RUN_STATE_STARTING,
+	DLB2_RUN_STATE_STARTED
+};
+
+struct dlb2_eventdev {
+	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
+	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each queue */
+	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
+	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each port */
+	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
+	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
+	struct dlb2_get_num_resources_args hw_rsrc_query_results;
+	uint32_t xstats_count_mode_queue;
+	struct dlb2_hw_dev qm_instance; /* strictly hw related */
+	uint64_t global_dequeue_wait_ticks;
+	struct dlb2_xstats_entry *xstats;
+	struct rte_eventdev *event_dev; /* backlink to dev */
+	uint32_t xstats_count_mode_dev;
+	uint32_t xstats_count_mode_port;
+	uint32_t xstats_count;
+	uint32_t inflights; /* use __atomic builtins */
+	uint32_t new_event_limit;
+	int max_num_events_override;
+	int num_dir_credits_override;
+	volatile enum dlb2_run_state run_state;
+	uint16_t num_dir_queues; /* total num of evdev dir queues requested */
+	uint16_t num_dir_credits;
+	uint16_t num_ldb_credits;
+	uint16_t num_queues; /* total queues */
+	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested */
+	uint16_t num_ports; /* total num of evdev ports requested */
+	uint16_t num_ldb_ports; /* total num of ldb ports requested */
+	uint16_t num_dir_ports; /* total num of dir ports requested */
+	bool umwait_allowed;
+	bool global_dequeue_wait; /* Not using per dequeue wait if true */
+	bool defer_sched;
+	enum dlb2_cq_poll_modes poll_mode;
+	uint8_t revision;
+	bool configured;
+	uint16_t max_ldb_credits;
+	uint16_t max_dir_credits;
+
+	/* force hw credit pool counters into exclusive cache lines */
+
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t ldb_credit_pool __rte_cache_aligned;
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t dir_credit_pool __rte_cache_aligned;
+};
+
+/* used for collecting and passing around the dev args */
+struct dlb2_qid_depth_thresholds {
+	int val[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_devargs {
+	int socket_id;
+	int max_num_events;
+	int num_dir_credits_override;
+	int dev_id;
+	int defer_sched;
+	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
+	enum dlb2_cos cos_id;
+};
+
+/* End Eventdev related defines and structs */
+
+/* Forwards for non-inlined functions */
+
+void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
+
+int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
+
+void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
+
+int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n);
+
+int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstat_names,
+		unsigned int *ids, unsigned int size);
+
+uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+					  const char *name, unsigned int *id);
+
+int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode,
+		int16_t queue_port_id,
+		const uint32_t ids[],
+		uint32_t nb_ids);
+
+int test_dlb2_eventdev(void);
+
+int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+				const char *name,
+				struct dlb2_devargs *dlb2_args);
+
+int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+				  const char *name);
+
+uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *queue);
+
+int dlb2_parse_params(const char *params,
+		      const char *name,
+		      struct dlb2_devargs *dlb2_args);
+
+/* Extern globals */
+extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
+
+#endif	/* _DLB2_PRIV_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v8 04/23] event/dlb2: add definitions shared with LKM or shared code

[Timothy McDaniel]

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_user.h | 679 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 679 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_user.h

diff --git a/drivers/event/dlb2/dlb2_user.h b/drivers/event/dlb2/dlb2_user.h
new file mode 100644
index 0000000..f4bda78
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_user.h
@@ -0,0 +1,679 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#define DLB2_MAX_NAME_LEN 64
+
+#include <linux/types.h>
+
+enum dlb2_error {
+	DLB2_ST_SUCCESS = 0,
+	DLB2_ST_NAME_EXISTS,
+	DLB2_ST_DOMAIN_UNAVAILABLE,
+	DLB2_ST_LDB_PORTS_UNAVAILABLE,
+	DLB2_ST_DIR_PORTS_UNAVAILABLE,
+	DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB2_ST_INVALID_DOMAIN_ID,
+	DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB2_ST_INVALID_LDB_QUEUE_ID,
+	DLB2_ST_INVALID_CQ_DEPTH,
+	DLB2_ST_INVALID_CQ_VIRT_ADDR,
+	DLB2_ST_INVALID_PORT_ID,
+	DLB2_ST_INVALID_QID,
+	DLB2_ST_INVALID_PRIORITY,
+	DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+	DLB2_ST_INVALID_DIR_QUEUE_ID,
+	DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB2_ST_DOMAIN_NOT_CONFIGURED,
+	DLB2_ST_INTERNAL_ERROR,
+	DLB2_ST_DOMAIN_IN_USE,
+	DLB2_ST_DOMAIN_NOT_FOUND,
+	DLB2_ST_QUEUE_NOT_FOUND,
+	DLB2_ST_DOMAIN_STARTED,
+	DLB2_ST_DOMAIN_NOT_STARTED,
+	DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB2_ST_DOMAIN_RESET_FAILED,
+	DLB2_ST_MBOX_ERROR,
+	DLB2_ST_INVALID_HIST_LIST_DEPTH,
+	DLB2_ST_NO_MEMORY,
+	DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB2_ST_INVALID_COS_ID,
+};
+
+static const char dlb2_error_strings[][128] = {
+	"DLB2_ST_SUCCESS",
+	"DLB2_ST_NAME_EXISTS",
+	"DLB2_ST_DOMAIN_UNAVAILABLE",
+	"DLB2_ST_LDB_PORTS_UNAVAILABLE",
+	"DLB2_ST_DIR_PORTS_UNAVAILABLE",
+	"DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+	"DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+	"DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+	"DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+	"DLB2_ST_INVALID_DOMAIN_ID",
+	"DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+	"DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+	"DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+	"DLB2_ST_INVALID_LDB_QUEUE_ID",
+	"DLB2_ST_INVALID_CQ_DEPTH",
+	"DLB2_ST_INVALID_CQ_VIRT_ADDR",
+	"DLB2_ST_INVALID_PORT_ID",
+	"DLB2_ST_INVALID_QID",
+	"DLB2_ST_INVALID_PRIORITY",
+	"DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+	"DLB2_ST_INVALID_DIR_QUEUE_ID",
+	"DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+	"DLB2_ST_DOMAIN_NOT_CONFIGURED",
+	"DLB2_ST_INTERNAL_ERROR",
+	"DLB2_ST_DOMAIN_IN_USE",
+	"DLB2_ST_DOMAIN_NOT_FOUND",
+	"DLB2_ST_QUEUE_NOT_FOUND",
+	"DLB2_ST_DOMAIN_STARTED",
+	"DLB2_ST_DOMAIN_NOT_STARTED",
+	"DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+	"DLB2_ST_DOMAIN_RESET_FAILED",
+	"DLB2_ST_MBOX_ERROR",
+	"DLB2_ST_INVALID_HIST_LIST_DEPTH",
+	"DLB2_ST_NO_MEMORY",
+	"DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+	"DLB2_ST_INVALID_COS_ID",
+};
+
+struct dlb2_cmd_response {
+	__u32 status; /* Interpret using enum dlb2_error */
+	__u32 id;
+};
+
+/*******************/
+/* 'dlb2' commands */
+/*******************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+	DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	This ioctl interface is the same in all driver versions and is always
+ *	the first ioctl.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+	/* Output parameters */
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ *	load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ *		percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ *	load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ *   ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+enum dlb2_cq_poll_modes {
+	DLB2_CQ_POLL_MODE_STD,
+	DLB2_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB2_CQ_POLL_MODE must be last */
+	NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/********************************/
+/* 'scheduling domain' commands */
+/********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_start_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
+#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
+
+
+#endif /* __DLB2_USER_H */
-- 
2.6.4

[dpdk-dev] [PATCH v8 05/23] event/dlb2: add inline functions

[Timothy McDaniel]

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_inline_fns.h | 42 ++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h

diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
new file mode 100644
index 0000000..cf7265c
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_inline_fns.h
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_INLINE_FNS_H_
+#define _DLB2_INLINE_FNS_H_
+
+/* Inline functions required in more than one source file. */
+
+static inline struct dlb2_eventdev *
+dlb2_pmd_priv(const struct rte_eventdev *eventdev)
+{
+	return eventdev->data->dev_private;
+}
+
+static inline void
+dlb2_movntdq_single(void *pp_addr, void *qe4)
+{
+	long long *_qe  = (long long *)qe4;
+	__m128i src_data0 = (__m128i){_qe[0], _qe[1]};
+
+	_mm_stream_si128(pp_addr, src_data0);
+}
+
+static inline void
+dlb2_cldemote(void *addr)
+{
+	/* Load addr into RSI, then demote the cache line of the address
+	 * contained in that register.
+	 */
+	asm volatile(".byte 0x0f, 0x1c, 0x06" :: "S" (addr));
+}
+
+static inline void
+dlb2_movdir64b(void *pp_addr, void *qe4)
+{
+	asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
+		     :
+		     : "a" (pp_addr), "d" (qe4));
+}
+
+#endif /* _DLB2_INLINE_FNS_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v8 06/23] event/dlb2: add eventdev probe

[Timothy McDaniel]

Add the eventdev portion of probe, and parse command line
options, but do not initialize hardware.

Changes since v2 patch-set probe:

Primary and secondary probe-time init has been removed, and
will be introduced in subsequent patches contained in
the v3 patch-set.

Hardware init has been moved to a subsequent patch in order to
minimize the patch size. The files dlb2/pf/dlb2_pf.c and
dlb2/pf/dlb_main.c include stubs of hardware init functions,
and these will be replaced with the real versions when the
hardware layer is committed.

Initialization of the flexible interface layer has been moved to
a subsequent patch in order to minimize patch size.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                      |  343 ++++
 drivers/event/dlb2/meson.build                 |    5 +-
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +++
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  621 ++++++
 drivers/event/dlb2/pf/dlb2_main.h              |   98 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  196 ++
 13 files changed, 7497 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2082dd6..1b11979 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2,6 +2,349 @@
  * Copyright(c) 2016-2020 Intel Corporation
  */
 
+#include <assert.h>
+#include <errno.h>
+#include <nmmintrin.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_cycles.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_io.h>
+#include <rte_kvargs.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_ring.h>
+#include <rte_string_fns.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+/*
+ * Resources exposed to eventdev. Some values overridden at runtime using
+ * values returned by the DLB kernel driver.
+ */
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
+#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
+#endif
+
+struct process_local_port_data
+dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
+
+#define DLB2_BASE_10 10
+
+static int
+dlb2_string_to_int(int *result, const char *str)
+{
+	long ret;
+	char *endptr;
+
+	if (str == NULL || result == NULL)
+		return -EINVAL;
+
+	errno = 0;
+	ret = strtol(str, &endptr, DLB2_BASE_10);
+	if (errno)
+		return -errno;
+
+	/* long int and int may be different width for some architectures */
+	if (ret < INT_MIN || ret > INT_MAX || endptr == str)
+		return -EINVAL;
+
+	*result = ret;
+	return 0;
+}
+
+static int
+set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
+{
+	int *socket_id = opaque;
+	int ret;
+
+	ret = dlb2_string_to_int(socket_id, value);
+	if (ret < 0)
+		return ret;
+
+	if (*socket_id > RTE_MAX_NUMA_NODES)
+		return -EINVAL;
+	return 0;
+}
+
+static int
+set_max_num_events(const char *key __rte_unused,
+		   const char *value,
+		   void *opaque)
+{
+	int *max_num_events = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(max_num_events, value);
+	if (ret < 0)
+		return ret;
+
+	if (*max_num_events < 0 || *max_num_events >
+			DLB2_MAX_NUM_LDB_CREDITS) {
+		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
+			     DLB2_MAX_NUM_LDB_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_num_dir_credits(const char *key __rte_unused,
+		    const char *value,
+		    void *opaque)
+{
+	int *num_dir_credits = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(num_dir_credits, value);
+	if (ret < 0)
+		return ret;
+
+	if (*num_dir_credits < 0 ||
+	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
+		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
+			     DLB2_MAX_NUM_DIR_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_dev_id(const char *key __rte_unused,
+	   const char *value,
+	   void *opaque)
+{
+	int *dev_id = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(dev_id, value);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+set_cos(const char *key __rte_unused,
+	const char *value,
+	void *opaque)
+{
+	enum dlb2_cos *cos_id = opaque;
+	int x = 0;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(&x, value);
+	if (ret < 0)
+		return ret;
+
+	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
+		DLB2_LOG_ERR(
+			"COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
+			x);
+		return -EINVAL;
+	}
+
+	*cos_id = x;
+
+	return 0;
+}
+
+
+static int
+set_qid_depth_thresh(const char *key __rte_unused,
+		     const char *value,
+		     void *opaque)
+{
+	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
+	int first, last, thresh, i;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	/* command line override may take one of the following 3 forms:
+	 * qid_depth_thresh=all:<threshold_value> ... all queues
+	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
+	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
+	 */
+	if (sscanf(value, "all:%d", &thresh) == 1) {
+		first = 0;
+		last = DLB2_MAX_NUM_QUEUES - 1;
+	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
+		/* we have everything we need */
+	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
+		last = first;
+	} else {
+		DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val, qid-qid:val, or qid:val\n");
+		return -EINVAL;
+	}
+
+	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
+		return -EINVAL;
+	}
+
+	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
+			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
+		return -EINVAL;
+	}
+
+	for (i = first; i <= last; i++)
+		qid_thresh->val[i] = thresh; /* indexed by qid */
+
+	return 0;
+}
+
+int
+dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+			    const char *name,
+			    struct dlb2_devargs *dlb2_args)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+	RTE_SET_USED(dlb2_args);
+
+	return 0;
+}
+
+int
+dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+			      const char *name)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+int
+dlb2_parse_params(const char *params,
+		  const char *name,
+		  struct dlb2_devargs *dlb2_args)
+{
+	int ret = 0;
+	static const char * const args[] = { NUMA_NODE_ARG,
+					     DLB2_MAX_NUM_EVENTS,
+					     DLB2_NUM_DIR_CREDITS,
+					     DEV_ID_ARG,
+					     DLB2_QID_DEPTH_THRESH_ARG,
+					     DLB2_COS_ARG,
+					     NULL };
+
+	if (params != NULL && params[0] != '\0') {
+		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
+
+		if (kvlist == NULL) {
+			RTE_LOG(INFO, PMD,
+				"Ignoring unsupported parameters when creating device '%s'\n",
+				name);
+		} else {
+			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
+						     set_numa_node,
+						     &dlb2_args->socket_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing numa node parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
+						 set_max_num_events,
+						 &dlb2_args->max_num_events);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist,
+					DLB2_NUM_DIR_CREDITS,
+					set_num_dir_credits,
+					&dlb2_args->num_dir_credits_override);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
+						 set_dev_id,
+						 &dlb2_args->dev_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(
+					kvlist,
+					DLB2_QID_DEPTH_THRESH_ARG,
+					set_qid_depth_thresh,
+					&dlb2_args->qid_depth_thresholds);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
+						 set_cos,
+						 &dlb2_args->cos_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing cos parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
 
+			rte_kvargs_free(kvlist);
+		}
+	}
+	return ret;
+}
 RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 435c1ee..1eab9b9 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,6 +7,9 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files('dlb2.c')
+sources = files('dlb2.c',
+		'pf/dlb2_main.c',
+		'pf/dlb2_pf.c'
+)
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
new file mode 100644
index 0000000..1d99f1e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_osdep_list.h"
+#include "dlb2_osdep_types.h"
+
+#define DLB2_MAX_NUM_VDEVS			16
+#define DLB2_MAX_NUM_DOMAINS			32
+#define DLB2_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS			64
+#define DLB2_MAX_NUM_DIR_PORTS			64
+#define DLB2_MAX_NUM_LDB_CREDITS		(8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS		(2 * 1024)
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB2_MAX_NUM_AQED_ENTRIES		2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB2_QID_PRIORITIES			8
+#define DLB2_NUM_ARB_WEIGHTS			8
+#define DLB2_MAX_WEIGHT				255
+#define DLB2_NUM_COS_DOMAINS			4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#ifdef FPGA
+#define DLB2_HZ					2000000
+#else
+#define DLB2_HZ					800000000
+#endif
+
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
+
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+	DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+	DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID		3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID		4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW		5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ	1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
+#define DLB2_VF_BASE_CQ_VECTOR_ID	     0
+#define DLB2_VF_LAST_CQ_VECTOR_ID	     30
+#define DLB2_VF_MBOX_VECTOR_ID		     31
+#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
+
+#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
+					     DLB2_MAX_NUM_DIR_PORTS + 1)
+
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE   0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
+				DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE   0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
+				DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE       0x2100000
+#define DLB2_LDB_PP_STRIDE     0x1000
+#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
+				DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE       0x2000000
+#define DLB2_DIR_PP_STRIDE     0x1000
+#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
+				DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
+struct dlb2_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb2_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb2_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
+struct dlb2_ldb_queue {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB2_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB2_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB2_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB2_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+	enum dlb2_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb2_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+	u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+	u32 bound[6] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & (1 << i))) {
+			group->slot_use_bitmap |= 1 << i;
+			return i;
+		}
+	}
+
+	return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+	int i, cnt = 0;
+
+	for (i = 0; i < 32; i++)
+		cnt += !!(group->slot_use_bitmap & (1 << i));
+
+	return cnt;
+}
+
+struct dlb2_hw_domain {
+	struct dlb2_function_resources *parent_func;
+	struct dlb2_list_entry func_list;
+	struct dlb2_list_head used_ldb_queues;
+	struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head used_dir_pq_pairs;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb2_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb2_bitmap;
+
+struct dlb2_function_resources {
+	struct dlb2_list_head avail_domains;
+	struct dlb2_list_head used_domains;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	struct dlb2_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+	struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+	struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+	struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
+struct dlb2_mbox {
+	u32 *mbox;
+	u32 *isr_in_progress;
+};
+
+struct dlb2_sw_mbox {
+	struct dlb2_mbox vdev_to_pf;
+	struct dlb2_mbox pf_to_vdev;
+	void (*pf_to_vdev_inject)(void *arg);
+	void *pf_to_vdev_inject_arg;
+};
+
+struct dlb2_hw {
+	/* BAR 0 address */
+	void  *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void  *func_kva;
+	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb2_hw_resources rsrcs;
+	struct dlb2_function_resources pf;
+	struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+	struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+	/* Virtualization */
+	int virt_mode;
+	struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
+	unsigned int pasid[DLB2_MAX_NUM_VDEVS];
+};
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
new file mode 100644
index 0000000..ce462c0
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
@@ -0,0 +1,596 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_BASE_DLB2_MBOX_H
+#define __DLB2_BASE_DLB2_MBOX_H
+
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+
+#define DLB2_MBOX_INTERFACE_VERSION 1
+
+/*
+ * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
+ * to send requests of its own (e.g. notifying a VF of an impending FLR).
+ * To avoid communication race conditions, e.g. the PF sends a response and then
+ * sends a request before the VF reads the response, the PF->VF mailbox is
+ * divided into two sections:
+ * - Bytes 0-47: PF responses
+ * - Bytes 48-63: PF requests
+ *
+ * Partitioning the PF->VF mailbox allows responses and requests to occupy the
+ * mailbox simultaneously.
+ */
+#define DLB2_PF2VF_RESP_BYTES	  48
+#define DLB2_PF2VF_RESP_BASE	  0
+#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
+
+#define DLB2_PF2VF_REQ_BYTES	  16
+#define DLB2_PF2VF_REQ_BASE	  (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
+#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
+
+/*
+ * Similarly, the VF->PF mailbox is divided into two sections:
+ * - Bytes 0-239: VF requests
+ * -- (Bytes 0-3 are unused due to a hardware errata)
+ * - Bytes 240-255: VF responses
+ */
+#define DLB2_VF2PF_REQ_BYTES	 236
+#define DLB2_VF2PF_REQ_BASE	 4
+#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
+
+#define DLB2_VF2PF_RESP_BYTES	  16
+#define DLB2_VF2PF_RESP_BASE	  (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
+#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
+
+/* VF-initiated commands */
+enum dlb2_mbox_cmd_type {
+	DLB2_MBOX_CMD_REGISTER,
+	DLB2_MBOX_CMD_UNREGISTER,
+	DLB2_MBOX_CMD_GET_NUM_RESOURCES,
+	DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
+	DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
+	DLB2_MBOX_CMD_CREATE_LDB_PORT,
+	DLB2_MBOX_CMD_CREATE_DIR_PORT,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT,
+	DLB2_MBOX_CMD_DISABLE_LDB_PORT,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT,
+	DLB2_MBOX_CMD_DISABLE_DIR_PORT,
+	DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_MAP_QID,
+	DLB2_MBOX_CMD_UNMAP_QID,
+	DLB2_MBOX_CMD_START_DOMAIN,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
+	DLB2_MBOX_CMD_ARM_CQ_INTR,
+	DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
+	DLB2_MBOX_CMD_GET_SN_ALLOCATION,
+	DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
+	DLB2_MBOX_CMD_GET_COS_BW,
+	DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
+	DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_QE_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_CMD_TYPES,
+};
+
+static const char dlb2_mbox_cmd_type_strings[][128] = {
+	"DLB2_MBOX_CMD_REGISTER",
+	"DLB2_MBOX_CMD_UNREGISTER",
+	"DLB2_MBOX_CMD_GET_NUM_RESOURCES",
+	"DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_LDB_PORT",
+	"DLB2_MBOX_CMD_CREATE_DIR_PORT",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_DISABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_DISABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_MAP_QID",
+	"DLB2_MBOX_CMD_UNMAP_QID",
+	"DLB2_MBOX_CMD_START_DOMAIN",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
+	"DLB2_MBOX_CMD_ARM_CQ_INTR",
+	"DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
+	"DLB2_MBOX_CMD_GET_SN_ALLOCATION",
+	"DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
+	"DLB2_MBOX_CMD_GET_COS_BW",
+	"DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
+	"DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
+};
+
+/* PF-initiated commands */
+enum dlb2_mbox_vf_cmd_type {
+	DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
+	DLB2_MBOX_VF_CMD_NOTIFICATION,
+	DLB2_MBOX_VF_CMD_IN_USE,
+
+	/* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_VF_CMD_TYPES,
+};
+
+static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
+	"DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
+	"DLB2_MBOX_VF_CMD_NOTIFICATION",
+	"DLB2_MBOX_VF_CMD_IN_USE",
+};
+
+#define DLB2_MBOX_CMD_TYPE(hdr) \
+	(((struct dlb2_mbox_req_hdr *)hdr)->type)
+#define DLB2_MBOX_CMD_STRING(hdr) \
+	dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
+
+enum dlb2_mbox_status_type {
+	DLB2_MBOX_ST_SUCCESS,
+	DLB2_MBOX_ST_INVALID_CMD_TYPE,
+	DLB2_MBOX_ST_VERSION_MISMATCH,
+	DLB2_MBOX_ST_INVALID_OWNER_VF,
+};
+
+static const char dlb2_mbox_status_type_strings[][128] = {
+	"DLB2_MBOX_ST_SUCCESS",
+	"DLB2_MBOX_ST_INVALID_CMD_TYPE",
+	"DLB2_MBOX_ST_VERSION_MISMATCH",
+	"DLB2_MBOX_ST_INVALID_OWNER_VF",
+};
+
+#define DLB2_MBOX_ST_TYPE(hdr) \
+	(((struct dlb2_mbox_resp_hdr *)hdr)->status)
+#define DLB2_MBOX_ST_STRING(hdr) \
+	dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
+
+/* This structure is always the first field in a request structure */
+struct dlb2_mbox_req_hdr {
+	u32 type;
+};
+
+/* This structure is always the first field in a response structure */
+struct dlb2_mbox_resp_hdr {
+	u32 status;
+};
+
+struct dlb2_mbox_register_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 min_interface_version;
+	u16 max_interface_version;
+};
+
+struct dlb2_mbox_register_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 interface_version;
+	u8 pf_id;
+	u8 vf_id;
+	u8 is_auxiliary_vf;
+	u8 primary_vf_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u16 num_sched_domains;
+	u16 num_ldb_queues;
+	u16 num_ldb_ports;
+	u16 num_cos_ldb_ports[4];
+	u16 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 max_contiguous_hist_list_entries;
+	u16 num_ldb_credits;
+	u16 num_dir_credits;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 num_ldb_queues;
+	u32 num_ldb_ports;
+	u32 num_cos_ldb_ports[4];
+	u32 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u8 cos_strict;
+	u8 padding0[3];
+	u32 padding1;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 num_sequence_numbers;
+	u32 num_qid_inflights;
+	u32 num_atomic_inflights;
+	u32 lock_id_comp_level;
+	u32 depth_threshold;
+	u32 padding;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 depth_threshold;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u16 cq_depth;
+	u16 cq_history_list_size;
+	u8 cos_id;
+	u8 cos_strict;
+	u16 padding1;
+	u64 cq_base_address;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u64 cq_base_address;
+	u16 cq_depth;
+	u16 padding0;
+	s32 queue_id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_map_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+	u32 priority;
+	u32 padding0;
+};
+
+struct dlb2_mbox_map_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_start_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+};
+
+struct dlb2_mbox_start_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 is_ldb;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding0;
+};
+
+/*
+ * The alert_id and aux_alert_data follows the format of the alerts defined in
+ * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
+ * the aux_alert_data value varies depending on the alert.
+ */
+struct dlb2_mbox_vf_alert_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 alert_id;
+	u32 aux_alert_data;
+};
+
+enum dlb2_mbox_vf_notification_type {
+	DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
+	DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
+
+	/* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
+	NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
+};
+
+struct dlb2_mbox_vf_notification_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 notification;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 in_use;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 num;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 cos_id;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 mode;
+};
+
+#endif /* __DLB2_BASE_DLB2_MBOX_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
new file mode 100644
index 0000000..c4c34eb
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
@@ -0,0 +1,230 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_H
+#define __DLB2_OSDEP_H
+
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <rte_string_fns.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include "../dlb2_main.h"
+#include "dlb2_resource.h"
+#include "../../dlb2_log.h"
+#include "../../dlb2_user.h"
+
+
+#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
+#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
+#define DLB2_CSR_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
+#define DLB2_FUNC_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
+/* Map to PMDs logging interface */
+#define DLB2_ERR(dev, fmt, args...) \
+	DLB2_LOG_ERR(fmt, ## args)
+
+#define DLB2_INFO(dev, fmt, args...) \
+	DLB2_LOG_INFO(fmt, ## args)
+
+#define DLB2_DEBUG(dev, fmt, args...) \
+	DLB2_LOG_DBG(fmt, ## args)
+
+/**
+ * os_udelay() - busy-wait for a number of microseconds
+ * @usecs: delay duration.
+ */
+static inline void os_udelay(int usecs)
+{
+	rte_delay_us(usecs);
+}
+
+/**
+ * os_msleep() - sleep for a number of milliseconds
+ * @usecs: delay duration.
+ */
+static inline void os_msleep(int msecs)
+{
+	rte_delay_ms(msecs);
+}
+
+#define DLB2_PP_BASE(__is_ldb) \
+	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
+
+/**
+ * os_map_producer_port() - map a producer port into the caller's address space
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ *
+ * This function maps the requested producer port memory into the caller's
+ * address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void *os_map_producer_port(struct dlb2_hw *hw,
+					 u8 port_id,
+					 bool is_ldb)
+{
+	uint64_t addr;
+	uint64_t pp_dma_base;
+
+	pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
+	addr = (pp_dma_base + (PAGE_SIZE * port_id));
+
+	return (void *)(uintptr_t)addr;
+}
+
+/**
+ * os_unmap_producer_port() - unmap a producer port
+ * @addr: mapped producer port address
+ *
+ * This function undoes os_map_producer_port() by unmapping the producer port
+ * memory from the caller's address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(addr);
+}
+
+/**
+ * os_fence_hcw() - fence an HCW to ensure it arrives at the device
+ * @hw: dlb2_hw handle for a particular device.
+ * @pp_addr: producer port address
+ */
+static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
+{
+	RTE_SET_USED(hw);
+
+	/* To ensure outstanding HCWs reach the device, read the PP address. IA
+	 * memory ordering prevents reads from passing older writes, and the
+	 * mfence also ensures this.
+	 */
+	rte_mb();
+
+	*(volatile u64 *)pp_addr;
+}
+
+/**
+ * DLB2_HW_ERR() - log an error message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_ERR(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_ERR(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/**
+ * DLB2_HW_DBG() - log an info message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_DBG(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_DEBUG(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/* The callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void *dlb2_complete_queue_map_unmap(void *__args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
+	int ret;
+
+	while (1) {
+		rte_spinlock_lock(&dlb2_dev->resource_mutex);
+
+		ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+		ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+		if (ret != 0) {
+			rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+			/* Relinquish the CPU so the application can process
+			 * its CQs, so this function doesn't deadlock.
+			 */
+			sched_yield();
+		} else {
+			break;
+		}
+	}
+
+	dlb2_dev->worker_launched = false;
+
+	rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+
+	return NULL;
+}
+
+
+/**
+ * os_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static inline void os_schedule_work(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+	pthread_t complete_queue_map_unmap_thread;
+	int ret;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
+				     "dlb_queue_unmap_waiter",
+				     NULL,
+				     dlb2_complete_queue_map_unmap,
+				     dlb2_dev);
+	if (ret)
+		DLB2_ERR(dlb2_dev,
+			 "Could not create queue complete map/unmap thread, err=%d\n",
+			 ret);
+	else
+		dlb2_dev->worker_launched = true;
+}
+
+/**
+ * os_worker_active() - query whether the map/unmap worker thread is active
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether a thread (launched by
+ * os_schedule_work()) is active. This function is used to determine
+ * whether or not to launch a worker thread.
+ */
+static inline bool os_worker_active(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	return dlb2_dev->worker_launched;
+}
+
+#endif /*  __DLB2_OSDEP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
new file mode 100644
index 0000000..423233b
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <rte_bitmap.h>
+#include <rte_string_fns.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include "../dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+	struct rte_bitmap *map;
+	unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+				    unsigned int len)
+{
+	struct dlb2_bitmap *bm;
+	void *mem;
+	uint32_t alloc_size;
+	uint32_t nbits = (uint32_t)len;
+
+	if (bitmap == NULL || nbits == 0)
+		return -EINVAL;
+
+	/* Allocate DLB2 bitmap control struct */
+	bm = rte_malloc("DLB2_PF",
+			sizeof(struct dlb2_bitmap),
+			RTE_CACHE_LINE_SIZE);
+
+	if (bm == NULL)
+		return -ENOMEM;
+
+	/* Allocate bitmap memory */
+	alloc_size = rte_bitmap_get_memory_footprint(nbits);
+	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
+	if (mem == NULL) {
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->map = rte_bitmap_init(len, mem, alloc_size);
+	if (bm->map == NULL) {
+		rte_free(mem);
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap == NULL)
+		return;
+
+	rte_free(bitmap->map);
+	rte_free(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++)
+		rte_bitmap_set(bitmap->map, i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	rte_bitmap_reset(bitmap->map);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set() - set a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
+				  unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_set(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+					unsigned int bit,
+					unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_set(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear() - clear a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
+				    unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_clear(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+					  unsigned int bit,
+					  unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_clear(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+						 unsigned int len)
+{
+	unsigned int i, j = 0;
+
+	if (bitmap  == NULL || bitmap->map == NULL || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			if (++j == len)
+				return i - j + 1;
+		} else {
+			j = 0;
+		}
+	}
+
+	/* No set bit range of length len? */
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			return i;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+	int weight = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			weight++;
+	}
+	return weight;
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				      bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+	int max_len = 0, len = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			len++;
+		} else {
+			if (len > max_len)
+				max_len = len;
+			len = 0;
+		}
+	}
+
+	if (len > max_len)
+		max_len = len;
+
+	return max_len;
+}
+
+/**
+ * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
+ * @dest: pointer to dlb2_bitmap structure, which will contain the results of
+ *	  the 'or' of src1 and src2.
+ * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
+ * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
+ *
+ * This function 'or's two bitmaps together and stores the result in a third
+ * bitmap. The source and destination bitmaps can be the same.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - One of the bitmaps is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
+				 struct dlb2_bitmap *src1,
+				 struct dlb2_bitmap *src2)
+{
+	unsigned int i, min;
+	int numset = 0;
+
+	if (dest  == NULL || dest->map == NULL ||
+	    src1  == NULL || src1->map == NULL ||
+	    src2  == NULL || src2->map == NULL)
+		return -EINVAL;
+
+	min = dest->len;
+	min = (min > src1->len) ? src1->len : min;
+	min = (min > src2->len) ? src2->len : min;
+
+	for (i = 0; i != min; i++) {
+		if  (rte_bitmap_get(src1->map, i) ||
+		     rte_bitmap_get(src2->map, i)) {
+			rte_bitmap_set(dest->map, i);
+			numset++;
+		} else {
+			rte_bitmap_clear(dest->map, i);
+		}
+	}
+
+	return numset;
+}
+
+#endif /*  __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
new file mode 100644
index 0000000..5531739
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_LIST_H
+#define __DLB2_OSDEP_LIST_H
+
+#include <rte_tailq.h>
+
+struct dlb2_list_entry {
+	TAILQ_ENTRY(dlb2_list_entry) node;
+};
+
+/* Dummy - just a struct definition */
+TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
+
+/* =================
+ * TAILQ Supplements
+ * =================
+ */
+
+#ifndef TAILQ_FOREACH_ENTRY
+#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)		\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter)						\
+		&& (ptr = container_of(iter, typeof(*(ptr)), name)); \
+	    (iter) = TAILQ_NEXT((iter), node))
+#endif
+
+#ifndef TAILQ_FOREACH_ENTRY_SAFE
+#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)	\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter) &&						\
+		(ptr = container_of(iter, typeof(*(ptr)), name)) &&\
+		((tvar) = TAILQ_NEXT((iter), node), 1);	\
+	    (iter) = (tvar))
+#endif
+
+/***********************/
+/*** List operations ***/
+/***********************/
+
+/**
+ * dlb2_list_init_head() - initialize the head of a list
+ * @head: list head
+ */
+static inline void dlb2_list_init_head(struct dlb2_list_head *head)
+{
+	TAILQ_INIT(head);
+}
+
+/**
+ * dlb2_list_add() - add an entry to a list
+ * @head: list head
+ * @entry: new list entry
+ */
+static inline void
+dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
+{
+	TAILQ_INSERT_TAIL(head, entry, node);
+}
+
+/**
+ * dlb2_list_del() - delete an entry from a list
+ * @entry: list entry
+ * @head: list head
+ */
+static inline void dlb2_list_del(struct dlb2_list_head *head,
+				 struct dlb2_list_entry *entry)
+{
+	TAILQ_REMOVE(head, entry, node);
+}
+
+/**
+ * dlb2_list_empty() - check if a list is empty
+ * @head: list head
+ *
+ * Return:
+ * Returns 1 if empty, 0 if not.
+ */
+static inline int dlb2_list_empty(struct dlb2_list_head *head)
+{
+	return TAILQ_EMPTY(head);
+}
+
+/**
+ * dlb2_list_splice() - splice a list
+ * @src_head: list to be added
+ * @ head: where src_head will be inserted
+ */
+static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
+				    struct dlb2_list_head *head)
+{
+	TAILQ_CONCAT(head, src_head, node);
+}
+
+/**
+ * DLB2_LIST_HEAD() - retrieve the head of the list
+ * @head: list head
+ * @type: type of the list variable
+ * @name: name of the list field within the containing struct
+ */
+#define DLB2_LIST_HEAD(head, type, name)                       \
+	(TAILQ_FIRST(&head) ?					\
+		container_of(TAILQ_FIRST(&head), type, name) :	\
+		NULL)
+
+/**
+ * DLB2_LIST_FOR_EACH() - iterate over a list
+ * @head: list head
+ * @ptr: pointer to struct containing a struct list
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ */
+#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
+	TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
+
+/**
+ * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
+ * an element is removed from the list while processing it.
+ * @ptr: pointer to struct containing a struct list
+ * @ptr_tmp: pointer to struct containing a struct list (temporary)
+ * @head: list head
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ * @iter_tmp: iterator variable (temporary)
+ */
+#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
+	TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
+
+#endif /*  __DLB2_OSDEP_LIST_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
new file mode 100644
index 0000000..0a48f7e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_TYPES_H
+#define __DLB2_OSDEP_TYPES_H
+
+#include <linux/types.h>
+
+#include <inttypes.h>
+#include <ctype.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Types for user mode PF PMD */
+typedef uint8_t         u8;
+typedef int8_t          s8;
+typedef uint16_t        u16;
+typedef int16_t         s16;
+typedef uint32_t        u32;
+typedef int32_t         s32;
+typedef uint64_t        u64;
+
+#define __iomem
+
+/* END types for user mode PF PMD */
+
+#endif /* __DLB2_OSDEP_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
new file mode 100644
index 0000000..43ecad4
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
@@ -0,0 +1,2527 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "dlb2_osdep_types.h"
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+	struct {
+		u32 isr_pend : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+	struct {
+		u32 vf0_reset_in_progress : 1;
+		u32 vf1_reset_in_progress : 1;
+		u32 vf2_reset_in_progress : 1;
+		u32 vf3_reset_in_progress : 1;
+		u32 vf4_reset_in_progress : 1;
+		u32 vf5_reset_in_progress : 1;
+		u32 vf6_reset_in_progress : 1;
+		u32 vf7_reset_in_progress : 1;
+		u32 vf8_reset_in_progress : 1;
+		u32 vf9_reset_in_progress : 1;
+		u32 vf10_reset_in_progress : 1;
+		u32 vf11_reset_in_progress : 1;
+		u32 vf12_reset_in_progress : 1;
+		u32 vf13_reset_in_progress : 1;
+		u32 vf14_reset_in_progress : 1;
+		u32 vf15_reset_in_progress : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+	struct {
+		u32 vec_mask : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+	struct {
+		u32 func_vf_bar_dis : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+	struct {
+		u32 total_vas : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+	struct {
+		u32 total_dir_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+	struct {
+		u32 total_ldb_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+	struct {
+		u32 total_dir_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+	struct {
+		u32 total_ldb_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+	struct {
+		u32 total_dir_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+	struct {
+		u32 total_ldb_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 meas : 1;
+		u32 debug : 7;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 cq_int_rearm : 1;
+		u32 dsi_error : 1;
+		u32 rsvd0 : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 rsvd0 : 3;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+	struct {
+		u32 qid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+	struct {
+		u32 vqid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+	struct {
+		u32 qid : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 debug : 8;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 isz : 1;
+		u32 dsi_error : 1;
+		u32 dlbrsvd : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 vf_synd0_parity : 1;
+		u32 vf_synd1_parity : 1;
+		u32 vf_synd2_parity : 1;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+	struct {
+		u32 sn_cfg_v : 1;
+		u32 fid_cfg_v : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS  0
+#define DLB2_CQ_ISR_MODE_MSI  1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI  3
+union dlb2_sys_ldb_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+	struct {
+		u32 keep_pf_ppid : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+	struct {
+		u32 illegal_hcw : 1;
+		u32 illegal_pp : 1;
+		u32 illegal_pasid : 1;
+		u32 illegal_qid : 1;
+		u32 disabled_qid : 1;
+		u32 illegal_ldb_qid_cfg : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+	struct {
+		u32 msix_0_ack : 1;
+		u32 msix_1_ack : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+	struct {
+		u32 msix_0_passthru : 1;
+		u32 msix_1_passthru : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED     0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+	struct {
+		u32 mode : 1;
+		u32 poll_mode : 1;
+		u32 poll_mask : 1;
+		u32 poll_lock : 1;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+	struct {
+		u32 cq : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 alarm : 1;
+		u32 cwd : 1;
+		u32 vf_pf_mb : 1;
+		u32 rsvd0 : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+	struct {
+		u32 qid_fid_limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+	struct {
+		u32 compress_code : 3;
+		u32 rsvd0 : 29;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+	(DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+	struct {
+		u32 count : 15;
+		u32 rsvd0 : 17;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+	struct {
+		u32 sn : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+	struct {
+		u32 mode : 3;
+		u32 slot : 4;
+		u32 rsvz0 : 1;
+		u32 grp : 1;
+		u32 rsvz1 : 1;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+	struct {
+		u32 en : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+	struct {
+		u32 write_pointer : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+	struct {
+		u32 base : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+	struct {
+		u32 limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+	struct {
+		u32 pop_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+	struct {
+		u32 push_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+	struct {
+		u32 depth : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+	struct {
+		u32 write_pointer : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 dlb_cor_alarm_enable : 1;
+		u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+		u32 cfg_64bytes_qe_dir_cq_mode : 1;
+		u32 pad_write_ldb : 1;
+		u32 pad_write_dir : 1;
+		u32 pad_first_write_ldb : 1;
+		u32 pad_first_write_dir : 1;
+		u32 rsvz0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+	struct {
+		u32 egress_credit_status_empty : 1;
+		u32 egress_credit_status_afull : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+		u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+		u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+		u32 chp_rop_pipe_credit_status_empty : 1;
+		u32 chp_rop_pipe_credit_status_afull : 1;
+		u32 qed_to_cq_pipe_credit_status_empty : 1;
+		u32 qed_to_cq_pipe_credit_status_afull : 1;
+		u32 egress_lsp_token_credit_status_empty : 1;
+		u32 egress_lsp_token_credit_status_afull : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 rsvz0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+	struct {
+		u32 sn_mode_0 : 3;
+		u32 rszv0 : 5;
+		u32 sn_mode_1 : 3;
+		u32 rszv1 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+	struct {
+		u32 unit_single_step_mode : 1;
+		u32 rr_en : 1;
+		u32 rszv0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+	struct {
+		u32 prio : 24;
+		u32 v : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+	struct {
+		u32 qid_p0 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p1 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p2 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p3 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+	struct {
+		u32 qid_p4 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p5 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p6 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p7 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+	struct {
+		u32 token_depth_select : 4;
+		u32 disable_wb_opt : 1;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+	struct {
+		u32 token_count : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+	struct {
+		u32 thresh : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+	(DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+	(DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+	struct {
+		u32 depth : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+	struct {
+		u32 cq : 8;
+		u32 qidix : 3;
+		u32 value : 1;
+		u32 nalb_haswork_v : 1;
+		u32 rlist_haswork_v : 1;
+		u32 slist_haswork_v : 1;
+		u32 inflight_ok_v : 1;
+		u32 aqed_nfull_v : 1;
+		u32 rsvz0 : 15;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+	struct {
+		u32 transfer : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+	struct {
+		u32 bw_range : 9;
+		u32 rsvz0 : 22;
+		u32 no_extra_credit : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+	struct {
+		u32 disab_atq_empty_arb : 1;
+		u32 inc_tok_unit_idle : 1;
+		u32 disab_rlist_pri : 1;
+		u32 inc_cmp_unit_idle : 1;
+		u32 rsvz0 : 2;
+		u32 dir_single_op : 1;
+		u32 dir_half_bw : 1;
+		u32 dir_single_out : 1;
+		u32 dir_disab_multi : 1;
+		u32 atq_single_op : 1;
+		u32 atq_half_bw : 1;
+		u32 atq_single_out : 1;
+		u32 atq_disab_multi : 1;
+		u32 dirrpl_single_op : 1;
+		u32 dirrpl_half_bw : 1;
+		u32 dirrpl_single_out : 1;
+		u32 lbrpl_single_op : 1;
+		u32 lbrpl_half_bw : 1;
+		u32 lbrpl_single_out : 1;
+		u32 ldb_single_op : 1;
+		u32 ldb_half_bw : 1;
+		u32 ldb_disab_multi : 1;
+		u32 atm_single_sch : 1;
+		u32 atm_single_cmp : 1;
+		u32 ldb_ce_tog_arb : 1;
+		u32 rsvz1 : 1;
+		u32 smon0_valid_sel : 2;
+		u32 smon0_value_sel : 1;
+		u32 smon0_compare_sel : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+	struct {
+		u32 chp_pf_reset_done : 1;
+		u32 rop_pf_reset_done : 1;
+		u32 lsp_pf_reset_done : 1;
+		u32 nalb_pf_reset_done : 1;
+		u32 ap_pf_reset_done : 1;
+		u32 dp_pf_reset_done : 1;
+		u32 qed_pf_reset_done : 1;
+		u32 dqed_pf_reset_done : 1;
+		u32 aqed_pf_reset_done : 1;
+		u32 sys_pf_reset_done : 1;
+		u32 pf_reset_active : 1;
+		u32 flrsm_state : 7;
+		u32 rsvd0 : 13;
+		u32 dlb_proc_reset_done : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+	struct {
+		u32 chp_pipeidle : 1;
+		u32 rop_pipeidle : 1;
+		u32 lsp_pipeidle : 1;
+		u32 nalb_pipeidle : 1;
+		u32 ap_pipeidle : 1;
+		u32 dp_pipeidle : 1;
+		u32 qed_pipeidle : 1;
+		u32 dqed_pipeidle : 1;
+		u32 aqed_pipeidle : 1;
+		u32 sys_pipeidle : 1;
+		u32 chp_unit_idle : 1;
+		u32 rop_unit_idle : 1;
+		u32 lsp_unit_idle : 1;
+		u32 nalb_unit_idle : 1;
+		u32 ap_unit_idle : 1;
+		u32 dp_unit_idle : 1;
+		u32 qed_unit_idle : 1;
+		u32 dqed_unit_idle : 1;
+		u32 aqed_unit_idle : 1;
+		u32 sys_unit_idle : 1;
+		u32 rsvd1 : 4;
+		u32 mstr_cfg_ring_idle : 1;
+		u32 mstr_cfg_mstr_idle : 1;
+		u32 mstr_flr_clkreq_b : 1;
+		u32 mstr_proc_idle : 1;
+		u32 mstr_proc_idle_masked : 1;
+		u32 rsvd0 : 2;
+		u32 dlb_func_idle : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+	struct {
+		u32 prochot : 1;
+		u32 pgcb_dlb_idle : 1;
+		u32 pgcb_dlb_pg_rdy_ack_b : 1;
+		u32 pmsm_pgcb_req_b : 1;
+		u32 pgbc_pmc_pg_req_b : 1;
+		u32 pmc_pgcb_pg_ack_b : 1;
+		u32 pmc_pgcb_fet_en_b : 1;
+		u32 pgcb_fet_en_b : 1;
+		u32 rsvz0 : 1;
+		u32 rsvz1 : 1;
+		u32 fuse_force_on : 1;
+		u32 fuse_proc_disable : 1;
+		u32 rsvz2 : 1;
+		u32 rsvz3 : 1;
+		u32 pm_fsm_d0tod3_ok : 1;
+		u32 pm_fsm_d3tod0_ok : 1;
+		u32 dlb_in_d3 : 1;
+		u32 rsvz4 : 7;
+		u32 pmsm : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+	struct {
+		u32 disable : 1;
+		u32 rsvz0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+	struct {
+		u32 isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+	struct {
+		u32 pf_isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+	struct {
+		u32 isr_pend : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+	struct {
+		u32 reset_in_progress : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+	struct {
+		u32 vf_msi_isr : 32;
+	} field;
+	u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
new file mode 100644
index 0000000..503fdf3
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
@@ -0,0 +1,1913 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_osdep_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_start_domain() - start a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: start domain arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int dlb2_hw_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp,
+			 bool vdev_request,
+			 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_request,
+		    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding unmap procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_finish_map_qid_procedures() - finish any pending map procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding map procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ *					interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+			  int port_id,
+			  bool is_ldb,
+			  bool vdev_request,
+			  unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+					 u32 *ldb_interrupts,
+					 u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+				 u32 *ldb_interrupts,
+				 u32 *dir_interrupts);
+
+/**
+ * dlb2_read_vf_intr_status() - read the VF interrupt status register
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function can be called from a VF's interrupt handler to determine
+ * which interrupts have fired. The first 31 bits correspond to CQ interrupt
+ * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
+ *
+ * Return:
+ * Returns a bit vector indicating which interrupt vectors are active.
+ */
+u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_intr_status() - ack VF interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function ACKs a VF's interrupts. Its interrupts argument should be the
+ * value returned by dlb2_read_vf_intr_status().
+ */
+void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function clears the VF's MSI interrupt pending register. Its interrupts
+ * argument should be contain the MSI vectors to ACK. For example, if MSI MME
+ * is in mode 0, then one bit 0 should ever be set.
+ */
+void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When done processing the PF mailbox request, this function unsets
+ * the PF's mailbox ISR register.
+ */
+void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
+
+/**
+ * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
+ *					vdevs
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the vdev->PF ISR fires, this function can be called to determine which
+ * vdev(s) are requesting service. This bitvector must be passed to
+ * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
+ * vdevs.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested service.
+ */
+u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ *
+ * When done processing all VF mailbox requests, this function unsets the VF's
+ * mailbox ISR register.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
+ *				    FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the VF FLR ISR fires, this function can be called to determine which
+ * VF(s) are requesting FLRs. This bitvector must passed to
+ * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
+ */
+u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF FLR requests, this function unsets the VF's FLR
+ * ISR register.
+ */
+void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF requests, this function communicates to the
+ * hardware that processing is complete.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
+			     u32 mbox_bitvec,
+			     u32 flr_bitvec);
+
+/**
+ * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the watchdog timer interrupt cause registers to
+ * determine which port(s) had a watchdog timeout, and notifies the
+ * application(s) that own the port(s).
+ */
+void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads and logs the alarm syndrome, then acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_request,
+			      unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of resources in use by the PF or a VF. It
+ * fills in the fields that args points to, except the following:
+ * - max_contiguous_atomic_inflights
+ * - max_contiguous_hist_list_entries
+ * - max_contiguous_ldb_credits
+ * - max_contiguous_dir_credits
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
+				   struct dlb2_get_num_resources_args *arg,
+				   bool vdev_request,
+				   unsigned int vdev_id);
+
+/**
+ * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
+ *				       the PF
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function sends a VF->PF mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the PF has processed
+ * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
+ * when the PF has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
+ *				 request
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * processing a VF->PF mailbox request. It should only be called after sending
+ * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vf_flr_complete() - check the status of a VF FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * executing the VF FLR. It should only be called after setting the VF's FLR
+ * bit.
+ */
+bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
+ *					vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function sends a PF->vdev mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the vdev has
+ * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
+ * determine when the vdev has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
+ *			       request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function returns a boolean indicating whether the vdev has finished
+ * processing a PF->vdev mailbox request. It should only be called after
+ * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
+			     unsigned int vf_id,
+			     void *data,
+			     int len);
+
+/**
+ * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
+			       unsigned int vf_id,
+			       void *data,
+			       int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_reset_vdev() - reset the hardware owned by a virtual device
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function resets the hardware owned by a vdev, by resetting the vdev's
+ * domains one by one.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function returns whether or not the vdev's resource assignments are
+ * locked. If locked, no resources can be added to or subtracted from the
+ * group.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_lock_vdev() - lock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sets a flag indicating that the vdev is using its resources.
+ * When the vdev is locked, its resource assignment cannot be changed.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_unlock_vdev() - unlock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function unlocks the vdev's resource assignment, allowing it to be
+ * modified.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of scheduling domains to assign to this vdev
+ *
+ * This function assigns num scheduling domains to the specified vdev. If the
+ * vdev already has domains assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB queues to assign to this vdev
+ *
+ * This function assigns num LDB queues to the specified vdev. If the vdev
+ * already has LDB queues assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports to the specified vdev. If the vdev
+ * already has LDB ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @cos: class-of-service ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports from class-of-service cos to the
+ * specified vdev. If the vdev already has LDB ports from this class-of-service
+ * assigned, this existing assignment is adjusted accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
+				   u32 id,
+				   u32 cos,
+				   u32 num);
+
+/**
+ * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR ports to assign to this vdev
+ *
+ * This function assigns num DIR ports to the specified vdev. If the vdev
+ * already has DIR ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB credit credits to assign to this vdev
+ *
+ * This function assigns num LDB credit to the specified vdev. If the vdev
+ * already has LDB credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR credits to assign to this vdev
+ *
+ * This function assigns num DIR credit to the specified vdev. If the vdev
+ * already has DIR credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of history list entries to assign to this vdev
+ *
+ * This function assigns num history list entries to the specified vdev. If the
+ * vdev already has history list entries assigned, this existing assignment is
+ * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
+ * this function may fail if a sufficiently large contiguous chunk is not
+ * available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of atomic inflights to assign to this vdev
+ *
+ * This function assigns num atomic inflights to the specified vdev. If the vdev
+ * already has atomic inflights assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of entries, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function takes any resources currently assigned to the vdev and
+ * reassigns them to the PF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_notify_vf() - send an alarm to a VF
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID
+ * @notification: notification
+ *
+ * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
+ *
+ * Return:
+ * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
+ */
+int dlb2_notify_vf(struct dlb2_hw *hw,
+		   unsigned int vf_id,
+		   u32 notification);
+
+/**
+ * dlb2_vdev_in_use() - query whether a virtual device is in use
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sends a mailbox request to the vdev to query whether the vdev
+ * is in use.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
+ * an internal error occurs.
+ */
+int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+enum dlb2_virt_mode {
+	DLB2_VIRT_NONE,
+	DLB2_VIRT_SRIOV,
+	DLB2_VIRT_SIOV,
+
+	/* NUM_DLB2_VIRT_MODES must be last */
+	NUM_DLB2_VIRT_MODES,
+};
+
+/**
+ * dlb2_hw_set_virt_mode() - set the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: either none, SR-IOV, or Scalable IOV.
+ *
+ * This function sets the virtualization mode of the device. This controls
+ * whether the device uses a software or hardware mailbox.
+ *
+ * This should be called by the PF driver when either SR-IOV or Scalable IOV is
+ * selected as the virtualization mechanism, and by the VF/VDEV driver during
+ * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
+ *
+ * Errors:
+ * EINVAL - Invalid mode.
+ */
+int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
+
+/**
+ * dlb2_hw_get_virt_mode() - get the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function gets the virtualization mode of the device.
+ */
+enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_register_sw_mbox() - register a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
+ * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
+ * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
+ * @inject_arg: user argument for pf2vdev_inject callback.
+ *
+ * When Scalable IOV is enabled, the VDCM must register a software mailbox for
+ * every virtual device during vdev creation.
+ *
+ * This function notifies the driver to use a software mailbox using the
+ * provided pointers, instead of the device's hardware mailbox. When the driver
+ * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
+ * go to the software mailbox instead of the hardware one. This is used in
+ * Scalable IOV virtualization.
+ */
+void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
+			      unsigned int vdev_id,
+			      u32 *vdev2pf_mbox,
+			      u32 *pf2vdev_mbox,
+			      void (*pf2vdev_inject)(void *),
+			      void *inject_arg);
+
+/**
+ * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function notifies the driver to stop using a previously registered
+ * software mailbox.
+ */
+void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ * @addr_lo: least-significant 32 bits of address.
+ * @data: 32 data bits.
+ *
+ * This sets up the CQ's IMS entry with the provided address and data values.
+ * This function should only be called if the device is configured for Scalable
+ * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
+ * not needed.
+ */
+void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb,
+				u32 addr_lo,
+				u32 data);
+
+/**
+ * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ *
+ * This clears the CQ's IMS entry, reverting it to its reset state.
+ */
+void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb);
+
+/**
+ * dlb2_hw_register_pasid() - register a vdev's PASID
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @pasid: the vdev's PASID.
+ *
+ * This function stores the user-supplied PASID, and uses it when configuring
+ * the vdev's CQs.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+int dlb2_hw_register_pasid(struct dlb2_hw *hw,
+			   unsigned int vdev_id,
+			   unsigned int pasid);
+
+/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vf_request,
+				unsigned int vf_id);
+
+/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ *	to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ *	port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ *	    cause the total bandwidth across all classes of service to exceed
+ *	    100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+enum dlb2_wd_tmo {
+	/* 40s watchdog timeout */
+	DLB2_WD_TMO_40S,
+	/* 10s watchdog timeout */
+	DLB2_WD_TMO_10S,
+	/* 1s watchdog timeout */
+	DLB2_WD_TMO_1S,
+
+	/* Must be last */
+	NUM_DLB2_WD_TMOS,
+};
+
+/**
+ * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
+ *	caller-specified timeout.
+ * @hw: dlb2_hw handle for a particular device.
+ * @tmo: watchdog timeout.
+ *
+ * This function should be called during device initialization and after reset.
+ * The watchdog timer interrupt must also be enabled per-CQ, using either
+ * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid timeout.
+ */
+int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
+
+/**
+ * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid directed port ID.
+ */
+int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid load-balanced port ID.
+ */
+int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *	ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * load-balanced CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_ldb_cq_interrupt().
+ */
+void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical directed port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * directed CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_dir_cq_interrupt().
+ */
+void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+#endif /* __DLB2_RESOURCE_H */
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
new file mode 100644
index 0000000..3c51cab
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -0,0 +1,621 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "base/dlb2_resource.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_regs.h"
+#include "dlb2_main.h"
+#include "../dlb2_user.h"
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+
+#define PF_ID_ZERO 0	/* PF ONLY! */
+#define NO_OWNER_VF 0	/* PF ONLY! */
+#define NOT_VF_REQ false /* PF ONLY! */
+
+#define DLB2_PCI_CFG_SPACE_SIZE 256
+#define DLB2_PCI_CAP_POINTER 0x34
+#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
+#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
+#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
+#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
+#define DLB2_PCI_EXT_CAP_ID_ERR 1
+#define DLB2_PCI_ERR_UNCOR_MASK 8
+#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
+
+#define DLB2_PCI_EXP_DEVCTL 8
+#define DLB2_PCI_LNKCTL 16
+#define DLB2_PCI_SLTCTL 24
+#define DLB2_PCI_RTCTL 28
+#define DLB2_PCI_EXP_DEVCTL2 40
+#define DLB2_PCI_LNKCTL2 48
+#define DLB2_PCI_SLTCTL2 56
+#define DLB2_PCI_CMD 4
+#define DLB2_PCI_X_CMD 2
+#define DLB2_PCI_EXP_DEVSTA 10
+#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
+#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
+
+#define DLB2_PCI_CAP_ID_EXP       0x10
+#define DLB2_PCI_CAP_ID_MSIX      0x11
+#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
+#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
+#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
+
+#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
+#define DLB2_PCI_PRI_ALLOC_REQ           0xC
+#define DLB2_PCI_PRI_CTRL                0x4
+#define DLB2_PCI_MSIX_FLAGS              0x2
+#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
+#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
+#define DLB2_PCI_ERR_ROOT_STATUS         0x30
+#define DLB2_PCI_ERR_COR_STATUS          0x10
+#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
+#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
+#define DLB2_PCI_ACS_CAP                 0x4
+#define DLB2_PCI_ACS_CTRL                0x6
+#define DLB2_PCI_ACS_SV                  0x1
+#define DLB2_PCI_ACS_RR                  0x4
+#define DLB2_PCI_ACS_CR                  0x8
+#define DLB2_PCI_ACS_UF                  0x10
+#define DLB2_PCI_ACS_EC                  0x20
+
+/* Stubs: Allow building partial probe patch */
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	int ret = 0;
+	RTE_SET_USED(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+/* End stubs */
+
+static int
+dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint32_t hdr;
+	size_t sz;
+	int pos;
+
+	pos = DLB2_PCI_CFG_SPACE_SIZE;
+	sz = sizeof(hdr);
+
+	while (pos > 0xFF) {
+		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
+			return -1;
+
+		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
+			return pos;
+
+		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint8_t pos;
+	int ret;
+	uint16_t hdr;
+
+	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
+	pos &= 0xFC;
+
+	if (ret != 1)
+		return -1;
+
+	while (pos > 0x3F) {
+		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
+		if (ret != 2)
+			return -1;
+
+		if (DLB2_PCI_CAP_ID(hdr) == id)
+			return pos;
+
+		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
+			return -1;
+
+		pos = DLB2_PCI_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+	rte_spinlock_init(&dlb2_dev->resource_mutex);
+	rte_spinlock_init(&dlb2_dev->measurement_lock);
+
+	return 0;
+}
+
+static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
+{
+	u32 retries = 0;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
+		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+		union dlb2_cfg_mstr_cfg_pm_status pm_st;
+		u32 addr;
+
+		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+			break;
+
+		rte_delay_ms(1);
+	};
+
+	if (retries == DLB2_READY_RETRY_LIMIT) {
+		printf("[%s()] wait for device ready timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+struct dlb2_dev *
+dlb2_probe(struct rte_pci_device *pdev)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret = 0;
+
+	DLB2_INFO(dlb2_dev, "probe\n");
+
+	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
+			      RTE_CACHE_LINE_SIZE);
+
+	if (dlb2_dev == NULL) {
+		ret = -ENOMEM;
+		goto dlb2_dev_malloc_fail;
+	}
+
+	/* PCI Bus driver has already mapped bar space into process.
+	 * Save off our IO register and FUNC addresses.
+	 */
+
+	/* BAR 0 */
+	if (pdev->mem_resource[0].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
+	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.func_kva,
+		  (void *)dlb2_dev->hw.func_phys_addr,
+		  (void *)(pdev->mem_resource[0].len));
+
+	/* BAR 2 */
+	if (pdev->mem_resource[2].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
+	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.csr_kva,
+		  (void *)dlb2_dev->hw.csr_phys_addr,
+		  (void *)(pdev->mem_resource[2].len));
+
+	dlb2_dev->pdev = pdev;
+
+	/* PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb2_pf_enable_pm(dlb2_dev);
+
+	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb2_pf_reset(dlb2_dev);
+	if (ret)
+		goto dlb2_reset_fail;
+
+	ret = dlb2_pf_init_driver_state(dlb2_dev);
+	if (ret)
+		goto init_driver_state_fail;
+
+	ret = dlb2_resource_init(&dlb2_dev->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	return dlb2_dev;
+
+resource_init_fail:
+	dlb2_resource_free(&dlb2_dev->hw);
+init_driver_state_fail:
+dlb2_reset_fail:
+pci_mmap_bad_addr:
+wait_for_device_ready_fail:
+	rte_free(dlb2_dev);
+dlb2_dev_malloc_fail:
+	rte_errno = ret;
+	return NULL;
+}
+
+int
+dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
+{
+	int ret = 0;
+	int i = 0;
+	uint32_t dword[16];
+	uint16_t cmd;
+	off_t off;
+
+	uint16_t dev_ctl_word;
+	uint16_t dev_ctl2_word;
+	uint16_t lnk_word;
+	uint16_t lnk_word2;
+	uint16_t slt_word;
+	uint16_t slt_word2;
+	uint16_t rt_ctl_word;
+	uint32_t pri_reqs_dword;
+	uint16_t pri_ctrl_word;
+
+	int pcie_cap_offset;
+	int pri_cap_offset;
+	int msix_cap_offset;
+	int err_cap_offset;
+	int acs_cap_offset;
+	int wait_count;
+
+	uint16_t devsta_busy_word;
+	uint16_t devctl_word;
+
+	struct rte_pci_device *pdev = dlb2_dev->pdev;
+
+	/* Save PCI config state */
+
+	for (i = 0; i < 16; i++) {
+		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
+			return ret;
+	}
+
+	pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
+
+	if (pcie_cap_offset < 0) {
+		printf("[%s()] failed to find the pcie capability\n",
+		       __func__);
+		return pcie_cap_offset;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
+		dev_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
+		lnk_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
+		slt_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_RTCTL;
+	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
+		rt_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
+		dev_ctl2_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
+		lnk_word2 = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
+		slt_word2 = 0;
+
+	off = DLB2_PCI_EXT_CAP_ID_PRI;
+	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (pri_cap_offset >= 0) {
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
+			pri_reqs_dword = 0;
+	}
+
+	/* clear the PCI command register before issuing the FLR */
+
+	off = DLB2_PCI_CMD;
+	cmd = 0;
+	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+		printf("[%s()] failed to write the pci command\n",
+		       __func__);
+		return ret;
+	}
+
+	/* issue the FLR */
+	for (wait_count = 0; wait_count < 4; wait_count++) {
+		int sleep_time;
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
+		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to read the pci device status\n",
+			       __func__);
+			return ret;
+		}
+
+		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
+			break;
+
+		sleep_time = (1 << (wait_count)) * 100;
+		rte_delay_ms(sleep_time);
+	}
+
+	if (wait_count == 4) {
+		printf("[%s()] wait for pci pending transactions timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to read the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
+
+	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to write the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	rte_delay_ms(100);
+
+	/* Restore PCI config state */
+
+	if (pcie_cap_offset >= 0) {
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie device control at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_RTCTL;
+		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	if (pri_cap_offset >= 0) {
+		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
+
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
+		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ERR;
+	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (err_cap_offset >= 0) {
+		uint32_t tmp;
+
+		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	for (i = 16; i > 0; i--) {
+		off = (i - 1) * 4;
+		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_CMD;
+	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
+		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+			printf("[%s()] failed to write the pci command\n",
+			       __func__);
+			return ret;
+		}
+	}
+
+	msix_cap_offset = dlb2_pci_find_capability(pdev,
+						   DLB2_PCI_CAP_ID_MSIX);
+	if (msix_cap_offset >= 0) {
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
+			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ACS;
+	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (acs_cap_offset >= 0) {
+		uint16_t acs_cap, acs_ctrl, acs_mask;
+		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
+		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
+			acs_cap = 0;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
+		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
+		acs_ctrl |= (acs_cap & acs_mask);
+
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
+		acs_mask |= DLB2_PCI_ACS_EC;
+		acs_ctrl &= ~acs_mask;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
new file mode 100644
index 0000000..f082a94
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -0,0 +1,98 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+#endif
+
+#include "base/dlb2_hw_types.h"
+#include "../dlb2_user.h"
+
+#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
+
+struct dlb2_dev;
+
+struct dlb2_port_memory {
+	struct dlb2_list_head list;
+	void *cq_base;
+	bool valid;
+};
+
+struct dlb2_dev {
+	struct rte_pci_device *pdev;
+	struct dlb2_hw hw;
+	/* struct list_head list; */
+	struct device *dlb2_device;
+	bool domain_reset_failed;
+	/* The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	rte_spinlock_t resource_mutex;
+	rte_spinlock_t measurement_lock;
+	bool worker_launched;
+	u8 revision;
+};
+
+struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
+int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
+int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+
+#endif /* __DLB2_MAIN_H */
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
new file mode 100644
index 0000000..ae4d785
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -0,0 +1,196 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_devargs.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_pci.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+#include "dlb2_main.h"
+#include "base/dlb2_hw_types.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_resource.h"
+
+static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
+
+/* Stubs: Allow building partial probe patch */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(arg);
+	RTE_SET_USED(vdev_req);
+	RTE_SET_USED(vdev_id);
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+/* End stubs */
+
+static void
+dlb2_pf_iface_fn_ptrs_init(void)
+{
+/* flexible iface fn ptr assignments will go here */
+}
+
+/* PCI DEV HOOKS */
+static int
+dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
+{
+	int ret = 0;
+	struct rte_pci_device *pci_dev;
+	struct dlb2_devargs dlb2_args = {
+		.socket_id = rte_socket_id(),
+		.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+		.num_dir_credits_override = -1,
+		.qid_depth_thresholds = { {0} },
+		.cos_id = DLB2_COS_DEFAULT
+	};
+	struct dlb2_eventdev *dlb2;
+
+	DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
+		     eventdev->data->dev_id, eventdev->data->socket_id);
+
+	dlb2_pf_iface_fn_ptrs_init();
+
+	pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
+
+		/* Probe the DLB2 PF layer */
+		dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
+
+		if (dlb2->qm_instance.pf_dev == NULL) {
+			DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
+				     rte_errno);
+			ret = -rte_errno;
+			goto dlb2_probe_failed;
+		}
+
+		/* Were we invoked with runtime parameters? */
+		if (pci_dev->device.devargs) {
+			ret = dlb2_parse_params(pci_dev->device.devargs->args,
+						pci_dev->device.devargs->name,
+						&dlb2_args);
+			if (ret) {
+				DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
+					     ret, rte_errno);
+				goto dlb2_probe_failed;
+			}
+		}
+
+		ret = dlb2_primary_eventdev_probe(eventdev,
+						  event_dlb2_pf_name,
+						  &dlb2_args);
+	} else {
+		ret = dlb2_secondary_eventdev_probe(eventdev,
+						    event_dlb2_pf_name);
+	}
+	if (ret)
+		goto dlb2_probe_failed;
+
+	DLB2_LOG_INFO("DLB2 PF Probe success\n");
+
+	return 0;
+
+dlb2_probe_failed:
+
+	DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
+
+	return ret;
+}
+
+#define EVENTDEV_INTEL_VENDOR_ID 0x8086
+
+static const struct rte_pci_id pci_id_dlb2_map[] = {
+	{
+		RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
+			       PCI_DEVICE_ID_INTEL_DLB2_PF)
+	},
+	{
+		.vendor_id = 0,
+	},
+};
+
+static int
+event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
+		     struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
+					     sizeof(struct dlb2_eventdev),
+					     dlb2_eventdev_pci_init,
+					     event_dlb2_pf_name);
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+}
+
+static int
+event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_remove(pci_dev, NULL);
+
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+
+}
+
+static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
+	.id_table = pci_id_dlb2_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe = event_dlb2_pci_probe,
+	.remove = event_dlb2_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
-- 
2.6.4

[dpdk-dev] [PATCH v8 07/23] event/dlb2: add flexible interface

[Timothy McDaniel]

This commit introduces the flexible interface. This
interface allows the core code to operate in PF mode (direct
hardware access) or bifurcated mode (hardware configured via
kernel driver). This driver currently only supports PF mode
but bifurcated mode will be added in a future DPDK patch-set.
Note that the flexible interface is not used for data path
operations, and thus there are no performance concerns
related to the use of function pointers.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_iface.c | 28 ++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.h | 29 +++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build  |  1 +
 3 files changed, 58 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h

diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
new file mode 100644
index 0000000..0fc9991
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+
+#include "dlb2_priv.h"
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+void (*dlb2_iface_low_level_io_init)(void);
+
+int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+				     uint8_t *revision);
+
+void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+				   enum dlb2_cq_poll_modes *mode);
+
+int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
new file mode 100644
index 0000000..4fb416e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_IFACE_H_
+#define _DLB2_IFACE_H_
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+extern void (*dlb2_iface_low_level_io_init)(void);
+
+extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+					    uint8_t *revision);
+
+extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+					  enum dlb2_cq_poll_modes *mode);
+
+extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
+
+#endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 1eab9b9..491e76d 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -8,6 +8,7 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
 endif
 
 sources = files('dlb2.c',
+		'dlb2_iface.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c'
 )
-- 
2.6.4

[dpdk-dev] [PATCH v8 08/23] event/dlb2: add probe-time hardware init

[Timothy McDaniel]

This commit adds probe-time low level hardware
initialization.  It also adds probe-time init for both
primary and secondary DPDK processes.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 188 +++++++++++++++++++-
 drivers/event/dlb2/meson.build             |   3 +-
 drivers/event/dlb2/pf/base/dlb2_resource.c | 274 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  23 +--
 drivers/event/dlb2/pf/dlb2_main.h          |   1 -
 drivers/event/dlb2/pf/dlb2_pf.c            |  78 ++++++--
 6 files changed, 523 insertions(+), 44 deletions(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 1b11979..16653db 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -31,6 +31,7 @@
 #include <rte_string_fns.h>
 
 #include "dlb2_priv.h"
+#include "dlb2_iface.h"
 #include "dlb2_inline_fns.h"
 
 /*
@@ -40,10 +41,108 @@
 #if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
 #error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
 #endif
+static struct rte_event_dev_info evdev_dlb2_default_info = {
+	.driver_name = "", /* probe will set */
+	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
+	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
+	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
+#else
+	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
+#endif
+	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
+	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
+	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
+	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
+	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
+	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+	.max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
+	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
+			  RTE_EVENT_DEV_CAP_EVENT_QOS |
+			  RTE_EVENT_DEV_CAP_BURST_MODE |
+			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
+			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
+			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
+};
 
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/* override defaults with value(s) provided on command line */
+static void
+dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
+				 int *qid_depth_thresholds)
+{
+	int q;
+
+	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
+		if (qid_depth_thresholds[q] != 0)
+			dlb2->ev_queues[q].depth_threshold =
+				qid_depth_thresholds[q];
+	}
+}
+
+static int
+dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
+	int ret;
+
+	/* Query driver resources provisioned for this device */
+
+	ret = dlb2_iface_get_num_resources(handle,
+					   &dlb2->hw_rsrc_query_results);
+	if (ret) {
+		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
+		return ret;
+	}
+
+	/* Complete filling in device resource info returned to evdev app,
+	 * overriding any default values.
+	 * The capabilities (CAPs) were set at compile time.
+	 */
+
+	evdev_dlb2_default_info.max_event_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	evdev_dlb2_default_info.max_event_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	evdev_dlb2_default_info.max_num_events =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	/* Save off values used when creating the scheduling domain. */
+
+	handle->info.num_sched_domains =
+		dlb2->hw_rsrc_query_results.num_sched_domains;
+
+	handle->info.hw_rsrc_max.nb_events_limit =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	handle->info.hw_rsrc_max.num_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues +
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.num_ldb_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	handle->info.hw_rsrc_max.num_ldb_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	handle->info.hw_rsrc_max.num_dir_ports =
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.reorder_window_size =
+		dlb2->hw_rsrc_query_results.num_hist_list_entries;
+
+	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
+
+	return 0;
+}
+
 #define DLB2_BASE_10 10
 
 static int
@@ -235,14 +334,71 @@ set_qid_depth_thresh(const char *key __rte_unused,
 	return 0;
 }
 
+static void
+dlb2_entry_points_init(struct rte_eventdev *dev)
+{
+	RTE_SET_USED(dev);
+
+	/* Eventdev PMD entry points */
+}
+
 int
 dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    const char *name,
 			    struct dlb2_devargs *dlb2_args)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
-	RTE_SET_USED(dlb2_args);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	dlb2->event_dev = dev; /* backlink */
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	dlb2->max_num_events_override = dlb2_args->max_num_events;
+	dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
+	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
+					    &dlb2->revision);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_hardware_init(&dlb2->qm_instance);
+
+	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
+			     err);
+		return err;
+	}
+
+	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	dlb2_init_queue_depth_thresholds(dlb2,
+					 dlb2_args->qid_depth_thresholds.val);
 
 	return 0;
 }
@@ -251,8 +407,30 @@ int
 dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
 			      const char *name)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 491e76d..28fef3f 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -10,7 +10,8 @@ endif
 sources = files('dlb2.c',
 		'dlb2_iface.c',
 		'pf/dlb2_main.c',
-		'pf/dlb2_pf.c'
+		'pf/dlb2_pf.c',
+		'pf/base/dlb2_resource.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
new file mode 100644
index 0000000..6de8b95
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_mbox.h"
+#include "dlb2_osdep.h"
+#include "dlb2_osdep_bitmap.h"
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+	int i;
+
+	dlb2_list_init_head(&domain->used_ldb_queues);
+	dlb2_list_init_head(&domain->used_dir_pq_pairs);
+	dlb2_list_init_head(&domain->avail_ldb_queues);
+	dlb2_list_init_head(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+	int i;
+
+	dlb2_list_init_head(&rsrc->avail_domains);
+	dlb2_list_init_head(&rsrc->used_domains);
+	dlb2_list_init_head(&rsrc->avail_ldb_queues);
+	dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
+	arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
+	arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
+	arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb2_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	struct dlb2_list_entry *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. This is application
+	 * dependent, but the driver interleaves port IDs as much as possible
+	 * to reduce the likelihood of this. This initial allocation maximizes
+	 * the average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	/* Zero-out resource tracking data structures */
+	memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+	memset(&hw->pf, 0, sizeof(hw->pf));
+
+	dlb2_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+		dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+		dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_domains, list);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_ldb_queues, list);
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+		struct dlb2_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
+			      &port->func_list);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+				DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+	if (ret)
+		goto unwind;
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+					DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+		if (ret)
+			goto unwind;
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb2_resource_free(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+	r0.field.disable = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 3c51cab..210b3186 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -19,6 +19,7 @@
 #include "dlb2_main.h"
 #include "../dlb2_user.h"
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 
 #define PF_ID_ZERO 0	/* PF ONLY! */
@@ -72,27 +73,6 @@
 #define DLB2_PCI_ACS_UF                  0x10
 #define DLB2_PCI_ACS_EC                  0x20
 
-/* Stubs: Allow building partial probe patch */
-void dlb2_resource_free(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-int dlb2_resource_init(struct dlb2_hw *hw)
-{
-	int ret = 0;
-	RTE_SET_USED(hw);
-
-	return ret;
-}
-
-void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-/* End stubs */
-
 static int
 dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
 {
@@ -149,7 +129,6 @@ static int
 dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
 {
 	rte_spinlock_init(&dlb2_dev->resource_mutex);
-	rte_spinlock_init(&dlb2_dev->measurement_lock);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index f082a94..f3bee71 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -38,7 +38,6 @@ struct dlb2_dev {
 	 * hardware registers.
 	 */
 	rte_spinlock_t resource_mutex;
-	rte_spinlock_t measurement_lock;
 	bool worker_launched;
 	u8 revision;
 };
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index ae4d785..7d64309 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -32,6 +32,7 @@
 #include <rte_string_fns.h>
 
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 #include "dlb2_main.h"
 #include "base/dlb2_hw_types.h"
@@ -40,35 +41,82 @@
 
 static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
 
-/* Stubs: Allow building partial probe patch */
-int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
-			      struct dlb2_get_num_resources_args *arg,
-			      bool vdev_req,
-			      unsigned int vdev_id)
+static void
+dlb2_pf_low_level_io_init(void)
+{
+	int i;
+	/* Addresses will be initialized at port create */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		/* First directed ports */
+		dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
+
+		/* Now load balanced ports */
+		dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
+	}
+}
+
+static int
+dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
+{
+	RTE_SET_USED(handle);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+static int
+dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
+			   uint8_t *revision)
 {
-	RTE_SET_USED(hw);
-	RTE_SET_USED(arg);
-	RTE_SET_USED(vdev_req);
-	RTE_SET_USED(vdev_id);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	*revision = dlb2_dev->revision;
 
 	return 0;
 }
 
-void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+static void
+dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+	dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_num_resources_args *rsrcs)
 {
-	RTE_SET_USED(hw);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
 }
 
-void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+static int
+dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
+			 enum dlb2_cq_poll_modes *mode)
 {
-	RTE_SET_USED(hw);
+	RTE_SET_USED(handle);
+
+	*mode = DLB2_CQ_POLL_MODE_SPARSE;
+
+	return 0;
 }
-/* End stubs */
 
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
-/* flexible iface fn ptr assignments will go here */
+	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
+	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
+	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
+	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
+	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v8 09/23] event/dlb2: add xstats

[Timothy McDaniel]

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
Reviewed-by: Chen, Mike Ximing <mike.ximing.chen@intel.com>
---
 drivers/event/dlb2/dlb2.c        |   35 +-
 drivers/event/dlb2/dlb2_xstats.c | 1240 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build   |    1 +
 3 files changed, 1273 insertions(+), 3 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 16653db..925824e 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,6 +70,21 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/*
+ * DUMMY - added so that xstats path will compile/link.
+ * Will be replaced by real version in a subsequent
+ * patch.
+ */
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	RTE_SET_USED(dlb2);
+	RTE_SET_USED(queue);
+
+	return 0;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -337,9 +352,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
-	RTE_SET_USED(dev);
-
-	/* Eventdev PMD entry points */
+	/* Expose PMD's eventdev interface */
+	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dump             = dlb2_eventdev_dump,
+		.xstats_get       = dlb2_eventdev_xstats_get,
+		.xstats_get_names = dlb2_eventdev_xstats_get_names,
+		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
+		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+	};
+
+	dev->dev_ops = &dlb2_eventdev_entry_ops;
 }
 
 int
@@ -391,6 +413,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Complete xtstats runtime initialization */
+	err = dlb2_xstats_init(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
+		return err;
+	}
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
new file mode 100644
index 0000000..21391f7
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_xstats.c
@@ -0,0 +1,1240 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eventdev.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+enum dlb2_xstats_type {
+	/* common to device and port */
+	rx_ok,				/**< Receive an event */
+	rx_drop,                        /**< Error bit set in received QE */
+	rx_interrupt_wait,		/**< Wait on an interrupt */
+	rx_umonitor_umwait,		/**< Block using umwait */
+	tx_ok,				/**< Transmit an event */
+	total_polls,			/**< Call dequeue_burst */
+	zero_polls,			/**< Call dequeue burst and return 0 */
+	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
+	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
+	tx_nospc_inflight_max,		/**< Reach the new_event_threshold */
+	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
+	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
+	/* device specific */
+	nb_events_limit,
+	inflight_events,
+	ldb_pool_size,
+	dir_pool_size,
+	/* port specific */
+	tx_new,				/**< Send an OP_NEW event */
+	tx_fwd,				/**< Send an OP_FORWARD event */
+	tx_rel,				/**< Send an OP_RELEASE event */
+	tx_implicit_rel,		/**< Issue an implicit event release */
+	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED event */
+	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL event */
+	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event */
+	tx_sched_directed,		/**< Send a directed event */
+	tx_invalid,                     /**< Send an event with an invalid op */
+	outstanding_releases,		/**< # of releases a port owes */
+	max_outstanding_releases,	/**< max # of releases a port can owe */
+	rx_sched_ordered,		/**< Dequeue an ordered event */
+	rx_sched_unordered,		/**< Dequeue an unordered event */
+	rx_sched_atomic,		/**< Dequeue an atomic event */
+	rx_sched_directed,		/**< Dequeue an directed event */
+	rx_sched_invalid,               /**< Dequeue event sched type invalid */
+	/* common to port and queue */
+	is_configured,			/**< Port is configured */
+	is_load_balanced,		/**< Port is LDB */
+	hw_id,				/**< Hardware ID */
+	/* queue specific */
+	num_links,			/**< Number of ports linked */
+	sched_type,			/**< Queue sched type */
+	enq_ok,				/**< # events enqueued to the queue */
+	current_depth,			/**< Current queue depth */
+	depth_threshold,		/**< Programmed depth threshold */
+	depth_le50_threshold,
+	/**< Depth LE to 50% of the configured hardware threshold */
+	depth_gt50_le75_threshold,
+	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold */
+	depth_gt75_le100_threshold,
+	/**< Depth GT 75%. but LE to the configured hardware threshold */
+	depth_gt100_threshold
+	/**< Depth GT 100% of the configured hw threshold */
+};
+
+typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
+		uint16_t obj_idx, /* port or queue id */
+		enum dlb2_xstats_type stat, int extra_arg);
+
+enum dlb2_xstats_fn_type {
+	DLB2_XSTATS_FN_DEV,
+	DLB2_XSTATS_FN_PORT,
+	DLB2_XSTATS_FN_QUEUE
+};
+
+struct dlb2_xstats_entry {
+	struct rte_event_dev_xstats_name name;
+	uint64_t reset_value; /* an offset to be taken away to emulate resets */
+	enum dlb2_xstats_fn_type fn_id;
+	enum dlb2_xstats_type stat;
+	enum rte_event_dev_xstats_mode mode;
+	int extra_arg;
+	uint16_t obj_idx;
+	uint8_t reset_allowed; /* when set, this value can be reset */
+};
+
+/* Some device stats are simply a summation of the corresponding port values */
+static uint64_t
+dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
+			     int which_stat)
+{
+	int i;
+	uint64_t val = 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
+
+		if (!port->setup_done)
+			continue;
+
+		switch (which_stat) {
+		case rx_ok:
+			val += port->stats.traffic.rx_ok;
+			break;
+		case rx_drop:
+			val += port->stats.traffic.rx_drop;
+			break;
+		case rx_interrupt_wait:
+			val += port->stats.traffic.rx_interrupt_wait;
+			break;
+		case rx_umonitor_umwait:
+			val += port->stats.traffic.rx_umonitor_umwait;
+			break;
+		case tx_ok:
+			val += port->stats.traffic.tx_ok;
+			break;
+		case total_polls:
+			val += port->stats.traffic.total_polls;
+			break;
+		case zero_polls:
+			val += port->stats.traffic.zero_polls;
+			break;
+		case tx_nospc_ldb_hw_credits:
+			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
+			break;
+		case tx_nospc_dir_hw_credits:
+			val += port->stats.traffic.tx_nospc_dir_hw_credits;
+			break;
+		case tx_nospc_inflight_max:
+			val += port->stats.traffic.tx_nospc_inflight_max;
+			break;
+		case tx_nospc_new_event_limit:
+			val += port->stats.traffic.tx_nospc_new_event_limit;
+			break;
+		case tx_nospc_inflight_credits:
+			val += port->stats.traffic.tx_nospc_inflight_credits;
+			break;
+		default:
+			return -1;
+		}
+	}
+	return val;
+}
+
+static uint64_t
+get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
+	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	switch (type) {
+	case rx_ok:
+	case rx_drop:
+	case rx_interrupt_wait:
+	case rx_umonitor_umwait:
+	case tx_ok:
+	case total_polls:
+	case zero_polls:
+	case tx_nospc_ldb_hw_credits:
+	case tx_nospc_dir_hw_credits:
+	case tx_nospc_inflight_max:
+	case tx_nospc_new_event_limit:
+	case tx_nospc_inflight_credits:
+		return dlb2_device_traffic_stat_get(dlb2, type);
+	case nb_events_limit:
+		return dlb2->new_event_limit;
+	case inflight_events:
+		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
+	case ldb_pool_size:
+		return dlb2->num_ldb_credits;
+	case dir_pool_size:
+		return dlb2->num_dir_credits;
+	default: return -1;
+	}
+}
+
+static uint64_t
+get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
+
+	switch (type) {
+	case rx_ok: return ev_port->stats.traffic.rx_ok;
+
+	case rx_drop: return ev_port->stats.traffic.rx_drop;
+
+	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
+
+	case rx_umonitor_umwait:
+		return ev_port->stats.traffic.rx_umonitor_umwait;
+
+	case tx_ok: return ev_port->stats.traffic.tx_ok;
+
+	case total_polls: return ev_port->stats.traffic.total_polls;
+
+	case zero_polls: return ev_port->stats.traffic.zero_polls;
+
+	case tx_nospc_ldb_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
+
+	case tx_nospc_dir_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
+
+	case tx_nospc_inflight_max:
+		return ev_port->stats.traffic.tx_nospc_inflight_max;
+
+	case tx_nospc_new_event_limit:
+		return ev_port->stats.traffic.tx_nospc_new_event_limit;
+
+	case tx_nospc_inflight_credits:
+		return ev_port->stats.traffic.tx_nospc_inflight_credits;
+
+	case is_configured: return ev_port->setup_done;
+
+	case is_load_balanced: return !ev_port->qm_port.is_directed;
+
+	case hw_id: return ev_port->qm_port.id;
+
+	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
+
+	case tx_fwd: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
+
+	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
+
+	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
+
+	case tx_sched_ordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case tx_sched_unordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case tx_sched_atomic:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case tx_sched_directed:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case tx_invalid: return ev_port->stats.tx_invalid;
+
+	case outstanding_releases: return ev_port->outstanding_releases;
+
+	case max_outstanding_releases:
+		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	case rx_sched_ordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case rx_sched_unordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case rx_sched_atomic:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case rx_sched_directed:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
+
+	default: return -1;
+	}
+}
+
+static uint64_t
+dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
+{
+	int port = 0;
+	uint64_t tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
+
+	return tally;
+}
+
+static uint64_t
+dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
+{
+	int port = 0;
+	uint64_t enq_ok_tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
+
+	return enq_ok_tally;
+}
+
+static uint64_t
+get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_queue *ev_queue =
+		&dlb2->ev_queues[obj_idx];
+
+	switch (type) {
+	case is_configured: return ev_queue->setup_done;
+
+	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
+
+	case hw_id: return ev_queue->qm_queue.id;
+
+	case num_links: return ev_queue->num_links;
+
+	case sched_type: return ev_queue->qm_queue.sched_type;
+
+	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
+
+	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
+
+	case depth_threshold: return ev_queue->depth_threshold;
+
+	case depth_le50_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_LE50);
+
+	case depth_gt50_le75_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT50_LE75);
+
+	case depth_gt75_le100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT75_LE100);
+
+	case depth_gt100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT100);
+
+	default: return -1;
+	}
+}
+
+int
+dlb2_xstats_init(struct dlb2_eventdev *dlb2)
+{
+	/*
+	 * define the stats names and types. Used to build up the device
+	 * xstats array
+	 * There are multiple set of stats:
+	 *   - device-level,
+	 *   - per-port,
+	 *   - per-qid,
+	 *
+	 * For each of these sets, we have three parallel arrays, one for the
+	 * names, the other for the stat type parameter to be passed in the fn
+	 * call to get that stat. The third array allows resetting or not.
+	 * All these arrays must be kept in sync
+	 */
+	static const char * const dev_stats[] = {
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"nb_events_limit",
+		"inflight_events",
+		"ldb_pool_size",
+		"dir_pool_size",
+	};
+	static const enum dlb2_xstats_type dev_types[] = {
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		nb_events_limit,
+		inflight_events,
+		ldb_pool_size,
+		dir_pool_size,
+	};
+	/* Note: generated device stats are not allowed to be reset. */
+	static const uint8_t dev_reset_allowed[] = {
+		0, /* rx_ok */
+		0, /* rx_drop */
+		0, /* rx_interrupt_wait */
+		0, /* rx_umonitor_umwait */
+		0, /* tx_ok */
+		0, /* total_polls */
+		0, /* zero_polls */
+		0, /* tx_nospc_ldb_hw_credits */
+		0, /* tx_nospc_dir_hw_credits */
+		0, /* tx_nospc_inflight_max */
+		0, /* tx_nospc_new_event_limit */
+		0, /* tx_nospc_inflight_credits */
+		0, /* nb_events_limit */
+		0, /* inflight_events */
+		0, /* ldb_pool_size */
+		0, /* dir_pool_size */
+	};
+	static const char * const port_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"tx_new",
+		"tx_fwd",
+		"tx_rel",
+		"tx_implicit_rel",
+		"tx_sched_ordered",
+		"tx_sched_unordered",
+		"tx_sched_atomic",
+		"tx_sched_directed",
+		"tx_invalid",
+		"outstanding_releases",
+		"max_outstanding_releases",
+		"rx_sched_ordered",
+		"rx_sched_unordered",
+		"rx_sched_atomic",
+		"rx_sched_directed",
+		"rx_sched_invalid"
+	};
+	static const enum dlb2_xstats_type port_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		tx_new,
+		tx_fwd,
+		tx_rel,
+		tx_implicit_rel,
+		tx_sched_ordered,
+		tx_sched_unordered,
+		tx_sched_atomic,
+		tx_sched_directed,
+		tx_invalid,
+		outstanding_releases,
+		max_outstanding_releases,
+		rx_sched_ordered,
+		rx_sched_unordered,
+		rx_sched_atomic,
+		rx_sched_directed,
+		rx_sched_invalid
+	};
+	static const uint8_t port_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		1, /* rx_ok */
+		1, /* rx_drop */
+		1, /* rx_interrupt_wait */
+		1, /* rx_umonitor_umwait */
+		1, /* tx_ok */
+		1, /* total_polls */
+		1, /* zero_polls */
+		1, /* tx_nospc_ldb_hw_credits */
+		1, /* tx_nospc_dir_hw_credits */
+		1, /* tx_nospc_inflight_max */
+		1, /* tx_nospc_new_event_limit */
+		1, /* tx_nospc_inflight_credits */
+		1, /* tx_new */
+		1, /* tx_fwd */
+		1, /* tx_rel */
+		1, /* tx_implicit_rel */
+		1, /* tx_sched_ordered */
+		1, /* tx_sched_unordered */
+		1, /* tx_sched_atomic */
+		1, /* tx_sched_directed */
+		1, /* tx_invalid */
+		0, /* outstanding_releases */
+		0, /* max_outstanding_releases */
+		1, /* rx_sched_ordered */
+		1, /* rx_sched_unordered */
+		1, /* rx_sched_atomic */
+		1, /* rx_sched_directed */
+		1  /* rx_sched_invalid */
+	};
+
+	/* QID specific stats */
+	static const char * const qid_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"num_links",
+		"sched_type",
+		"enq_ok",
+		"current_depth",
+		"depth_threshold",
+		"depth_le50_threshold",
+		"depth_gt50_le75_threshold",
+		"depth_gt75_le100_threshold",
+		"depth_gt100_threshold",
+	};
+	static const enum dlb2_xstats_type qid_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		num_links,
+		sched_type,
+		enq_ok,
+		current_depth,
+		depth_threshold,
+		depth_le50_threshold,
+		depth_gt50_le75_threshold,
+		depth_gt75_le100_threshold,
+		depth_gt100_threshold,
+	};
+	static const uint8_t qid_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		0, /* num_links */
+		0, /* sched_type */
+		1, /* enq_ok */
+		0, /* current_depth */
+		0, /* depth_threshold */
+		1, /* depth_le50_threshold */
+		1, /* depth_gt50_le75_threshold */
+		1, /* depth_gt75_le100_threshold */
+		1, /* depth_gt100_threshold */
+	};
+
+	/* ---- end of stat definitions ---- */
+
+	/* check sizes, since a missed comma can lead to strings being
+	 * joined by the compiler.
+	 */
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
+
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_reset_allowed));
+
+	/* other vars */
+	const unsigned int count = RTE_DIM(dev_stats) +
+			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
+			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
+	unsigned int i, port, qid, stat_id = 0;
+
+	dlb2->xstats = rte_zmalloc_socket(NULL,
+			sizeof(dlb2->xstats[0]) * count, 0,
+			dlb2->qm_instance.info.socket_id);
+	if (dlb2->xstats == NULL)
+		return -ENOMEM;
+
+#define sname dlb2->xstats[stat_id].name.name
+	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
+		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
+			.fn_id = DLB2_XSTATS_FN_DEV,
+			.stat = dev_types[i],
+			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
+			.reset_allowed = dev_reset_allowed[i],
+		};
+		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
+	}
+	dlb2->xstats_count_mode_dev = stat_id;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
+		dlb2->xstats_offset_for_port[port] = stat_id;
+
+		uint32_t count_offset = stat_id;
+
+		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_PORT,
+				.obj_idx = port,
+				.stat = port_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_PORT,
+				.reset_allowed = port_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "port_%u_%s",
+				 port, port_stats[i]);
+		}
+
+		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_port = stat_id - dlb2->xstats_count_mode_dev;
+
+	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
+		uint32_t count_offset = stat_id;
+
+		dlb2->xstats_offset_for_qid[qid] = stat_id;
+
+		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_QUEUE,
+				.obj_idx = qid,
+				.stat = qid_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
+				.reset_allowed = qid_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "qid_%u_%s",
+				 qid, qid_stats[i]);
+		}
+
+		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_queue = stat_id -
+		(dlb2->xstats_count_mode_dev + dlb2->xstats_count_mode_port);
+#undef sname
+
+	dlb2->xstats_count = stat_id;
+
+	return 0;
+}
+
+void
+dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
+{
+	rte_free(dlb2->xstats);
+	dlb2->xstats_count = 0;
+}
+
+int
+dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstats_names,
+		unsigned int *ids, unsigned int size)
+{
+	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+	uint32_t start_offset = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			break;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			break;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
+		break;
+	default:
+		return -EINVAL;
+	};
+
+	if (xstats_mode_count > size || ids == NULL || xstats_names == NULL)
+		return xstats_mode_count;
+
+	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
+		if (dlb2->xstats[i].mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != dlb2->xstats[i].obj_idx)
+			continue;
+
+		xstats_names[xidx] = dlb2->xstats[i].name;
+		if (ids)
+			ids[xidx] = start_offset + xidx;
+		xidx++;
+	}
+	return xidx;
+}
+
+static int
+dlb2_xstats_update(struct dlb2_eventdev *dlb2,
+		enum rte_event_dev_xstats_mode mode,
+		uint8_t queue_port_id, const unsigned int ids[],
+		uint64_t values[], unsigned int n, const uint32_t reset)
+{
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			goto invalid_value;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			goto invalid_value;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		break;
+	default:
+		goto invalid_value;
+	};
+
+	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
+		dlb2_xstats_fn fn;
+
+		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != xs->obj_idx)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			goto invalid_value;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+
+		if (values)
+			values[xidx] = val;
+
+		if (xs->reset_allowed && reset)
+			xs->reset_value += val;
+
+		xidx++;
+	}
+
+	return xidx;
+
+invalid_value:
+	return -EINVAL;
+}
+
+int
+dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	const uint32_t reset = 0;
+
+	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
+				  reset);
+}
+
+uint64_t
+dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+				 const char *name, unsigned int *id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	dlb2_xstats_fn fn;
+
+	for (i = 0; i < dlb2->xstats_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (strncmp(xs->name.name, name,
+			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
+			if (id != NULL)
+				*id = i;
+
+			switch (xs->fn_id) {
+			case DLB2_XSTATS_FN_DEV:
+				fn = get_dev_stat;
+				break;
+			case DLB2_XSTATS_FN_PORT:
+				fn = get_port_stat;
+				break;
+			case DLB2_XSTATS_FN_QUEUE:
+				fn = get_queue_stat;
+				break;
+			default:
+				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
+					  xs->fn_id);
+				return (uint64_t)-1;
+			}
+
+			return fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+		}
+	}
+	if (id != NULL)
+		*id = (uint32_t)-1;
+	return (uint64_t)-1;
+}
+
+static void
+dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
+			uint32_t num)
+{
+	uint32_t i;
+	dlb2_xstats_fn fn;
+
+	for (i = start; i < start + num; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (!xs->reset_allowed)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			return;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
+		xs->reset_value = val;
+	}
+}
+
+static int
+dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
+			const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_QUEUE,
+					queue_id, ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	if (ids == NULL)
+		dlb2_xstats_reset_range(dlb2,
+			dlb2->xstats_offset_for_qid[queue_id],
+			dlb2->xstats_count_per_qid[queue_id]);
+
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
+		       const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+	int offset = dlb2->xstats_offset_for_port[port_id];
+	int nb_stat = dlb2->xstats_count_per_port[port_id];
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_PORT, port_id,
+					ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
+		      uint32_t nb_ids)
+{
+	uint32_t i;
+
+	if (ids) {
+		for (i = 0; i < nb_ids; i++) {
+			uint32_t id = ids[i];
+
+			if (id >= dlb2->xstats_count_mode_dev)
+				return -EINVAL;
+			dlb2_xstats_reset_range(dlb2, id, 1);
+		}
+	} else {
+		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
+			dlb2_xstats_reset_range(dlb2, i, 1);
+	}
+
+	return 0;
+}
+
+int
+dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+			   enum rte_event_dev_xstats_mode mode,
+			   int16_t queue_port_id,
+			   const uint32_t ids[],
+			   uint32_t nb_ids)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	uint32_t i;
+
+	/* handle -1 for queue_port_id here, looping over all ports/queues */
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
+			return -EINVAL;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+				if (dlb2_xstats_reset_port(dlb2, i,
+							   ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
+			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
+						   ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
+				if (dlb2_xstats_reset_queue(dlb2, i,
+							    ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
+			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
+						    ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	};
+
+	return 0;
+}
+
+void
+dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	int i;
+
+	if (f == NULL) {
+		printf("Invalid file pointer\n");
+		return;
+	}
+
+	if (dev == NULL) {
+		fprintf(f, "Invalid event device\n");
+		return;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (dlb2 == NULL) {
+		fprintf(f, "DLB2 Event device cannot be dumped!\n");
+		return;
+	}
+
+	if (!dlb2->configured)
+		fprintf(f, "DLB2 Event device is not configured\n");
+
+	handle = &dlb2->qm_instance;
+
+	fprintf(f, "================\n");
+	fprintf(f, "DLB2 Device Dump\n");
+	fprintf(f, "================\n");
+
+	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
+		dlb2->umwait_allowed ? "yes" : "no");
+
+	/* Generic top level device information */
+
+	fprintf(f, "device is configured and run state =");
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		fprintf(f, "STOPPED\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
+		fprintf(f, "STOPPING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
+		fprintf(f, "STARTING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
+		fprintf(f, "STARTED\n");
+	else
+		fprintf(f, "UNEXPECTED\n");
+
+	fprintf(f, "domain ID=%u, socket_id=%u, evdev=%p\n",
+		handle->domain_id, handle->info.socket_id, dlb2->event_dev);
+
+	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
+		dlb2->num_dir_ports, dlb2->num_dir_queues);
+
+	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
+		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
+
+	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
+		handle->cfg.resources.num_atomic_inflights,
+		handle->cfg.resources.num_hist_list_entries);
+
+	fprintf(f, "results from most recent hw resource query:\n");
+
+	fprintf(f, "\tnum_sched_domains = %u\n",
+		dlb2->hw_rsrc_query_results.num_sched_domains);
+
+	fprintf(f, "\tnum_ldb_queues = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_queues);
+
+	fprintf(f, "\tnum_ldb_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_ports);
+
+	fprintf(f, "\tnum_dir_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_ports);
+
+	fprintf(f, "\tnum_atomic_inflights = %u\n",
+		dlb2->hw_rsrc_query_results.num_atomic_inflights);
+
+	fprintf(f, "\tnum_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.num_hist_list_entries);
+
+	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
+
+	fprintf(f, "\tnum_ldb_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_credits);
+
+	fprintf(f, "\tnum_dir_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_credits);
+
+	/* Port level information */
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
+		int j;
+
+		if (!p->enq_configured)
+			fprintf(f, "Port_%d is not configured\n", i);
+
+		fprintf(f, "Port_%d\n", i);
+		fprintf(f, "=======\n");
+
+		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
+			p->id, p->setup_done);
+
+		fprintf(f, "\tconfig state=%d, port state=%d\n",
+			p->qm_port.config_state, p->qm_port.state);
+
+		fprintf(f, "\tport is %s\n",
+			p->qm_port.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\toutstanding releases=%u\n",
+			p->outstanding_releases);
+
+		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
+			p->inflight_max, p->inflight_credits);
+
+		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
+			p->credit_update_quanta, p->implicit_release);
+
+		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (p->link[j].valid)
+				fprintf(f, "id=%u prio=%u ",
+					p->link[j].queue_id,
+					p->link[j].priority);
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
+
+		fprintf(f, "\tcached_ldb_credits=%u\n",
+			p->qm_port.cached_ldb_credits);
+
+		fprintf(f, "\tldb_credits = %u\n",
+			p->qm_port.ldb_credits);
+
+		fprintf(f, "\tcached_dir_credits = %u\n",
+			p->qm_port.cached_dir_credits);
+
+		fprintf(f, "\tdir_credits = %u\n",
+			p->qm_port.dir_credits);
+
+		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
+			p->qm_port.gen_bit,
+			p->qm_port.cq_idx,
+			p->qm_port.cq_depth);
+
+		fprintf(f, "\tinterrupt armed=%d\n",
+			p->qm_port.int_armed);
+
+		fprintf(f, "\tPort statistics\n");
+
+		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
+			p->stats.traffic.rx_ok);
+
+		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
+			p->stats.traffic.rx_drop);
+
+		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
+			p->stats.traffic.rx_interrupt_wait);
+
+		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
+			p->stats.traffic.rx_umonitor_umwait);
+
+		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
+			p->stats.traffic.tx_ok);
+
+		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
+			p->stats.traffic.total_polls);
+
+		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
+			p->stats.traffic.zero_polls);
+
+		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_ldb_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_dir_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_max);
+
+		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_new_event_limit);
+
+		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_credits);
+
+		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
+
+		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
+
+		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
+
+		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
+			p->stats.tx_implicit_rel);
+
+		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
+			p->stats.tx_invalid);
+
+		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
+			p->stats.rx_sched_invalid);
+	}
+
+	/* Queue level information */
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
+		int j, k;
+
+		if (!q->setup_done)
+			fprintf(f, "Queue_%d is not configured\n", i);
+
+		fprintf(f, "Queue_%d\n", i);
+		fprintf(f, "========\n");
+
+		fprintf(f, "\tevqueue_%u is set up\n", q->id);
+
+		fprintf(f, "\tqueue is %s\n",
+			q->qm_queue.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
+
+		for (j = 0; j < dlb2->num_ports; j++) {
+			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
+
+			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++) {
+				if (p->link[k].valid &&
+				    p->link[k].queue_id == q->id)
+					fprintf(f, "id=%u prio=%u ",
+						p->id, p->link[k].priority);
+			}
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\tcurrent depth: %u events\n",
+			dlb2_get_queue_depth(dlb2, q));
+
+		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
+			q->qm_queue.num_qid_inflights, q->qm_queue.sched_type);
+	}
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 28fef3f..dc16c96 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files('dlb2.c',
 		'dlb2_iface.c',
+		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c'
-- 
2.6.4

[dpdk-dev] [PATCH v8 10/23] event/dlb2: add infos get and configure

[Timothy McDaniel]

Add support for configuring the DLB2 hardware.
In particular, this patch configures the DLB2
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  116 +
 drivers/event/dlb2/dlb2.c                  |  313 +++
 drivers/event/dlb2/dlb2_iface.c            |    5 +
 drivers/event/dlb2/dlb2_iface.h            |    4 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 3237 ++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |   15 +
 drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
 7 files changed, 3734 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index e3091c9..5f6c486 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -32,9 +32,125 @@ detailed understanding of the hardware, but these details are important when
 writing high-performance code. This section describes the places where the
 eventdev API and DLB2 misalign.
 
+Scheduling Domain Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are 32 scheduling domainis the DLB2.
+When one is configured, it allocates load-balanced and
+directed queues, ports, credits, and other hardware resources. Some
+resource allocations are user-controlled -- the number of queues, for example
+-- and others, like credit pools (one directed and one load-balanced pool per
+scheduling domain), are not.
+
+The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
+setup argument and the per-port ``new_event_threshold`` argument apply as
+defined in the eventdev header file. The limit is applied to all enqueues,
+regardless of whether it will consume a directed or load-balanced credit.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Reconfiguration
+~~~~~~~~~~~~~~~
+
+The Eventdev API allows one to reconfigure a device, its ports, and its queues
+by first stopping the device, calling the configuration function(s), then
+restarting the device. The DLB2 does not support configuring an individual queue
+or port without first reconfiguring the entire device, however, so there are
+certain reconfiguration sequences that are valid in the eventdev API but not
+supported by the PMD.
+
+Specifically, the PMD supports the following configuration sequence:
+1. Configure and start the device
+2. Stop the device
+3. (Optional) Reconfigure the device
+4. (Optional) If step 3 is run:
+
+   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
+   b. The reconfigured port(s) lose their previous queue links.
+
+5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
+6. Restart the device. If the device is reconfigured in step 3 but one or more
+   of its ports or queues are not, the PMD will apply their previous
+   configuration (including port->queue links) at this time.
+
+The PMD does not support the following configuration sequences:
+1. Configure and start the device
+2. Stop the device
+3. Setup queue or setup port
+4. Start the device
+
+This sequence is not supported because the event device must be reconfigured
+before its ports or queues can be.
+
+Atomic Inflights Allocation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
+inflight event in a temporary buffer that is divided among load-balanced
+queues. If a queue's atomic buffer storage fills up, this can result in
+head-of-line-blocking. For example:
+
+- An LDB queue allocated N atomic buffer entries
+- All N entries are filled with events from flow X, which is pinned to CQ 0.
+
+Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
+scheduled. The likelihood of this case depends on the eventdev configuration,
+traffic behavior, event processing latency, potential for a worker to be
+interrupted or otherwise delayed, etc.
+
+By default, the PMD allocates 16 buffer entries for each load-balanced queue,
+which provides an even division across all 128 queues but potentially wastes
+buffer space (e.g. if not all queues are used, or aren't used for atomic
+scheduling).
+
+The PMD provides a dev arg to override the default per-queue allocation. To
+increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,atm_inflights=64
+
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 925824e..6798e66 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -85,6 +85,25 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
 	return 0;
 }
 
+static void
+dlb2_free_qe_mem(struct dlb2_port *qm_port)
+{
+	if (qm_port == NULL)
+		return;
+
+	rte_free(qm_port->qe4);
+	qm_port->qe4 = NULL;
+
+	rte_free(qm_port->int_arm_qe);
+	qm_port->int_arm_qe = NULL;
+
+	rte_free(qm_port->consume_qe);
+	qm_port->consume_qe = NULL;
+
+	rte_memzone_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz);
+	dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz = NULL;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -350,10 +369,304 @@ set_qid_depth_thresh(const char *key __rte_unused,
 }
 
 static void
+dlb2_eventdev_info_get(struct rte_eventdev *dev,
+		       struct rte_event_dev_info *dev_info)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret;
+
+	ret = dlb2_hw_query_resources(dlb2);
+	if (ret) {
+		const struct rte_eventdev_data *data = dev->data;
+
+		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+			     ret, data->dev_id);
+		/* fn is void, so fall through and return values set up in
+		 * probe
+		 */
+	}
+
+	/* Add num resources currently owned by this domain.
+	 * These would become available if the scheduling domain were reset due
+	 * to the application recalling eventdev_configure to *reconfigure* the
+	 * domain.
+	 */
+	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
+	evdev_dlb2_default_info.max_event_queues += dlb2->num_ldb_queues;
+	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
+
+	evdev_dlb2_default_info.max_event_queues =
+		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
+			RTE_EVENT_MAX_QUEUES_PER_DEV);
+
+	evdev_dlb2_default_info.max_num_events =
+		RTE_MIN(evdev_dlb2_default_info.max_num_events,
+			dlb2->max_num_events_override);
+
+	*dev_info = evdev_dlb2_default_info;
+}
+
+static int
+dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
+			    const struct dlb2_hw_rsrcs *resources_asked)
+{
+	int ret = 0;
+	struct dlb2_create_sched_domain_args *cfg;
+
+	if (resources_asked == NULL) {
+		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
+		ret = EINVAL;
+		goto error_exit;
+	}
+
+	/* Map generic qm resources to dlb2 resources */
+	cfg = &handle->cfg.resources;
+
+	/* DIR ports and queues */
+
+	cfg->num_dir_ports = resources_asked->num_dir_ports;
+
+	cfg->num_dir_credits = resources_asked->num_dir_credits;
+
+	/* LDB queues */
+
+	cfg->num_ldb_queues = resources_asked->num_ldb_queues;
+
+	/* LDB ports */
+
+	cfg->cos_strict = 0; /* Best effort */
+	cfg->num_cos_ldb_ports[0] = 0;
+	cfg->num_cos_ldb_ports[1] = 0;
+	cfg->num_cos_ldb_ports[2] = 0;
+	cfg->num_cos_ldb_ports[3] = 0;
+
+	switch (handle->cos_id) {
+	case DLB2_COS_0:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[0] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_1:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[1] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_2:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[2] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_3:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[3] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_DEFAULT:
+		/* all ldb ports are don't care ports from a cos perspective */
+		cfg->num_ldb_ports =
+			resources_asked->num_ldb_ports;
+		break;
+	}
+
+	cfg->num_ldb_credits =
+		resources_asked->num_ldb_credits;
+
+	cfg->num_atomic_inflights =
+		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
+		cfg->num_ldb_queues;
+
+	cfg->num_hist_list_entries = resources_asked->num_ldb_ports *
+		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d\n",
+		     cfg->num_ldb_queues,
+		     resources_asked->num_ldb_ports,
+		     cfg->num_dir_ports,
+		     cfg->num_atomic_inflights,
+		     cfg->num_hist_list_entries,
+		     cfg->num_ldb_credits,
+		     cfg->num_dir_credits);
+
+	/* Configure the QM */
+
+	ret = dlb2_iface_sched_domain_create(handle, cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: domain create failed, ret = %d, extra status: %s\n",
+			     ret,
+			     dlb2_error_strings[cfg->response.status]);
+
+		goto error_exit;
+	}
+
+	handle->domain_id = cfg->response.id;
+	handle->cfg.configured = true;
+
+error_exit:
+
+	return ret;
+}
+
+static void
+dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	enum dlb2_configuration_state config_state;
+	int i, j;
+
+	dlb2_iface_domain_reset(dlb2);
+
+	/* Free all dynamically allocated port memory */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
+
+	/* If reconfiguring, mark the device's queues and ports as "previously
+	 * configured." If the user doesn't reconfigure them, the PMD will
+	 * reapply their previous configuration when the device is started.
+	 */
+	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
+		DLB2_NOT_CONFIGURED;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		dlb2->ev_ports[i].qm_port.config_state = config_state;
+		/* Reset setup_done so ports can be reconfigured */
+		dlb2->ev_ports[i].setup_done = false;
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			dlb2->ev_ports[i].link[j].mapped = false;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++)
+		dlb2->ev_queues[i].qm_queue.config_state = config_state;
+
+	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
+		dlb2->ev_queues[i].setup_done = false;
+
+	dlb2->num_ports = 0;
+	dlb2->num_ldb_ports = 0;
+	dlb2->num_dir_ports = 0;
+	dlb2->num_queues = 0;
+	dlb2->num_ldb_queues = 0;
+	dlb2->num_dir_queues = 0;
+	dlb2->configured = false;
+}
+
+/* Note: 1 QM instance per QM device, QM instance/device == event device */
+static int
+dlb2_eventdev_configure(const struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
+	const struct rte_eventdev_data *data = dev->data;
+	const struct rte_event_dev_config *config = &data->dev_conf;
+	int ret;
+
+	/* If this eventdev is already configured, we must release the current
+	 * scheduling domain before attempting to configure a new one.
+	 */
+	if (dlb2->configured) {
+		dlb2_hw_reset_sched_domain(dev, true);
+
+		ret = dlb2_hw_query_resources(dlb2);
+		if (ret) {
+			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+				     ret, data->dev_id);
+			return ret;
+		}
+	}
+
+	if (config->nb_event_queues > rsrcs->num_queues) {
+		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_queues,
+			     rsrcs->num_queues);
+		return -EINVAL;
+	}
+	if (config->nb_event_ports > (rsrcs->num_ldb_ports
+			+ rsrcs->num_dir_ports)) {
+		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_ports,
+			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
+		return -EINVAL;
+	}
+	if (config->nb_events_limit > rsrcs->nb_events_limit) {
+		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_events_limit,
+			     rsrcs->nb_events_limit);
+		return -EINVAL;
+	}
+
+	if (config->event_dev_cfg & RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT)
+		dlb2->global_dequeue_wait = false;
+	else {
+		uint32_t timeout32;
+
+		dlb2->global_dequeue_wait = true;
+
+		/* note size mismatch of timeout vals in eventdev lib. */
+		timeout32 = config->dequeue_timeout_ns;
+
+		dlb2->global_dequeue_wait_ticks =
+			timeout32 * (rte_get_timer_hz() / 1E9);
+	}
+
+	/* Does this platform support umonitor/umwait? */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
+		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
+			DLB2_LOG_ERR("invalid value (%d) for RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
+				     RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
+			return -EINVAL;
+		}
+		dlb2->umwait_allowed = true;
+	}
+
+	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
+	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs->num_dir_ports;
+	/* 1 dir queue per dir port */
+	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports;
+
+	/* Scale down nb_events_limit by 4 for directed credits, since there
+	 * are 4x as many load-balanced credits.
+	 */
+	rsrcs->num_ldb_credits = 0;
+	rsrcs->num_dir_credits = 0;
+
+	if (rsrcs->num_ldb_queues)
+		rsrcs->num_ldb_credits = config->nb_events_limit;
+	if (rsrcs->num_dir_ports)
+		rsrcs->num_dir_credits = config->nb_events_limit / 4;
+	if (dlb2->num_dir_credits_override != -1)
+		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
+
+	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
+		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
+		return -ENODEV;
+	}
+
+	dlb2->new_event_limit = config->nb_events_limit;
+	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
+
+	/* Save number of ports/queues for this event dev */
+	dlb2->num_ports = config->nb_event_ports;
+	dlb2->num_queues = config->nb_event_queues;
+	dlb2->num_dir_ports = rsrcs->num_dir_ports;
+	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
+	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
+	dlb2->num_dir_queues = dlb2->num_dir_ports;
+	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
+	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
+	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
+	dlb2->max_dir_credits = rsrcs->num_dir_credits;
+
+	dlb2->configured = true;
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dev_infos_get    = dlb2_eventdev_info_get,
+		.dev_configure    = dlb2_eventdev_configure,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 0fc9991..a829b9b 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -26,3 +26,8 @@ int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
+
+int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				struct dlb2_create_sched_domain_args *args);
+
+void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4fb416e..6663dab 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -26,4 +26,8 @@ extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
 
+extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				 struct dlb2_create_sched_domain_args *args);
+
+extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 6de8b95..63a68a6 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -12,6 +12,27 @@
 #include "dlb2_regs.h"
 #include "dlb2_resource.h"
 
+#include "../../dlb2_priv.h"
+#include "../../dlb2_inline_fns.h"
+
+#define DLB2_DOM_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, domain_list, iter)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, func_list, iter)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, domain_list, it, it_tmp)
+
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, func_list, it, it_tmp)
+
 static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
 {
 	int i;
@@ -272,3 +293,3219 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
 
 	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
 }
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+	union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+	r0.field.count = domain->num_ldb_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+	r1.field.count = domain->num_dir_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+		       struct dlb2_function_resources *rsrcs,
+		       u32 domain_id,
+		       u32 cos_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	RTE_SET_USED(iter);
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+				   typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				   struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_ports,
+				   u32 cos_id,
+				   struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_ldb_port *port;
+
+		port = dlb2_get_next_ldb_port(hw, rsrcs,
+					      domain->id.phys_id, cos_id);
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_ports[cos_id],
+			      &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_ports[cos_id],
+			      &port->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_create_sched_domain_args *args,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      num,
+						      i,
+						      resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+					ret = __dlb2_attach_ldb_ports(hw,
+								      rsrcs,
+								      domain,
+								      1,
+								      cos_id,
+								      resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret < 0)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      1,
+						      j,
+						      resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 u32 num_ports,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+					   typeof(*port));
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_dir_pq_pairs, &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_dir_pq_pairs, &port->domain_list);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+					struct dlb2_hw_domain *domain,
+					u32 num_atomic_inflights,
+					struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+				     struct dlb2_hw_domain *domain,
+				     u32 num_hist_list_entries,
+				     struct dlb2_cmd_response *resp)
+{
+	struct dlb2_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb2_bitmap_find_set_bit_range(bitmap,
+						      num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+				  struct dlb2_function_resources *rsrcs,
+				  struct dlb2_hw_domain *domain,
+				  u32 num_queues,
+				  struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb2_ldb_queue *queue;
+
+		queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+					    typeof(*queue));
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_queues, &queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_queues, &queue->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+			     struct dlb2_function_resources *rsrcs,
+			     struct dlb2_hw_domain *domain,
+			     struct dlb2_create_sched_domain_args *args,
+			     struct dlb2_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb2_attach_ldb_queues(hw,
+				     rsrcs,
+				     domain,
+				     args->num_ldb_queues,
+				     resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_ports(hw,
+				    rsrcs,
+				    domain,
+				    args,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_ports(hw,
+				    rsrcs,
+				    domain,
+				    args->num_dir_ports,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_credits(rsrcs,
+				      domain,
+				      args->num_ldb_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_credits(rsrcs,
+				      domain,
+				      args->num_dir_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+						   domain,
+						   args->num_hist_list_entries,
+						   resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_atomic_inflights(rsrcs,
+					   domain,
+					   args->num_atomic_inflights,
+					   resp);
+	if (ret < 0)
+		return ret;
+
+	dlb2_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+				  struct dlb2_create_sched_domain_args *args,
+				  struct dlb2_cmd_response *resp)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb2_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+				  struct dlb2_create_sched_domain_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		    args->num_ldb_queues);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		    args->num_ldb_ports);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		    args->num_cos_ldb_ports[0]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		    args->cos_strict);
+	DLB2_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		    args->num_dir_ports);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		    args->num_atomic_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		    args->num_hist_list_entries);
+	DLB2_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		    args->num_ldb_credits);
+	DLB2_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		    args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ *	domain and its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+	if (ret)
+		return ret;
+
+	domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available domains\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (domain->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_domains contains configured domains.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	dlb2_init_domain_rsrc_lists(domain);
+
+	ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to verify args.\n",
+			    __func__);
+
+		return ret;
+	}
+
+	dlb2_list_del(&rsrcs->avail_domains, &domain->func_list);
+
+	dlb2_list_add(&rsrcs->used_domains, &domain->func_list);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+	DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+			     struct dlb2_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb2_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	int ret;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_disable(hw, port);
+
+		ret = dlb2_drain_dir_cq(hw, port);
+		if (ret < 0)
+			return ret;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_dir_pq_pair *queue)
+{
+	union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *queue)
+{
+	return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+				      struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		dlb2_movdir64b(pp_addr, hcw);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int ret, i;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			if (toggle_port)
+				dlb2_ldb_port_cq_disable(hw, port);
+
+			ret = dlb2_drain_ldb_cq(hw, port);
+			if (ret < 0)
+				return ret;
+
+			if (toggle_port)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+
+	return 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_atm_active r1;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_ldb_queue *queue)
+{
+	return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb2_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+					   struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to unmap domain queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = true;
+
+			dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+			   u32 id,
+			   bool vdev_req,
+			   unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iter1) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter2)
+			if (queue->id.virt_id == id)
+				return queue;
+	}
+
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue, iter1)
+		if (queue->id.virt_id == id)
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+						      u32 id,
+						      bool vdev_req,
+						      unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iteration;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	RTE_SET_USED(iteration);
+
+	if (id >= DLB2_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iteration)
+		if (domain->id.virt_id == id)
+			return domain;
+
+	return NULL;
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot,
+					   enum dlb2_qid_map_state new_state)
+{
+	enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb2_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB2_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB2_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB2_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB2_HW_DBG(hw,
+		    "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB2_HW_ERR(hw,
+		    "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+				enum dlb2_qid_map_state state,
+				int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+				      enum dlb2_qid_map_state state,
+				      struct dlb2_ldb_queue *queue,
+				      int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+					      struct dlb2_hw_domain *domain,
+					      struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain,
+					     struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+						struct dlb2_ldb_port *port,
+						int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 0;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 1;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+					struct dlb2_ldb_port *p,
+					struct dlb2_ldb_queue *q,
+					u8 priority)
+{
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_lsp_cq2qid0 r1;
+	union dlb2_atm_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix_00 r3;
+	union dlb2_lsp_qid2cqidix2_00 r4;
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+	    !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+	r0.field.v |= 1 << i;
+	r0.field.prio |= (priority & 0x7) << i * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+	else
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		r1.field.qid_p0 = q->id.phys_id;
+	if (i == 1 || i == 5)
+		r1.field.qid_p1 = q->id.phys_id;
+	if (i == 2 || i == 6)
+		r1.field.qid_p2 = q->id.phys_id;
+	if (i == 3 || i == 7)
+		r1.field.qid_p3 = q->id.phys_id;
+
+	if (i < 4)
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+	else
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r4.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+						  p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		r2.field.cq_p0 |= 1 << i;
+		r3.field.cq_p0 |= 1 << i;
+		r4.field.cq_p0 |= 1 << i;
+		break;
+
+	case 1:
+		r2.field.cq_p1 |= 1 << i;
+		r3.field.cq_p1 |= 1 << i;
+		r4.field.cq_p1 |= 1 << i;
+		break;
+
+	case 2:
+		r2.field.cq_p2 |= 1 << i;
+		r3.field.cq_p2 |= 1 << i;
+		r4.field.cq_p2 |= 1 << i;
+		break;
+
+	case 3:
+		r2.field.cq_p3 |= 1 << i;
+		r3.field.cq_p3 |= 1 << i;
+		r4.field.cq_p3 |= 1 << i;
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r3.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		    r4.val);
+
+	dlb2_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB2_QUEUE_MAPPED;
+
+	return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	/* Set the atomic scheduling haswork bit */
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.rlist_haswork_v = r0.field.count > 0;
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      u8 slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.rlist_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.nalb_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+					      struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+	r0.field.limit = queue->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+						struct dlb2_ldb_queue *queue)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+						struct dlb2_hw_domain *domain,
+						struct dlb2_ldb_port *port,
+						struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_lsp_qid_ldb_infl_cnt r0;
+	enum dlb2_qid_map_state state;
+	int slot, ret, i;
+	u8 prio;
+	RTE_SET_USED(iter);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: non-zero QID inflight count\n",
+			    __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			state = DLB2_QUEUE_MAPPED;
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+					 struct dlb2_ldb_port *port,
+					 struct dlb2_ldb_queue *queue,
+					 u8 priority)
+{
+	union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	int domain_id, slot, ret;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+		DLB2_HW_ERR(hw,
+			    "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		if (port->enabled)
+			dlb2_ldb_port_cq_enable(hw, port);
+
+		dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain,
+					struct dlb2_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		union dlb2_lsp_qid_ldb_infl_cnt r0;
+		struct dlb2_ldb_queue *queue;
+		int qid;
+
+		if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: unable to find queue %d\n",
+				    __func__, qid);
+			continue;
+		}
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count)
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb2_ldb_port_cq_disable(hw, port);
+
+		dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count) {
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+
+			dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port,
+				   struct dlb2_ldb_queue *queue)
+{
+	enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_atm_qid2cqidix_00 r1;
+	union dlb2_lsp_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix2_00 r3;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB2_QUEUE_MAPPED;
+	in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: QID %d isn't mapped\n",
+			    __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+	r0.field.v &= ~(1 << i);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		r1.field.cq_p0 &= ~(1 << i);
+		r2.field.cq_p0 &= ~(1 << i);
+		r3.field.cq_p0 &= ~(1 << i);
+		break;
+
+	case 1:
+		r1.field.cq_p1 &= ~(1 << i);
+		r2.field.cq_p1 &= ~(1 << i);
+		r3.field.cq_p1 &= ~(1 << i);
+		break;
+
+	case 2:
+		r1.field.cq_p2 &= ~(1 << i);
+		r2.field.cq_p2 &= ~(1 << i);
+		r3.field.cq_p2 &= ~(1 << i);
+		break;
+
+	case 3:
+		r1.field.cq_p3 &= ~(1 << i);
+		r2.field.cq_p3 &= ~(1 << i);
+		r3.field.cq_p3 &= ~(1 << i);
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+		    r1.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+		    r3.val);
+
+	dlb2_flush_csr(hw);
+
+	unmapped = DLB2_QUEUE_UNMAPPED;
+
+	return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_ldb_port *port,
+				 struct dlb2_ldb_queue *queue,
+				 u8 prio)
+{
+	if (domain->started)
+		return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   int slot)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb2_ldb_port_qid_map *map;
+		struct dlb2_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain,
+					  struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (r0.field.count > 0)
+		return false;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = false;
+
+			dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_dir_vpp_v r1;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		unsigned int offs;
+		u32 virt_id;
+
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+	}
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_ldb_vpp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			unsigned int offs;
+			u32 virt_id;
+
+			if (hw->virt_mode == DLB2_VIRT_SRIOV)
+				virt_id = port->id.virt_id;
+			else
+				virt_id = port->id.phys_id;
+
+			offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+			DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+	union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+				    r0.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+				    r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+	union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+			    r0.val);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+			    r1.val);
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+		union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+		union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+				    r1.val);
+
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID_V(idx),
+				    r2.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID2QID(idx),
+				    r3.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID_V(idx),
+				    r1.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID2QID(idx),
+				    r2.val);
+		}
+	}
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+					       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_sn_chk_enbl r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.en = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			int i;
+
+			for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+				if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					    __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		port->enabled = false;
+
+		dlb2_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	union dlb2_sys_dir_pp_v r1;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+			    r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_ldb_pp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_ldb_queue *queue;
+	int i;
+	RTE_SET_USED(iter);
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty ldb queue %d\n",
+				    __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port, iter) {
+			if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb2_ldb_cq_token_count(hw, ldb_port)) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to empty ldb port %d\n",
+					    __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir queue %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb2_dir_cq_token_count(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir port %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						   struct dlb2_ldb_port *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP2PP(offs),
+			    DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP_V(offs),
+			    DLB2_SYS_VF_LDB_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    DLB2_SYS_LDB_PP_V_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_DSBL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_BASE_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID0(port->id.phys_id),
+		    DLB2_LSP_CQ2QID0_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID1(port->id.phys_id),
+		    DLB2_LSP_CQ2QID1_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+		    DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			__dlb2_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+				       struct dlb2_dir_pq_pair *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_DSBL_RST);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_FMT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP2PP(offs),
+			    DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP_V(offs),
+			    DLB2_SYS_VF_DIR_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		__dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+			    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_ITS(queue_id),
+			    DLB2_SYS_LDB_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN(queue_id),
+			    DLB2_CHP_ORD_QID_SN_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+			    DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_V(queue_id),
+			    DLB2_SYS_LDB_QID_V_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_CFG_V(queue_id),
+			    DLB2_SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB2_CSR_WR(hw,
+				    offs[queue->sn_group],
+				    DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX2(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX2_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_ATM_QID2CQIDIX(queue_id, i),
+				    DLB2_ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb2_domain_reset_dir_port_registers(hw, domain);
+
+	dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_dir_pq_pair *tmp_dir_port;
+	struct dlb2_ldb_queue *tmp_ldb_queue;
+	struct dlb2_ldb_port *tmp_ldb_port;
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_list_head *list;
+	int ret, i;
+	RTE_SET_USED(tmp_dir_port);
+	RTE_SET_USED(tmp_ldb_queue);
+	RTE_SET_USED(tmp_ldb_port);
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list = &domain->used_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb2_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		dlb2_list_del(&domain->used_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list = &domain->avail_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		ldb_queue->owned = false;
+
+		dlb2_list_del(&domain->avail_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		list = &domain->used_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB2_QUEUE_UNMAPPED;
+
+			dlb2_list_del(&domain->used_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list = &domain->avail_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			ldb_port->owned = false;
+
+			dlb2_list_del(&domain->avail_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list = &domain->used_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		dlb2_list_del(&domain->used_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list = &domain->avail_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				    domain->hist_list_entry_base,
+				    domain->total_hist_list_entries);
+	if (ret) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			    __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	dlb2_list_del(&rsrcs->used_domains, &domain->func_list);
+	dlb2_list_add(&rsrcs->avail_domains, &domain->func_list);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain,
+					    struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_ldb_port *port;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (!dlb2_list_empty(&domain->used_ldb_ports[i]))
+			break;
+	}
+
+	if (i == DLB2_NUM_COS_DOMAINS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: No configured LDB ports\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb2_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+	 * prior to this in dlb2_domain_drain_mapped_queues().
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings != 0 ||
+		    dlb2_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ *	hardware resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain  == NULL || !domain->configured)
+		return -EINVAL;
+
+	/* Disable VPPs */
+	if (vdev_req) {
+		dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+		dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+	}
+
+	/* Disable CQ interrupts */
+	dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+	dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+	dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb2_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_mapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	dlb2_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb2_domain_disable_dir_cqs(hw, domain);
+
+	/* Disable PPs */
+	dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+	ret = dlb2_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Reset the QID and port state. */
+	dlb2_domain_reset_registers(hw, domain);
+
+	/* Hardware reset complete. Reset the domain's software state */
+	ret = dlb2_domain_reset_software_state(hw, domain);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 210b3186..285ad2c 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -598,3 +598,18 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
 
 	return 0;
 }
+
+int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+			    struct dlb2_create_sched_domain_args *args,
+			    struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_sched_domain(hw, args, resp, NOT_VF_REQ,
+					   PF_ID_ZERO);
+}
+
+int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 7d64309..9310f72 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -108,15 +108,59 @@ dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
 	return 0;
 }
 
+static int
+dlb2_pf_sched_domain_create(struct dlb2_hw_dev *handle,
+			    struct dlb2_create_sched_domain_args *arg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (dlb2_dev->domain_reset_failed) {
+		response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = dlb2_pf_create_sched_domain(&dlb2_dev->hw, arg, &response);
+	if (ret)
+		goto done;
+
+done:
+
+	arg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static void
+dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret;
+
+	dlb2_dev = (struct dlb2_dev *)dlb2->qm_instance.pf_dev;
+	ret = dlb2_pf_reset_domain(&dlb2_dev->hw, dlb2->qm_instance.domain_id);
+	if (ret)
+		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
 	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
 	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_domain_reset = dlb2_pf_domain_reset;
 	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
 	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v8 11/23] event/dlb2: add queue and port default conf

[Timothy McDaniel]

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 6798e66..ef1c000 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -661,12 +661,42 @@ dlb2_eventdev_configure(const struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
+				    uint8_t port_id,
+				    struct rte_event_port_conf *port_conf)
+{
+	RTE_SET_USED(port_id);
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	port_conf->new_event_threshold = dlb2->new_event_limit;
+	port_conf->dequeue_depth = 32;
+	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
+	port_conf->event_port_cfg = 0;
+}
+
+static void
+dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
+				     uint8_t queue_id,
+				     struct rte_event_queue_conf *queue_conf)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(queue_id);
+
+	queue_conf->nb_atomic_flows = 1024;
+	queue_conf->nb_atomic_order_sequences = 64;
+	queue_conf->event_queue_cfg = 0;
+	queue_conf->priority = 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v8 12/23] event/dlb2: add queue setup

[Timothy McDaniel]

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  73 +++--
 drivers/event/dlb2/dlb2.c                  | 312 +++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |  12 +
 drivers/event/dlb2/dlb2_iface.h            |  12 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 464 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  82 +++++
 7 files changed, 926 insertions(+), 39 deletions(-)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 5f6c486..b154cac 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -47,45 +47,40 @@ setup argument and the per-port ``new_event_threshold`` argument apply as
 defined in the eventdev header file. The limit is applied to all enqueues,
 regardless of whether it will consume a directed or load-balanced credit.
 
-Load-balanced and Directed Ports
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
-does not have the same concept, but it has a similar one: ports and queues that
-are singly-linked (i.e. linked to a single queue or port, respectively).
-
-The ``rte_event_dev_info_get()`` function reports the number of available
-event ports and queues (among other things). For the DLB2 PMD, max_event_ports
-and max_event_queues report the number of available load-balanced ports and
-queues, and max_single_link_event_port_queue_pairs reports the number of
-available directed ports and queues.
-
-When a scheduling domain is created in ``rte_event_dev_configure()``, the user
-specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
-control the total number of ports (load-balanced and directed) and the number
-of directed ports. Hence, the number of requested load-balanced ports is
-``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
-specifies the total number of queues (load-balanced and directed). The number
-of directed queues comes from ``nb_single_link_event_port_queues``, since
-directed ports and queues come in pairs.
-
-When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
-whether it should be configured as a directed (the flag is set) or a
-load-balanced (the flag is unset) port. Similarly, the
-``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
-whether it is a directed or load-balanced queue.
-
-Load-balanced ports can only be linked to load-balanced queues, and directed
-ports can only be linked to directed queues. Furthermore, directed ports can
-only be linked to a single directed queue (and vice versa), and that link
-cannot change after the eventdev is started.
-
-The eventdev API does not have a directed scheduling type. To support directed
-traffic, the dlb PMD detects when an event is being sent to a directed queue
-and overrides its scheduling type. Note that the originally selected scheduling
-type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
-will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
-port.
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
 
 Flow ID
 ~~~~~~~
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index ef1c000..8c1e06d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -688,6 +688,317 @@ dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
 	queue_conf->priority = 0;
 }
 
+static int32_t
+dlb2_get_sn_allocation(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_allocation_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_set_sn_allocation(struct dlb2_eventdev *dlb2, int group, int num)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_set_sn_allocation_args cfg;
+	int ret;
+
+	cfg.num = num;
+	cfg.group = group;
+
+	ret = dlb2_iface_set_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: set_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int32_t
+dlb2_get_sn_occupancy(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_occupancy_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_occupancy(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_occupancy ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+/* Query the current sequence number allocations and, if they conflict with the
+ * requested LDB queue configuration, attempt to re-allocate sequence numbers.
+ * This is best-effort; if it fails, the PMD will attempt to configure the
+ * load-balanced queue and return an error.
+ */
+static void
+dlb2_program_sn_allocation(struct dlb2_eventdev *dlb2,
+			   const struct rte_event_queue_conf *queue_conf)
+{
+	int grp_occupancy[DLB2_NUM_SN_GROUPS];
+	int grp_alloc[DLB2_NUM_SN_GROUPS];
+	int i, sequence_numbers;
+
+	sequence_numbers = (int)queue_conf->nb_atomic_order_sequences;
+
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		int total_slots;
+
+		grp_alloc[i] = dlb2_get_sn_allocation(dlb2, i);
+		if (grp_alloc[i] < 0)
+			return;
+
+		total_slots = DLB2_MAX_LDB_SN_ALLOC / grp_alloc[i];
+
+		grp_occupancy[i] = dlb2_get_sn_occupancy(dlb2, i);
+		if (grp_occupancy[i] < 0)
+			return;
+
+		/* DLB has at least one available slot for the requested
+		 * sequence numbers, so no further configuration required.
+		 */
+		if (grp_alloc[i] == sequence_numbers &&
+		    grp_occupancy[i] < total_slots)
+			return;
+	}
+
+	/* None of the sequence number groups are configured for the requested
+	 * sequence numbers, so we have to reconfigure one of them. This is
+	 * only possible if a group is not in use.
+	 */
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		if (grp_occupancy[i] == 0)
+			break;
+	}
+
+	if (i == DLB2_NUM_SN_GROUPS) {
+		printf("[%s()] No groups with %d sequence_numbers are available or have free slots\n",
+		       __func__, sequence_numbers);
+		return;
+	}
+
+	/* Attempt to configure slot i with the requested number of sequence
+	 * numbers. Ignore the return value -- if this fails, the error will be
+	 * caught during subsequent queue configuration.
+	 */
+	dlb2_set_sn_allocation(dlb2, i, sequence_numbers);
+}
+
+static int32_t
+dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 const struct rte_event_queue_conf *evq_conf)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_queue *queue = &ev_queue->qm_queue;
+	struct dlb2_create_ldb_queue_args cfg;
+	int32_t ret;
+	uint32_t qm_qid;
+	int sched_type = -1;
+
+	if (evq_conf == NULL)
+		return -EINVAL;
+
+	if (evq_conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_ALL_TYPES) {
+		if (evq_conf->nb_atomic_order_sequences != 0)
+			sched_type = RTE_SCHED_TYPE_ORDERED;
+		else
+			sched_type = RTE_SCHED_TYPE_PARALLEL;
+	} else
+		sched_type = evq_conf->schedule_type;
+
+	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
+	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
+	cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences;
+
+	if (sched_type != RTE_SCHED_TYPE_ORDERED) {
+		cfg.num_sequence_numbers = 0;
+		cfg.num_qid_inflights = 2048;
+	}
+
+	/* App should set this to the number of hardware flows they want, not
+	 * the overall number of flows they're going to use. E.g. if app is
+	 * using 64 flows and sets compression to 64, best-case they'll get
+	 * 64 unique hashed flows in hardware.
+	 */
+	switch (evq_conf->nb_atomic_flows) {
+	/* Valid DLB2 compression levels */
+	case 64:
+	case 128:
+	case 256:
+	case 512:
+	case (1 * 1024): /* 1K */
+	case (2 * 1024): /* 2K */
+	case (4 * 1024): /* 4K */
+	case (64 * 1024): /* 64K */
+		cfg.lock_id_comp_level = evq_conf->nb_atomic_flows;
+		break;
+	default:
+		/* Invalid compression level */
+		cfg.lock_id_comp_level = 0; /* no compression */
+	}
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create LB event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	qm_qid = cfg.response.id;
+
+	/* Save off queue config for debug, resource lookups, and reconfig */
+	queue->num_qid_inflights = cfg.num_qid_inflights;
+	queue->num_atm_inflights = cfg.num_atomic_inflights;
+
+	queue->sched_type = sched_type;
+	queue->config_state = DLB2_CONFIGURED;
+
+	DLB2_LOG_DBG("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n",
+		     qm_qid,
+		     cfg.num_atomic_inflights,
+		     cfg.num_sequence_numbers,
+		     cfg.num_qid_inflights);
+
+	return qm_qid;
+}
+
+static int
+dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int32_t qm_qid;
+
+	if (queue_conf->nb_atomic_order_sequences)
+		dlb2_program_sn_allocation(dlb2, queue_conf);
+
+	qm_qid = dlb2_hw_create_ldb_queue(dlb2, ev_queue, queue_conf);
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the load-balanced queue\n");
+
+		return qm_qid;
+	}
+
+	dlb2->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int dlb2_num_dir_queues_setup(struct dlb2_eventdev *dlb2)
+{
+	int i, num = 0;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].setup_done &&
+		    dlb2->ev_queues[i].qm_queue.is_directed)
+			num++;
+	}
+
+	return num;
+}
+
+static void
+dlb2_queue_link_teardown(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue)
+{
+	struct dlb2_eventdev_port *ev_port;
+	int i, j;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (!ev_port->link[j].valid ||
+			    ev_port->link[j].queue_id != ev_queue->id)
+				continue;
+
+			ev_port->link[j].valid = false;
+			ev_port->num_links--;
+		}
+	}
+
+	ev_queue->num_links = 0;
+}
+
+static int
+dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
+			  uint8_t ev_qid,
+			  const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_queue *ev_queue;
+	int ret;
+
+	if (queue_conf == NULL)
+		return -EINVAL;
+
+	if (ev_qid >= dlb2->num_queues)
+		return -EINVAL;
+
+	ev_queue = &dlb2->ev_queues[ev_qid];
+
+	ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg &
+		RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+	ev_queue->id = ev_qid;
+	ev_queue->conf = *queue_conf;
+
+	if (!ev_queue->qm_queue.is_directed) {
+		ret = dlb2_eventdev_ldb_queue_setup(dev, ev_queue, queue_conf);
+	} else {
+		/* The directed queue isn't setup until link time, at which
+		 * point we know its directed port ID. Directed queue setup
+		 * will only fail if this queue is already setup or there are
+		 * no directed queues left to configure.
+		 */
+		ret = 0;
+
+		ev_queue->qm_queue.config_state = DLB2_NOT_CONFIGURED;
+
+		if (ev_queue->setup_done ||
+		    dlb2_num_dir_queues_setup(dlb2) == dlb2->num_dir_queues)
+			ret = -EINVAL;
+	}
+
+	/* Tear down pre-existing port->queue links */
+	if (!ret && dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_queue_link_teardown(dlb2, ev_queue);
+
+	if (!ret)
+		ev_queue->setup_done = true;
+
+	return ret;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -696,6 +1007,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a829b9b..4c07574 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -31,3 +31,15 @@ int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				struct dlb2_create_sched_domain_args *args);
 
 void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_ldb_queue_args *cfg);
+
+int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_sn_allocation_args *args);
+
+int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_set_sn_allocation_args *args);
+
+int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_occupancy_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 6663dab..4c88fe0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -30,4 +30,16 @@ extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				 struct dlb2_create_sched_domain_args *args);
 
 extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+extern int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_queue_args *cfg);
+
+extern int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_allocation_args *args);
+
+extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_set_sn_allocation_args *args);
+
+extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				  struct dlb2_get_sn_occupancy_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 63a68a6..e6ea0d7 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3509,3 +3509,467 @@ unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
 
 	return num;
 }
+
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_ldb_queue *queue,
+				     struct dlb2_create_ldb_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+	union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+	union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+	union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+	union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+	union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+	union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+	union dlb2_sys_ldb_qid_its r7 = { {0} };
+	union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+	union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+	union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+	union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+	union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+	union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+	struct dlb2_sn_group *sn_group;
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r3.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	r4.field.limit = args->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+	r5.field.limit = queue->aqed_limit;
+
+	if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+		r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+		    r5.val);
+
+	switch (args->lock_id_comp_level) {
+	case 64:
+		r6.field.compress_code = 1;
+		break;
+	case 128:
+		r6.field.compress_code = 2;
+		break;
+	case 256:
+		r6.field.compress_code = 3;
+		break;
+	case 512:
+		r6.field.compress_code = 4;
+		break;
+	case 1024:
+		r6.field.compress_code = 5;
+		break;
+	case 2048:
+		r6.field.compress_code = 6;
+		break;
+	case 4096:
+		r6.field.compress_code = 7;
+		break;
+	case 0:
+	case 65536:
+		r6.field.compress_code = 0;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+		    r6.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r7.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+		    r7.val);
+
+	r8.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+		    r8.val);
+
+	r9.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+		    r9.val);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	r10.field.qid_fid_limit = 512;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+		    r10.val);
+
+	/* Configure SNs */
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	r11.field.mode = sn_group->mode;
+	r11.field.slot = queue->sn_slot;
+	r11.field.grp  = sn_group->id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+	r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+	r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		r0.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+		r1.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+		r2.field.vqid = queue->id.virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+			    r2.val);
+	}
+
+	r13.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+				  struct dlb2_ldb_queue *queue,
+				  struct dlb2_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb2_sn_group_full(group)) {
+			slot = dlb2_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no sequence number slots available\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+				struct dlb2_hw_domain *domain,
+				struct dlb2_ldb_queue *queue,
+				struct dlb2_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_ldb_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (dlb2_list_empty(&domain->avail_ldb_queues)) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb2_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_ldb_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		    args->num_sequence_numbers);
+	DLB2_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		    args->num_qid_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		    args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+
+	dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			    __func__, __LINE__);
+		return ret;
+	}
+
+	dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_queues, &queue->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_queues, &queue->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+						unsigned int group_id,
+						unsigned long val)
+{
+	DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+	DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+	DLB2_HW_DBG(hw, "\tValue:    %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val)
+{
+	u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+	union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+	struct dlb2_sn_group *group;
+	int mode;
+
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	group = &hw->rsrcs.sn_groups[group_id];
+
+	/*
+	 * Once the first load-balanced queue using an SN group is configured,
+	 * the group cannot be changed.
+	 */
+	if (group->slot_use_bitmap != 0)
+		return -EPERM;
+
+	for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+		if (val == valid_allocations[mode])
+			break;
+
+	if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+		return -EINVAL;
+
+	group->mode = mode;
+	group->sequence_numbers_per_queue = val;
+
+	r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+	r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+	DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+	dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 285ad2c..14bb04b 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -613,3 +613,13 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
 {
 	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_ldb_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 9310f72..a6cd178 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -150,6 +150,84 @@ dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
 		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
 }
 
+static int
+dlb2_pf_ldb_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_ldb_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_ldb_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw_dev *handle,
+			 struct dlb2_get_sn_occupancy_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_number_occupancy(&dlb2_dev->hw,
+						       args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_numbers(&dlb2_dev->hw, args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_set_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_set_group_sequence_numbers(&dlb2_dev->hw, args->group,
+					      args->num);
+
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -161,6 +239,10 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
+	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
+	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
+	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v8 13/23] event/dlb2: add port setup

[Timothy McDaniel]

Configure the load balanced (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  75 +++
 drivers/event/dlb2/dlb2.c                  | 498 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   8 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 922 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  28 +
 drivers/event/dlb2/pf/dlb2_pf.c            | 180 ++++++
 7 files changed, 1719 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index b154cac..bbd6ac8 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -82,6 +82,81 @@ The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
 the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
 load-balanced queues can use the full 16-bit flow ID range.
 
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 8c1e06d..7898e42 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -999,6 +999,503 @@ dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
 	return ret;
 }
 
+static int
+dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_cq_pop_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_cq_pop_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL)	{
+		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->consume_qe = qe;
+
+	qe->qe_valid = 0;
+	qe->qe_frag = 0;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
+	 * and so on.
+	 */
+	qe->tokens = 0;	/* set at run time */
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_enqueue_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_enqueue_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL) {
+		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->int_arm_qe = qe;
+
+	/* V2 - INT ARM is CQ_TOKEN + FRAG */
+	qe->qe_valid = 0;
+	qe->qe_frag = 1;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
+{
+	int ret, sz;
+
+	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct dlb2_enqueue_qe);
+
+	qm_port->qe4 = rte_zmalloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
+
+	if (qm_port->qe4 == NULL) {
+		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	ret = dlb2_init_consume_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	return 0;
+
+error_exit:
+
+	dlb2_free_qe_mem(qm_port);
+
+	return ret;
+}
+
+static int
+dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_ldb_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_CQ_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	cfg.cq_history_list_size = DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	if (handle->cos_id == DLB2_COS_DEFAULT)
+		cfg.cos_id = 0;
+	else
+		cfg.cos_id = handle->cos_id;
+
+	cfg.cos_strict = 0;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* If there are no directed ports, the kernel driver will ignore this
+	 * port's directed credit settings. Don't use enqueue_depth if it would
+	 * require more directed credits than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2; /* back ptr */
+	/*
+	 * Allocate and init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_ldb_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(qm_port->cfg.ldb));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     qm_port->ldb_credits,
+		     qm_port->dir_credits);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
+
+	return ret;
+}
+
+static void
+dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port)
+{
+	struct dlb2_eventdev_queue *ev_queue;
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (!ev_port->link[i].valid)
+			continue;
+
+		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
+
+		ev_port->link[i].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+}
+
+static int
+dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (dlb2 == NULL || handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* Directed queues are configured at link time. */
+	cfg.queue_id = -1;
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* Don't use enqueue_depth if it would require more directed credits
+	 * than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2;  /* back ptr */
+
+	/*
+	 * Init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE to be aligned
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_dir_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = cfg.cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(qm_port->cfg.dir));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     dir_credit_high_watermark,
+		     ldb_credit_high_watermark);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_setup(struct rte_eventdev *dev,
+			 uint8_t ev_port_id,
+			 const struct rte_event_port_conf *port_conf)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_eventdev_port *ev_port;
+	int ret;
+
+	if (dev == NULL || port_conf == NULL) {
+		DLB2_LOG_ERR("Null parameter\n");
+		return -EINVAL;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
+		return -EINVAL;
+
+	if (port_conf->dequeue_depth >
+		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
+	    port_conf->enqueue_depth >
+		evdev_dlb2_default_info.max_event_port_enqueue_depth)
+		return -EINVAL;
+
+	ev_port = &dlb2->ev_ports[ev_port_id];
+	/* configured? */
+	if (ev_port->setup_done) {
+		DLB2_LOG_ERR("evport %d is already configured\n", ev_port_id);
+		return -EINVAL;
+	}
+
+	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
+		RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (!ev_port->qm_port.is_directed) {
+		ret = dlb2_hw_create_ldb_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the lB port ve portId=%d\n",
+				     ev_port_id);
+
+			return ret;
+		}
+	} else {
+		ret = dlb2_hw_create_dir_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the DIR port\n");
+			return ret;
+		}
+	}
+
+	/* Save off port config for reconfig */
+	ev_port->conf = *port_conf;
+
+	ev_port->id = ev_port_id;
+	ev_port->enq_configured = true;
+	ev_port->setup_done = true;
+	ev_port->inflight_max = port_conf->new_event_threshold;
+	ev_port->implicit_release = !(port_conf->event_port_cfg &
+		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
+	ev_port->outstanding_releases = 0;
+	ev_port->inflight_credits = 0;
+	ev_port->credit_update_quanta = RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
+	ev_port->dlb2 = dlb2; /* reverse link */
+
+	/* Tear down pre-existing port->queue links */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
+
+	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1009,6 +1506,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
+		.port_setup       = dlb2_eventdev_port_setup,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 4c07574..01818c7 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -43,3 +43,11 @@ int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				   struct dlb2_get_sn_occupancy_args *args);
+
+int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_dir_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4c88fe0..a6b923a 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -42,4 +42,12 @@ extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				  struct dlb2_get_sn_occupancy_args *args);
+
+extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_ldb_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_dir_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index e6ea0d7..a633d3e 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3973,3 +3973,925 @@ int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_ldb_port *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_pp2vas r0 = { {0} };
+	union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+		union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_ldb_port *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_ldb_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+	union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+	union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+	union dlb2_chp_hist_list_lim r5 = { {0} };
+	union dlb2_chp_hist_list_base r6 = { {0} };
+	union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+	union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+	union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+	union dlb2_sys_ldb_cq_at r10 = { {0} };
+	union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+	union dlb2_chp_ldb_cq2vas r12 = { {0} };
+	union dlb2_lsp_cq2priov r13 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r14.field.token_count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    r14.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	r5.field.limit = port->hist_list_entry_limit - 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+	r6.field.base = port->hist_list_entry_base;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	r7.field.limit = args->cq_history_list_size;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+	r8.field.push_ptr = r6.field.base;
+	r8.field.generation = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    r8.val);
+
+	r9.field.pop_ptr = r6.field.base;
+	r9.field.generation = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+	/* Disable the port's QID mappings */
+	r13.field.v = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+	return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_ldb_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb2_ldb_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+	if (ret < 0)
+		return ret;
+
+	dlb2_ldb_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_ldb_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		    args->cq_history_list_size);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+	DLB2_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		    args->cos_strict);
+}
+
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_ldb_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+		resp->status = DLB2_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		if (dlb2_list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	} else {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			if (!dlb2_list_empty(&domain->avail_ldb_ports[i]))
+				break;
+		}
+
+		if (i == DLB2_NUM_COS_DOMAINS) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ *	its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	int ret, cos_id, i;
+
+	dlb2_log_create_ldb_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->cos_strict) {
+		cos_id = args->cos_id;
+
+		port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+					  typeof(*port));
+	} else {
+		int idx;
+
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+			port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+						  typeof(*port));
+			if (port)
+				break;
+		}
+
+		cos_id = idx;
+	}
+
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (port->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_ldb_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_ports[cos_id], &port->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_ports[cos_id], &port->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_dir_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+}
+
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+			    bool vdev_req,
+			    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+
+	return NULL;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_dir_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the queue is already configured, validate
+	 * the queue ID, its domain, and whether the queue is configured.
+	 */
+	if (args->queue_id != -1) {
+		struct dlb2_dir_pq_pair *queue;
+
+		queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+						    vdev_req,
+						    domain);
+
+		if (queue == NULL || queue->domain_id.phys_id !=
+				domain->id.phys_id ||
+				!queue->queue_configured) {
+			resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the port's queue is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->queue_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_dir_pq_pair *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_dir_pp2vas r0 = { {0} };
+	union dlb2_sys_dir_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+		union dlb2_sys_dir_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_dir_pq_pair *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_dir_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+	union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+	union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+	union dlb2_sys_dir_cq_fmt r9 = { {0} };
+	union dlb2_sys_dir_cq_at r10 = { {0} };
+	union dlb2_sys_dir_cq_pasid r11 = { {0} };
+	union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r13.field.count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    r13.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.disable_wb_opt = 0;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	r9.field.keep_pf_ppid = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+	return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_dir_pq_pair *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_dir_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb2_dir_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+
+	if (ret < 0)
+		return ret;
+
+	dlb2_dir_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ *	and queue. The port/queue pair have the same ID and name.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *port;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (args->queue_id != -1)
+		port = dlb2_get_domain_used_dir_pq(args->queue_id,
+						   vdev_req,
+						   domain);
+	else
+		port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					  typeof(*port));
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_dir_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs, &port->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs, &port->domain_list);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 14bb04b..2a089f6 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -623,3 +623,31 @@ dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_ldb_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
+
+int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_dir_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a6cd178..2a82b8f 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -228,6 +228,184 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static void *
+dlb2_alloc_coherent_aligned(const struct rte_memzone **mz, uintptr_t *phys,
+			    size_t size, int align)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t core_id = rte_lcore_id();
+	unsigned int socket_id;
+
+	snprintf(mz_name, sizeof(mz_name) - 1, "event_dlb2_pf_%lx",
+		 (unsigned long)rte_get_timer_cycles());
+	if (core_id == (unsigned int)LCORE_ID_ANY)
+		core_id = rte_get_main_lcore();
+	socket_id = rte_lcore_to_socket_id(core_id);
+	*mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
+					 RTE_MEMZONE_IOVA_CONTIG, align);
+	if (*mz == NULL) {
+		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu bytes - %s\n",
+			     size, rte_strerror(rte_errno));
+		*phys = 0;
+		return NULL;
+	}
+	*phys = (*mz)->iova;
+	return (*mz)->addr;
+}
+
+static int
+dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_ldb_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret, cq_alloc_depth;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = false;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	cq_alloc_depth = RTE_MAX(cfg->cq_depth, DLB2_MIN_HARDWARE_CQ_DEPTH);
+	alloc_sz = cq_alloc_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].cq_base = (void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+	return ret;
+}
+
+static int
+dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_dir_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = true;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	alloc_sz = cfg->cq_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
+		(void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -240,6 +418,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v8 14/23] event/dlb2: add port link

[Timothy McDaniel]

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 302 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 633 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
 6 files changed, 1007 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7898e42..5068d6d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1496,6 +1496,307 @@ dlb2_eventdev_port_setup(struct rte_eventdev *dev,
 	return 0;
 }
 
+static int16_t
+dlb2_hw_map_ldb_qid_to_port(struct dlb2_hw_dev *handle,
+			    uint32_t qm_port_id,
+			    uint16_t qm_qid,
+			    uint8_t priority)
+{
+	struct dlb2_map_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	/* Build message */
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+	cfg.priority = EV_TO_DLB2_PRIO(priority);
+
+	ret = dlb2_iface_map_qid(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: map qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		DLB2_LOG_ERR("dlb2: grp=%d, qm_port=%d, qm_qid=%d prio=%d\n",
+			     handle->domain_id, cfg.port_id,
+			     cfg.qid,
+			     cfg.priority);
+	} else {
+		DLB2_LOG_DBG("dlb2: mapped queue %d to qm_port %d\n",
+			     qm_qid, qm_port_id);
+	}
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port,
+			  struct dlb2_eventdev_queue *ev_queue,
+			  uint8_t priority)
+{
+	int first_avail = -1;
+	int ret, i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid) {
+			if (ev_port->link[i].queue_id == ev_queue->id &&
+			    ev_port->link[i].priority == priority) {
+				if (ev_port->link[i].mapped)
+					return 0; /* already mapped */
+				first_avail = i;
+			}
+		} else if (first_avail == -1)
+			first_avail = i;
+	}
+	if (first_avail == -1) {
+		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
+			     ev_port->qm_port.id);
+		return -EINVAL;
+	}
+
+	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
+					  ev_port->qm_port.id,
+					  ev_queue->qm_queue.id,
+					  priority);
+
+	if (!ret)
+		ev_port->link[first_avail].mapped = true;
+
+	return ret;
+}
+
+static int32_t
+dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 int32_t qm_port_id)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_queue_args cfg;
+	int32_t ret;
+
+	/* The directed port is always configured before its queue */
+	cfg.port_id = qm_port_id;
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_dir_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	int32_t qm_qid;
+
+	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port->qm_port.id);
+
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the DIR queue\n");
+		return qm_qid;
+	}
+
+	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int
+dlb2_do_port_link(struct rte_eventdev *dev,
+		  struct dlb2_eventdev_queue *ev_queue,
+		  struct dlb2_eventdev_port *ev_port,
+		  uint8_t prio)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int err;
+
+	/* Don't link until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	if (ev_queue->qm_queue.is_directed)
+		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue, ev_port);
+	else
+		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue, prio);
+
+	if (err) {
+		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
+			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
+			     ev_queue->id, ev_port->id);
+
+		rte_errno = err;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
+			uint8_t queue_id,
+			bool link_exists,
+			int index)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	bool port_is_dir, queue_is_dir;
+
+	if (queue_id > dlb2->num_queues) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	ev_queue = &dlb2->ev_queues[queue_id];
+
+	if (!ev_queue->setup_done &&
+	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	port_is_dir = ev_port->qm_port.is_directed;
+	queue_is_dir = ev_queue->qm_queue.is_directed;
+
+	if (port_is_dir != queue_is_dir) {
+		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
+			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
+			     port_is_dir ? "DIR" : "LDB", ev_port->id);
+
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if there is space for the requested link */
+	if (!link_exists && index == -1) {
+		DLB2_LOG_ERR("no space for new link\n");
+		rte_errno = -ENOSPC;
+		return -1;
+	}
+
+	/* Check if the directed port is already linked */
+	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if the directed queue is already linked */
+	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
+			     ev_queue->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
+			const uint8_t queues[], const uint8_t priorities[],
+			uint16_t nb_links)
+
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i, j;
+
+	RTE_SET_USED(dev);
+
+	if (ev_port == NULL) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!ev_port->setup_done &&
+	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
+	 * queues pointer.
+	 */
+	if (nb_links == 0) {
+		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
+		     nb_links,
+		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
+		     ev_port->id);
+
+	for (i = 0; i < nb_links; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		uint8_t queue_id, prio;
+		bool found = false;
+		int index = -1;
+
+		queue_id = queues[i];
+		prio = priorities[i];
+
+		/* Check if the link already exists. */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].valid) {
+				if (ev_port->link[j].queue_id == queue_id) {
+					found = true;
+					index = j;
+					break;
+				}
+			} else if (index == -1) {
+				index = j;
+			}
+
+		/* could not link */
+		if (index == -1)
+			break;
+
+		/* Check if already linked at the requested priority */
+		if (found && ev_port->link[j].priority == prio)
+			continue;
+
+		if (dlb2_validate_port_link(ev_port, queue_id, found, index))
+			break; /* return index of offending queue */
+
+		ev_queue = &dlb2->ev_queues[queue_id];
+
+		if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+			break; /* return index of offending queue */
+
+		ev_queue->num_links++;
+
+		ev_port->link[index].queue_id = queue_id;
+		ev_port->link[index].priority = prio;
+		ev_port->link[index].valid = true;
+		/* Entry already exists?  If so, then must be prio change */
+		if (!found)
+			ev_port->num_links++;
+	}
+	return i;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1507,6 +1808,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_link        = dlb2_eventdev_port_link,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 01818c7..7d1ba73 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -51,3 +51,9 @@ int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 				  struct dlb2_create_dir_port_args *cfg,
 				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_dir_queue_args *cfg);
+
+int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+			  struct dlb2_map_qid_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index a6b923a..4eddd14 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -50,4 +50,10 @@ extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 struct dlb2_create_dir_port_args *cfg,
 					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+					struct dlb2_create_dir_queue_args *cfg);
+
+extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+				 struct dlb2_map_qid_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index a633d3e..14ad262 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -4895,3 +4895,636 @@ int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_dir_pq_pair *queue,
+				     struct dlb2_create_dir_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_dir_vasqid_v r0 = { {0} };
+	union dlb2_sys_dir_qid_its r1 = { {0} };
+	union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+	union dlb2_sys_dir_qid_v r5 = { {0} };
+
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r0.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r1.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+		    r1.val);
+
+	r2.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+		    r2.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		r3.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+		r4.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+	}
+
+	r5.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+	queue->queue_configured = true;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_dir_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_dir_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = dlb2_get_domain_used_dir_pq(args->port_id,
+						   vdev_req,
+						   domain);
+
+		if (port == NULL || port->domain_id.phys_id !=
+				domain->id.phys_id || !port->port_configured) {
+			resp->status = DLB2_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the queue's port is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->port_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->port_id != -1)
+		queue = dlb2_get_domain_used_dir_pq(args->port_id,
+						    vdev_req,
+						    domain);
+	else
+		queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					   typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &queue->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs,
+			      &queue->domain_list);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot,
+					      struct dlb2_map_qid_args *args)
+{
+	union dlb2_lsp_cq2priov r0;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+	r0.field.v |= 1 << slot;
+	r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+	dlb2_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+					      struct dlb2_ldb_queue *queue,
+					      struct dlb2_cmd_response *resp)
+{
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB2_QUEUE_UNMAPPED;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+			  bool vdev_req,
+			  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter)
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+			      bool vdev_req,
+			      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+
+		DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+	}
+
+	return NULL;
+}
+
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+				    u32 domain_id,
+				    struct dlb2_map_qid_args *args,
+				    struct dlb2_cmd_response *resp,
+				    bool vdev_req,
+				    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	struct dlb2_ldb_queue *queue;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB2_QID_PRIORITIES) {
+		resp->status = DLB2_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_map_qid_args *args,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	DLB2_HW_DBG(hw, "\tPriority:  %d\n",
+		    args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	int ret, i, id;
+	u8 prio;
+
+	dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_map_qid_args(hw,
+				       domain_id,
+				       args,
+				       resp,
+				       vdev_req,
+				       vdev_id);
+	if (ret)
+		return ret;
+
+	prio = args->priority;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	if (port->num_pending_removals)
+		dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].pending_priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+		if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb2_qid_map_state st;
+
+			if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+				DLB2_HW_ERR(hw,
+					    "[%s():%d] Internal error: port slot tracking failed\n",
+					    __func__, __LINE__);
+				return -EFAULT;
+			}
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb2_port_slot_state_transition(hw, port, queue,
+							      i, st);
+			if (ret)
+				return ret;
+
+			DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 2a089f6..a010e25 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -651,3 +651,13 @@ dlb2_pf_create_dir_port(struct dlb2_hw *hw,
 				       NOT_VF_REQ,
 				       PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_dir_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 2a82b8f..37be4b1 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -406,6 +406,54 @@ dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_dir_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_dir_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_dir_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
+		struct dlb2_map_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_map_qid(&dlb2_dev->hw,
+			      handle->domain_id,
+			      cfg,
+			      &response,
+			      false,
+			      0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -419,7 +467,9 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
 	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
+	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v8 15/23] event/dlb2: add port unlink and port unlinks in progress

[Timothy McDaniel]

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 163 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 283 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  51 ++++++
 5 files changed, 509 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 5068d6d..c221a61 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1797,6 +1797,166 @@ dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
 	return i;
 }
 
+static int16_t
+dlb2_hw_unmap_ldb_qid_from_port(struct dlb2_hw_dev *handle,
+				uint32_t qm_port_id,
+				uint16_t qm_qid)
+{
+	struct dlb2_unmap_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+
+	ret = dlb2_iface_unmap_qid(handle, &cfg);
+	if (ret < 0)
+		DLB2_LOG_ERR("dlb2: unmap qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_detach_ldb(struct dlb2_eventdev *dlb2,
+			    struct dlb2_eventdev_port *ev_port,
+			    struct dlb2_eventdev_queue *ev_queue)
+{
+	int ret, i;
+
+	/* Don't unlink until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid &&
+		    ev_port->link[i].queue_id == ev_queue->id)
+			break; /* found */
+	}
+
+	/* This is expected with eventdev API!
+	 * It blindly attemmpts to unmap all queues.
+	 */
+	if (i == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_LOG_DBG("dlb2: ignoring LB QID %d not mapped for qm_port %d.\n",
+			     ev_queue->qm_queue.id,
+			     ev_port->qm_port.id);
+		return 0;
+	}
+
+	ret = dlb2_hw_unmap_ldb_qid_from_port(&dlb2->qm_instance,
+					      ev_port->qm_port.id,
+					      ev_queue->qm_queue.id);
+	if (!ret)
+		ev_port->link[i].mapped = false;
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
+			  uint8_t queues[], uint16_t nb_unlinks)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (queues == NULL || nb_unlinks == 0) {
+		DLB2_LOG_DBG("dlb2: queues is NULL or nb_unlinks is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	/* FIXME: How to handle unlink on a directed port? */
+	if (ev_port->qm_port.is_directed) {
+		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
+			     ev_port->id);
+		rte_errno = 0;
+		return nb_unlinks; /* as if success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	for (i = 0; i < nb_unlinks; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		int ret, j;
+
+		if (queues[i] >= dlb2->num_queues) {
+			DLB2_LOG_ERR("dlb2: invalid queue id %d\n", queues[i]);
+			rte_errno = -EINVAL;
+			return i; /* return index of offending queue */
+		}
+
+		ev_queue = &dlb2->ev_queues[queues[i]];
+
+		/* Does a link exist? */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].queue_id == queues[i] &&
+			    ev_port->link[j].valid)
+				break;
+
+		if (j == DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+			continue;
+
+		ret = dlb2_event_queue_detach_ldb(dlb2, ev_port, ev_queue);
+		if (ret) {
+			DLB2_LOG_ERR("unlink err=%d for port %d queue %d\n",
+				     ret, ev_port->id, queues[i]);
+			rte_errno = -ENOENT;
+			return i; /* return index of offending queue */
+		}
+
+		ev_port->link[j].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+
+	return nb_unlinks;
+}
+
+static int
+dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
+				       void *event_port)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	struct dlb2_pending_port_unmaps_args cfg;
+	int ret;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	cfg.port_id = ev_port->qm_port.id;
+	dlb2 = ev_port->dlb2;
+	handle = &dlb2->qm_instance;
+	ret = dlb2_iface_pending_port_unmaps(handle, &cfg);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: num_unlinks_in_progress ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1809,6 +1969,9 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
 		.port_link        = dlb2_eventdev_port_link,
+		.port_unlink      = dlb2_eventdev_port_unlink,
+		.port_unlinks_in_progress =
+				    dlb2_eventdev_port_unlinks_in_progress,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 7d1ba73..bd241f3 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -57,3 +57,9 @@ int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 			  struct dlb2_map_qid_args *cfg);
+
+int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+			    struct dlb2_unmap_qid_args *cfg);
+
+int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4eddd14..e892036 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -56,4 +56,10 @@ extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 				 struct dlb2_map_qid_args *cfg);
+
+extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+				   struct dlb2_unmap_qid_args *cfg);
+
+extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 14ad262..4a3d96d 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5528,3 +5528,286 @@ int dlb2_hw_map_qid(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_unmap_qid_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+		DLB2_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			    hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+				      u32 domain_id,
+				      struct dlb2_unmap_qid_args *args,
+				      struct dlb2_cmd_response *resp,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	struct dlb2_ldb_port *port;
+	int slot;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			    __func__, args->qid);
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		return 0;
+
+	resp->status = DLB2_ST_INVALID_QID;
+	return -EINVAL;
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	bool unmap_complete;
+	int i, ret, id;
+
+	dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_unmap_qid_args(hw,
+					 domain_id,
+					 args,
+					 resp,
+					 vdev_req,
+					 vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+		st = DLB2_QUEUE_UNMAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		st = DLB2_QUEUE_UNMAP_IN_PROG;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB2_QUEUE_MAPPED;
+	if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: no available CQ slots\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asynchronous procedure. It requires
+	 * stopping the DLB2 from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb2_ldb_port_cq_disable(hw, port);
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete && !os_worker_active(hw))
+		os_schedule_work(hw);
+
+unmap_qid_done:
+	resp->status = 0;
+
+	return 0;
+}
+
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+				  struct dlb2_pending_port_unmaps_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+
+	dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 37be4b1..33ea73e 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -454,6 +454,55 @@ dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw_dev *handle,
+		  struct dlb2_unmap_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_unmap_qid(&dlb2_dev->hw,
+				handle->domain_id,
+				cfg,
+				&response,
+				false,
+				0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
+			    struct dlb2_pending_port_unmaps_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_pending_port_unmaps(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -470,6 +519,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
+	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v8 16/23] event/dlb2: add eventdev start

[Timothy McDaniel]

Add support for the eventdev start entry point.
We delay initializing some resources until
eventdev start, since the number of linked queues can be
used to determine if we are dealing with a ldb or dir resource.
If this is a device restart, then the previous configuration
will be reapplied.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 133 +++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   3 +
 drivers/event/dlb2/dlb2_iface.h            |   3 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 123 ++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +++
 drivers/event/dlb2/pf/dlb2_pf.c            |  25 ++++++
 6 files changed, 297 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index c221a61..051836d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1957,6 +1957,138 @@ dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
 	return cfg.response.id;
 }
 
+static int
+dlb2_eventdev_reapply_configuration(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret, i;
+
+	/* If an event queue or port was previously configured, but hasn't been
+	 * reconfigured, reapply its original configuration.
+	 */
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+
+		ev_queue = &dlb2->ev_queues[i];
+
+		if (ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_queue_setup(dev, i, &ev_queue->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure queue %d", i);
+			return ret;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+
+		if (ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_port_setup(dev, i, &ev_port->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure ev_port %d",
+				     i);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_apply_port_links(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int i;
+
+	/* Perform requested port->queue links */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+		int j;
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			struct dlb2_eventdev_queue *ev_queue;
+			uint8_t prio, queue_id;
+
+			if (!ev_port->link[j].valid)
+				continue;
+
+			prio = ev_port->link[j].priority;
+			queue_id = ev_port->link[j].queue_id;
+
+			if (dlb2_validate_port_link(ev_port, queue_id, true, j))
+				return -EINVAL;
+
+			ev_queue = &dlb2->ev_queues[queue_id];
+
+			if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_start(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_start_domain_args cfg;
+	int ret, i;
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_ERR("bad state %d for dev_start\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return -EINVAL;
+	}
+	dlb2->run_state = DLB2_RUN_STATE_STARTING;
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	/* If the device was configured more than once, some event ports and/or
+	 * queues may need to be reconfigured.
+	 */
+	ret = dlb2_eventdev_reapply_configuration(dev);
+	if (ret)
+		return ret;
+
+	/* The DLB PMD delays port links until the device is started. */
+	ret = dlb2_eventdev_apply_port_links(dev);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		if (!dlb2->ev_ports[i].setup_done) {
+			DLB2_LOG_ERR("dlb2: port %d not setup", i);
+			return -ESTALE;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0) {
+			DLB2_LOG_ERR("dlb2: queue %d is not linked", i);
+			return -ENOLINK;
+		}
+	}
+
+	ret = dlb2_iface_sched_domain_start(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: sched_domain_start ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STARTED;
+	DLB2_LOG_DBG("dlb2: sched_domain_start completed OK\n");
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1964,6 +2096,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.dev_start        = dlb2_eventdev_start,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index bd241f3..a86191d 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -63,3 +63,6 @@ int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				     struct dlb2_start_domain_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index e892036..7ead374 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -62,4 +62,7 @@ extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				struct dlb2_start_domain_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 4a3d96d..c14a2f3 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5811,3 +5811,126 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 struct dlb2_cmd_response *resp,
+					 bool vdev_req,
+					 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_start_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 start domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_hw_start_domain() - Lock the domain configuration
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @arg: User-provided arguments (unused, here for ioctl callback template).
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int
+dlb2_hw_start_domain(struct dlb2_hw *hw,
+		     u32 domain_id,
+		     __attribute((unused)) struct dlb2_start_domain_args *arg,
+		     struct dlb2_cmd_response *resp,
+		     bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_queue;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+	RTE_SET_USED(arg);
+	RTE_SET_USED(iter);
+
+	dlb2_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb2_verify_start_domain_args(hw,
+					    domain_id,
+					    resp,
+					    vdev_req,
+					    vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB2 will drop all
+	 * incoming traffic to those queues.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, ldb_queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r0.val);
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+	}
+
+	dlb2_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index a010e25..aee8336 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -661,3 +661,13 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_start_domain(struct dlb2_hw *hw,
+		     u32 id,
+		     struct dlb2_start_domain_args *args,
+		     struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_start_domain(hw, id, args, resp, NOT_VF_REQ,
+				    PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 33ea73e..fff2e57 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -503,6 +503,30 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
 
 	return ret;
 }
+
+static int
+dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
+			   struct dlb2_start_domain_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_start_domain(&dlb2_dev->hw,
+				   handle->domain_id,
+				   cfg,
+				   &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -520,6 +544,7 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v8 17/23] event/dlb2: add enqueue and its burst variants

[Timothy McDaniel]

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst | 118 +++++++++
 drivers/event/dlb2/dlb2.c     | 578 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 696 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index bbd6ac8..aa8bf01 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -163,6 +163,124 @@ Flow ID
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Hardware Credits
+~~~~~~~~~~~~~~~~
+
+DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
+event storage, with each unit of storage represented by a credit. A port spends
+a credit to enqueue an event, and hardware refills the ports with credits as the
+events are scheduled to ports. Refills come from credit pools, and each port is
+a member of a load-balanced credit pool and a directed credit pool. The
+load-balanced credits are used to enqueue to load-balanced queues, and directed
+credits are used for directed queues.
+
+A DLB2 eventdev contains one load-balanced and one directed credit pool. These
+pools' sizes are controlled by the nb_events_limit field in struct
+rte_event_dev_config. The load-balanced pool is sized to contain
+nb_events_limit credits, and the directed pool is sized to contain
+nb_events_limit/4 credits. The directed pool size can be overridden with the
+num_dir_credits vdev argument, like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,num_dir_credits=<value>
+
+This can be used if the default allocation is too low or too high for the
+specific application needs. The PMD also supports a vdev arg that limits the
+max_num_events reported by rte_event_dev_info_get():
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,max_num_events=<value>
+
+By default, max_num_events is reported as the total available load-balanced
+credits. If multiple DLB2-based applications are being used, it may be desirable
+to control how many load-balanced credits each application uses, particularly
+when application(s) are written to configure nb_events_limit equal to the
+reported max_num_events.
+
+Each port is a member of both credit pools. A port's credit allocation is
+defined by its low watermark, high watermark, and refill quanta. These three
+parameters are calculated by the dlb PMD like so:
+
+- The load-balanced high watermark is set to the port's enqueue_depth.
+  The directed high watermark is set to the minimum of the enqueue_depth and
+  the directed pool size divided by the total number of ports.
+- The refill quanta is set to half the high watermark.
+- The low watermark is set to the minimum of 16 and the refill quanta.
+
+When the eventdev is started, each port is pre-allocated a high watermark's
+worth of credits. For example, if an eventdev contains four ports with enqueue
+depths of 32 and a load-balanced credit pool size of 4096, each port will start
+with 32 load-balanced credits, and there will be 3968 credits available to
+replenish the ports. Thus, a single port is not capable of enqueueing up to the
+nb_events_limit (without any events being dequeued), since the other ports are
+retaining their initial credit allocation; in short, all ports must enqueue in
+order to reach the limit.
+
+If a port attempts to enqueue and has no credits available, the enqueue
+operation will fail and the application must retry the enqueue. Credits are
+replenished asynchronously by the DLB2 hardware.
+
+Software Credits
+~~~~~~~~~~~~~~~~
+
+The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
+credit scheme on top of the hardware credit scheme in order to comply with
+the per-port backpressure described in the eventdev API.
+
+The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
+credit when it enqueues to a queue, and credits are later replenished after the
+events are dequeued and released.
+
+In the software credit scheme, a credit is consumed when a new (.op =
+RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
+replenished when the event is released from the system (either explicitly with
+RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
+
+In this model, an event is "in the system" from its first enqueue into eventdev
+until it is last dequeued. If the event goes through multiple event queues, it
+is still considered "in the system" while a worker thread is processing it.
+
+A port will fail to enqueue if the number of events in the system exceeds its
+``new_event_threshold`` (specified at port setup time). A port will also fail
+to enqueue if it lacks enough hardware credits to enqueue; load-balanced
+credits are used to enqueue to a load-balanced queue, and directed credits are
+used to enqueue to a directed queue.
+
+The out-of-credit situations are typically transient, and an eventdev
+application using the DLB2 ought to retry its enqueues if they fail.
+If enqueue fails, DLB2 PMD sets rte_errno as follows:
+
+- -ENOSPC: Credit exhaustion (either hardware or software)
+- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
+
+Depending on the pipeline the application has constructed, it's possible to
+enter a credit deadlock scenario wherein the worker thread lacks the credit
+to enqueue an event, and it must dequeue an event before it can recover the
+credit. If the worker thread retries its enqueue indefinitely, it will not
+make forward progress. Such deadlock is possible if the application has event
+"loops", in which an event in dequeued from queue A and later enqueued back to
+queue A.
+
+Due to this, workers should stop retrying after a time, release the events it
+is attempting to enqueue, and dequeue more events. It is important that the
+worker release the events and don't simply set them aside to retry the enqueue
+again later, because the port has limited history list size (by default, twice
+the port's dequeue_depth).
+
+Priority
+~~~~~~~~
+
+The DLB2 supports event priority and per-port queue service priority, as
+described in the eventdev header file. The DLB2 does not support 'global' event
+queue priority established at queue creation time.
+
+DLB2 supports 8 event and queue service priority levels. For both priority
+types, the PMD uses the upper three bits of the priority field to determine the
+DLB2 priority, discarding the 5 least significant bits. The 5 least significant
+event priority bits are not preserved when an event is enqueued.
+
 Reconfiguration
 ~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 051836d..7d2cf76 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2089,6 +2089,578 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
 	return 0;
 }
 
+static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
+	{
+		/* Load-balanced cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
+	},
+	{
+		/* Directed cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
+	},
+};
+
+static inline uint32_t
+dlb2_port_credits_get(struct dlb2_port *qm_port,
+		      enum dlb2_hw_queue_types type)
+{
+	uint32_t credits = *qm_port->credit_pool[type];
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	if (unlikely(credits < batch_size))
+		batch_size = credits;
+
+	if (likely(credits &&
+		   __atomic_compare_exchange_n(
+			qm_port->credit_pool[type],
+			&credits, credits - batch_size, false,
+			__ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
+		return batch_size;
+	else
+		return 0;
+}
+
+static inline void
+dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port)
+{
+	uint16_t quanta = ev_port->credit_update_quanta;
+
+	if (ev_port->inflight_credits >= quanta * 2) {
+		/* Replenish credits, saving one quanta for enqueues */
+		uint16_t val = ev_port->inflight_credits - quanta;
+
+		__atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits -= val;
+	}
+}
+
+static inline int
+dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
+						__ATOMIC_SEQ_CST);
+	const int num = 1;
+
+	if (unlikely(ev_port->inflight_max < sw_inflights)) {
+		DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
+		rte_errno = -ENOSPC;
+		return 1;
+	}
+
+	if (ev_port->inflight_credits < num) {
+		/* check if event enqueue brings ev_port over max threshold */
+		uint32_t credit_update_quanta = ev_port->credit_update_quanta;
+
+		if (sw_inflights + credit_update_quanta >
+				dlb2->new_event_limit) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_new_event_limit,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+
+		__atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
+				   __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits += (credit_update_quanta);
+
+		if (ev_port->inflight_credits < num) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_inflight_credits,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_ldb_credits == 0)) {
+		qm_port->cached_ldb_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_LDB_QUEUE);
+		if (unlikely(qm_port->cached_ldb_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
+			1);
+			DLB2_LOG_DBG("ldb credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_dir_credits == 0)) {
+		qm_port->cached_dir_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_DIR_QUEUE);
+		if (unlikely(qm_port->cached_dir_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
+			1);
+			DLB2_LOG_DBG("dir credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static __rte_always_inline void
+dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
+	      struct process_local_port_data *port_data)
+{
+	dlb2_movdir64b(port_data->pp_addr, qe4);
+}
+
+static inline int
+dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_cq_pop_qe *qe;
+
+	RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
+
+	qe = qm_port->consume_qe;
+
+	qe->tokens = num - 1;
+
+	/* No store fence needed since no pointer is being sent, and CQ token
+	 * pops can be safely reordered with other HCWs.
+	 */
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	dlb2_movntdq_single(port_data->pp_addr, qe);
+
+	DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
+
+	qm_port->owed_tokens = 0;
+
+	return 0;
+}
+
+static inline void
+dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
+		   bool do_sfence,
+		   struct process_local_port_data *port_data)
+{
+	/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
+	 * application writes complete before enqueueing the QE.
+	 */
+	if (do_sfence)
+		rte_wmb();
+
+	dlb2_pp_write(qm_port->qe4, port_data);
+}
+
+static inline void
+dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
+{
+	struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
+	int num = qm_port->owed_tokens;
+
+	qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
+	qe[idx].tokens = num - 1;
+
+	qm_port->owed_tokens = 0;
+}
+
+static inline void
+dlb2_event_build_hcws(struct dlb2_port *qm_port,
+		      const struct rte_event ev[],
+		      int num,
+		      uint8_t *sched_type,
+		      uint8_t *queue_id)
+{
+	struct dlb2_enqueue_qe *qe;
+	uint16_t sched_word[4];
+	__m128i sse_qe[2];
+	int i;
+
+	qe = qm_port->qe4;
+
+	sse_qe[0] = _mm_setzero_si128();
+	sse_qe[1] = _mm_setzero_si128();
+
+	switch (num) {
+	case 4:
+		/* Construct the metadata portion of two HCWs in one 128b SSE
+		 * register. HCW metadata is constructed in the SSE registers
+		 * like so:
+		 * sse_qe[0][63:0]:   qe[0]'s metadata
+		 * sse_qe[0][127:64]: qe[1]'s metadata
+		 * sse_qe[1][63:0]:   qe[2]'s metadata
+		 * sse_qe[1][127:64]: qe[3]'s metadata
+		 */
+
+		/* Convert the event operation into a command byte and store it
+		 * in the metadata:
+		 * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
+		 * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
+		 * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
+		 * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
+		 */
+#define DLB2_QE_CMD_BYTE 7
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[0].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[1].op],
+				DLB2_QE_CMD_BYTE + 8);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[2].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[3].op],
+				DLB2_QE_CMD_BYTE + 8);
+
+		/* Store priority, scheduling type, and queue ID in the sched
+		 * word array because these values are re-used when the
+		 * destination is a directed queue.
+		 */
+		sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
+				sched_type[0] << 8 |
+				queue_id[0];
+		sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
+				sched_type[1] << 8 |
+				queue_id[1];
+		sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
+				sched_type[2] << 8 |
+				queue_id[2];
+		sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
+				sched_type[3] << 8 |
+				queue_id[3];
+
+		/* Store the event priority, scheduling type, and queue ID in
+		 * the metadata:
+		 * sse_qe[0][31:16] = sched_word[0]
+		 * sse_qe[0][95:80] = sched_word[1]
+		 * sse_qe[1][31:16] = sched_word[2]
+		 * sse_qe[1][95:80] = sched_word[3]
+		 */
+#define DLB2_QE_QID_SCHED_WORD 1
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[0],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[1],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[2],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[3],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+
+		/* If the destination is a load-balanced queue, store the lock
+		 * ID. If it is a directed queue, DLB places this field in
+		 * bytes 10-11 of the received QE, so we format it accordingly:
+		 * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
+		 * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
+		 * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
+		 * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
+		 */
+#define DLB2_QE_LOCK_ID_WORD 2
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[0] == DLB2_SCHED_DIRECTED) ?
+					sched_word[0] : ev[0].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[1] == DLB2_SCHED_DIRECTED) ?
+					sched_word[1] : ev[1].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[2] == DLB2_SCHED_DIRECTED) ?
+					sched_word[2] : ev[2].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[3] == DLB2_SCHED_DIRECTED) ?
+					sched_word[3] : ev[3].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+
+		/* Store the event type and sub event type in the metadata:
+		 * sse_qe[0][15:0]  = flow_id[0]
+		 * sse_qe[0][79:64] = flow_id[1]
+		 * sse_qe[1][15:0]  = flow_id[2]
+		 * sse_qe[1][79:64] = flow_id[3]
+		 */
+#define DLB2_QE_EV_TYPE_WORD 0
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[0].sub_event_type << 8 |
+						ev[0].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[1].sub_event_type << 8 |
+						ev[1].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[2].sub_event_type << 8 |
+						ev[2].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[3].sub_event_type << 8 |
+						ev[3].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+
+		/* Store the metadata to memory (use the double-precision
+		 * _mm_storeh_pd because there is no integer function for
+		 * storing the upper 64b):
+		 * qe[0] metadata = sse_qe[0][63:0]
+		 * qe[1] metadata = sse_qe[0][127:64]
+		 * qe[2] metadata = sse_qe[1][63:0]
+		 * qe[3] metadata = sse_qe[1][127:64]
+		 */
+		_mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
+		_mm_storeh_pd((double *)&qe[1].u.opaque_data,
+			      (__m128d)sse_qe[0]);
+		_mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
+		_mm_storeh_pd((double *)&qe[3].u.opaque_data,
+			      (__m128d)sse_qe[1]);
+
+		qe[0].data = ev[0].u64;
+		qe[1].data = ev[1].u64;
+		qe[2].data = ev[2].u64;
+		qe[3].data = ev[3].u64;
+
+		break;
+	case 3:
+	case 2:
+	case 1:
+		/* At least one QE will be valid, so only zero out three */
+		qe[1].cmd_byte = 0;
+		qe[2].cmd_byte = 0;
+		qe[3].cmd_byte = 0;
+
+		for (i = 0; i < num; i++) {
+			qe[i].cmd_byte =
+				cmd_byte_map[qm_port->is_directed][ev[i].op];
+			qe[i].sched_type = sched_type[i];
+			qe[i].data = ev[i].u64;
+			qe[i].qid = queue_id[i];
+			qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
+			qe[i].lock_id = ev[i].flow_id;
+			if (sched_type[i] == DLB2_SCHED_DIRECTED) {
+				struct dlb2_msg_info *info =
+					(struct dlb2_msg_info *)&qe[i].lock_id;
+
+				info->qid = queue_id[i];
+				info->sched_type = DLB2_SCHED_DIRECTED;
+				info->priority = qe[i].priority;
+			}
+			qe[i].u.event_type.major = ev[i].event_type;
+			qe[i].u.event_type.sub = ev[i].sub_event_type;
+		}
+		break;
+	}
+}
+
+static inline int
+dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
+			struct dlb2_port *qm_port,
+			const struct rte_event ev[],
+			uint8_t *sched_type,
+			uint8_t *queue_id)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	uint16_t *cached_credits = NULL;
+	struct dlb2_queue *qm_queue;
+
+	ev_queue = &dlb2->ev_queues[ev->queue_id];
+	qm_queue = &ev_queue->qm_queue;
+	*queue_id = qm_queue->id;
+
+	/* Ignore sched_type and hardware credits on release events */
+	if (ev->op == RTE_EVENT_OP_RELEASE)
+		goto op_check;
+
+	if (!qm_queue->is_directed) {
+		/* Load balanced destination queue */
+
+		if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_ldb_credits;
+
+		switch (ev->sched_type) {
+		case RTE_SCHED_TYPE_ORDERED:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
+			if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
+				DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
+					     *queue_id);
+				rte_errno = -EINVAL;
+				return 1;
+			}
+			*sched_type = DLB2_SCHED_ORDERED;
+			break;
+		case RTE_SCHED_TYPE_ATOMIC:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
+			*sched_type = DLB2_SCHED_ATOMIC;
+			break;
+		case RTE_SCHED_TYPE_PARALLEL:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
+			if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
+				*sched_type = DLB2_SCHED_ORDERED;
+			else
+				*sched_type = DLB2_SCHED_UNORDERED;
+			break;
+		default:
+			DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
+			DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
+			rte_errno = -EINVAL;
+			return 1;
+		}
+	} else {
+		/* Directed destination queue */
+
+		if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_dir_credits;
+
+		DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
+
+		*sched_type = DLB2_SCHED_DIRECTED;
+	}
+
+op_check:
+	switch (ev->op) {
+	case RTE_EVENT_OP_NEW:
+		/* Check that a sw credit is available */
+		if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		ev_port->inflight_credits--;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_FORWARD:
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_RELEASE:
+		ev_port->inflight_credits++;
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+
+		/* Replenish s/w credits if enough are cached */
+		dlb2_replenish_sw_credits(dlb2, ev_port);
+		break;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+	if (ev->op != RTE_EVENT_OP_RELEASE) {
+		DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
+		DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
+	}
+#endif
+
+	return 0;
+}
+
+static inline uint16_t
+dlb2_event_enqueue_burst(void *event_port,
+			 const struct rte_event events[],
+			 uint16_t num)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct process_local_port_data *port_data;
+	int i, cnt;
+
+	RTE_ASSERT(ev_port->enq_configured);
+	RTE_ASSERT(events != NULL);
+
+	cnt = 0;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
+		int j = 0;
+
+		for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
+			const struct rte_event *ev = &events[i + j];
+
+			if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
+						    &sched_types[j],
+						    &queue_ids[j]))
+				break;
+		}
+
+		if (j == 0)
+			break;
+
+		dlb2_event_build_hcws(qm_port, &events[i], j,
+				      sched_types, queue_ids);
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+
+		if (j < DLB2_NUM_QES_PER_CACHE_LINE)
+			break;
+	}
+
+	return cnt;
+}
+
+static inline uint16_t
+dlb2_event_enqueue(void *event_port,
+		   const struct rte_event events[])
+{
+	return dlb2_event_enqueue_burst(event_port, events, 1);
+}
+
+static uint16_t
+dlb2_event_enqueue_new_burst(void *event_port,
+			     const struct rte_event events[],
+			     uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
+static uint16_t
+dlb2_event_enqueue_forward_burst(void *event_port,
+				 const struct rte_event events[],
+				 uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2112,7 +2684,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
 	};
 
+	/* Expose PMD's eventdev interface */
+
 	dev->dev_ops = &dlb2_eventdev_entry_ops;
+	dev->enqueue = dlb2_event_enqueue;
+	dev->enqueue_burst = dlb2_event_enqueue_burst;
+	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
+	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
 }
 
 int
-- 
2.6.4

Re: [dpdk-dev] [PATCH v8 17/23] event/dlb2: add enqueue and its burst variants

[Jerin Jacob]

On Sat, Oct 31, 2020 at 7:36 AM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> Add support for enqueue and its variants.
>
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> Reviewed-by: Gage Eads <gage.eads@intel.com>
> ---
>  doc/guides/eventdevs/dlb2.rst | 118 +++++++++
>  drivers/event/dlb2/dlb2.c     | 578 ++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 696 insertions(+)



Found following build error at this patch with clang

[2262/2494] Linking target drivers/librte_bus_vmbus.so.21.0
[2263/2494] Compiling C object
drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_dlb2.c.o
FAILED: drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_dlb2.c.o
ccache clang -Idrivers/libtmp_rte_event_dlb2.a.p -Idrivers
-I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
-Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
-I../config -Ilib/librte_eal/include -I../lib/librte_eal/i
nclude -Ilib/librte_eal/linux/include
-I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
-I../lib/librte_eal/x86/include -Ilib/librte_eal/common
-I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
-Ilib/librte_kvargs
 -I../lib/librte_kvargs -Ilib/librte_metrics -I../lib/librte_metrics
-Ilib/librte_telemetry -I../lib/librte_telemetry -Ilib/librte_ring
-I../lib/librte_ring -Ilib/librte_ethdev -I../lib/librte_ethdev
-Ilib/librte_net -I../lib/librte_net -Il
ib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
-I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
-Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_rcu
-I../lib/librte_rcu -Ilib/librte_timer -I../lib/librte_timer -
Ilib/librte_cryptodev -I../lib/librte_cryptodev -Ilib/librte_pci
-I../lib/librte_pci -Idrivers/bus/pci -I../drivers/bus/pci
-I../drivers/bus/pci/linux -Xclang -fcolor-diagnostics -pipe
-D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2
-g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
-Wformat-nonliteral -Wformat-security -Wmissing-declarations
-Wmissing-prototypes -Wnested-externs -Wold-style-definition
-Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -
Wwrite-strings -Wno-address-of-packed-member
-Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
-DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_dlb2.c.o -MF
drivers/libtmp_rt
e_event_dlb2.a.p/event_dlb2_dlb2.c.o.d -o
drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_dlb2.c.o -c
../drivers/event/dlb2/dlb2.c
../drivers/event/dlb2/dlb2.c:2231:1: error: unused function
'dlb2_consume_qe_immediate' [-Werror,-Wunused-function]
dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
^
../drivers/event/dlb2/dlb2.c:2271:1: error: unused function
'dlb2_construct_token_pop_qe' [-Werror,-Wunused-function]
dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
^
2 errors generated.
[2264/2494] Generating rte_event_octeontx.sym_chk with a meson_exe.py
custom command
[2265/2494] Generating rte_baseband_fpga_lte_fec.sym_chk with a
meson_exe.py custom command
[2266/2494] Generating rte_event_octeontx2.sym_chk with a meson_exe.py
custom command
[2267/2494] Generating rte_event_sw.sym_chk with a meson_exe.py custom command
[2268/2494] Linking target lib/librte_stack.so.21.0
[2269/2494] Linking target lib/librte_jobstats.so.21.0
[2270/2494] Linking target lib/librte_cfgfile.so.21.0


>
> diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
> index bbd6ac8..aa8bf01 100644
> --- a/doc/guides/eventdevs/dlb2.rst
> +++ b/doc/guides/eventdevs/dlb2.rst
> @@ -163,6 +163,124 @@ Flow ID
>  The flow ID field is preserved in the event when it is scheduled in the
>  DLB2.
>
> +Hardware Credits
> +~~~~~~~~~~~~~~~~
> +
> +DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
> +event storage, with each unit of storage represented by a credit. A port spends
> +a credit to enqueue an event, and hardware refills the ports with credits as the
> +events are scheduled to ports. Refills come from credit pools, and each port is
> +a member of a load-balanced credit pool and a directed credit pool. The
> +load-balanced credits are used to enqueue to load-balanced queues, and directed
> +credits are used for directed queues.
> +
> +A DLB2 eventdev contains one load-balanced and one directed credit pool. These
> +pools' sizes are controlled by the nb_events_limit field in struct
> +rte_event_dev_config. The load-balanced pool is sized to contain
> +nb_events_limit credits, and the directed pool is sized to contain
> +nb_events_limit/4 credits. The directed pool size can be overridden with the
> +num_dir_credits vdev argument, like so:
> +
> +    .. code-block:: console
> +
> +       --vdev=dlb1_event,num_dir_credits=<value>
> +
> +This can be used if the default allocation is too low or too high for the
> +specific application needs. The PMD also supports a vdev arg that limits the
> +max_num_events reported by rte_event_dev_info_get():
> +
> +    .. code-block:: console
> +
> +       --vdev=dlb1_event,max_num_events=<value>
> +
> +By default, max_num_events is reported as the total available load-balanced
> +credits. If multiple DLB2-based applications are being used, it may be desirable
> +to control how many load-balanced credits each application uses, particularly
> +when application(s) are written to configure nb_events_limit equal to the
> +reported max_num_events.
> +
> +Each port is a member of both credit pools. A port's credit allocation is
> +defined by its low watermark, high watermark, and refill quanta. These three
> +parameters are calculated by the dlb PMD like so:
> +
> +- The load-balanced high watermark is set to the port's enqueue_depth.
> +  The directed high watermark is set to the minimum of the enqueue_depth and
> +  the directed pool size divided by the total number of ports.
> +- The refill quanta is set to half the high watermark.
> +- The low watermark is set to the minimum of 16 and the refill quanta.
> +
> +When the eventdev is started, each port is pre-allocated a high watermark's
> +worth of credits. For example, if an eventdev contains four ports with enqueue
> +depths of 32 and a load-balanced credit pool size of 4096, each port will start
> +with 32 load-balanced credits, and there will be 3968 credits available to
> +replenish the ports. Thus, a single port is not capable of enqueueing up to the
> +nb_events_limit (without any events being dequeued), since the other ports are
> +retaining their initial credit allocation; in short, all ports must enqueue in
> +order to reach the limit.
> +
> +If a port attempts to enqueue and has no credits available, the enqueue
> +operation will fail and the application must retry the enqueue. Credits are
> +replenished asynchronously by the DLB2 hardware.
> +
> +Software Credits
> +~~~~~~~~~~~~~~~~
> +
> +The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
> +credit scheme on top of the hardware credit scheme in order to comply with
> +the per-port backpressure described in the eventdev API.
> +
> +The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
> +credit when it enqueues to a queue, and credits are later replenished after the
> +events are dequeued and released.
> +
> +In the software credit scheme, a credit is consumed when a new (.op =
> +RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
> +replenished when the event is released from the system (either explicitly with
> +RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
> +
> +In this model, an event is "in the system" from its first enqueue into eventdev
> +until it is last dequeued. If the event goes through multiple event queues, it
> +is still considered "in the system" while a worker thread is processing it.
> +
> +A port will fail to enqueue if the number of events in the system exceeds its
> +``new_event_threshold`` (specified at port setup time). A port will also fail
> +to enqueue if it lacks enough hardware credits to enqueue; load-balanced
> +credits are used to enqueue to a load-balanced queue, and directed credits are
> +used to enqueue to a directed queue.
> +
> +The out-of-credit situations are typically transient, and an eventdev
> +application using the DLB2 ought to retry its enqueues if they fail.
> +If enqueue fails, DLB2 PMD sets rte_errno as follows:
> +
> +- -ENOSPC: Credit exhaustion (either hardware or software)
> +- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
> +
> +Depending on the pipeline the application has constructed, it's possible to
> +enter a credit deadlock scenario wherein the worker thread lacks the credit
> +to enqueue an event, and it must dequeue an event before it can recover the
> +credit. If the worker thread retries its enqueue indefinitely, it will not
> +make forward progress. Such deadlock is possible if the application has event
> +"loops", in which an event in dequeued from queue A and later enqueued back to
> +queue A.
> +
> +Due to this, workers should stop retrying after a time, release the events it
> +is attempting to enqueue, and dequeue more events. It is important that the
> +worker release the events and don't simply set them aside to retry the enqueue
> +again later, because the port has limited history list size (by default, twice
> +the port's dequeue_depth).
> +
> +Priority
> +~~~~~~~~
> +
> +The DLB2 supports event priority and per-port queue service priority, as
> +described in the eventdev header file. The DLB2 does not support 'global' event
> +queue priority established at queue creation time.
> +
> +DLB2 supports 8 event and queue service priority levels. For both priority
> +types, the PMD uses the upper three bits of the priority field to determine the
> +DLB2 priority, discarding the 5 least significant bits. The 5 least significant
> +event priority bits are not preserved when an event is enqueued.
> +
>  Reconfiguration
>  ~~~~~~~~~~~~~~~
>
> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> index 051836d..7d2cf76 100644
> --- a/drivers/event/dlb2/dlb2.c
> +++ b/drivers/event/dlb2/dlb2.c
> @@ -2089,6 +2089,578 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
>         return 0;
>  }
>
> +static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
> +       {
> +               /* Load-balanced cmd bytes */
> +               [RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
> +               [RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
> +               [RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
> +       },
> +       {
> +               /* Directed cmd bytes */
> +               [RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
> +               [RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
> +               [RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
> +       },
> +};
> +
> +static inline uint32_t
> +dlb2_port_credits_get(struct dlb2_port *qm_port,
> +                     enum dlb2_hw_queue_types type)
> +{
> +       uint32_t credits = *qm_port->credit_pool[type];
> +       uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
> +
> +       if (unlikely(credits < batch_size))
> +               batch_size = credits;
> +
> +       if (likely(credits &&
> +                  __atomic_compare_exchange_n(
> +                       qm_port->credit_pool[type],
> +                       &credits, credits - batch_size, false,
> +                       __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
> +               return batch_size;
> +       else
> +               return 0;
> +}
> +
> +static inline void
> +dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
> +                         struct dlb2_eventdev_port *ev_port)
> +{
> +       uint16_t quanta = ev_port->credit_update_quanta;
> +
> +       if (ev_port->inflight_credits >= quanta * 2) {
> +               /* Replenish credits, saving one quanta for enqueues */
> +               uint16_t val = ev_port->inflight_credits - quanta;
> +
> +               __atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
> +               ev_port->inflight_credits -= val;
> +       }
> +}
> +
> +static inline int
> +dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
> +                             struct dlb2_eventdev_port *ev_port)
> +{
> +       uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
> +                                               __ATOMIC_SEQ_CST);
> +       const int num = 1;
> +
> +       if (unlikely(ev_port->inflight_max < sw_inflights)) {
> +               DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
> +               rte_errno = -ENOSPC;
> +               return 1;
> +       }
> +
> +       if (ev_port->inflight_credits < num) {
> +               /* check if event enqueue brings ev_port over max threshold */
> +               uint32_t credit_update_quanta = ev_port->credit_update_quanta;
> +
> +               if (sw_inflights + credit_update_quanta >
> +                               dlb2->new_event_limit) {
> +                       DLB2_INC_STAT(
> +                       ev_port->stats.traffic.tx_nospc_new_event_limit,
> +                       1);
> +                       rte_errno = -ENOSPC;
> +                       return 1;
> +               }
> +
> +               __atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
> +                                  __ATOMIC_SEQ_CST);
> +               ev_port->inflight_credits += (credit_update_quanta);
> +
> +               if (ev_port->inflight_credits < num) {
> +                       DLB2_INC_STAT(
> +                       ev_port->stats.traffic.tx_nospc_inflight_credits,
> +                       1);
> +                       rte_errno = -ENOSPC;
> +                       return 1;
> +               }
> +       }
> +
> +       return 0;
> +}
> +
> +static inline int
> +dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
> +{
> +       if (unlikely(qm_port->cached_ldb_credits == 0)) {
> +               qm_port->cached_ldb_credits =
> +                       dlb2_port_credits_get(qm_port,
> +                                             DLB2_LDB_QUEUE);
> +               if (unlikely(qm_port->cached_ldb_credits == 0)) {
> +                       DLB2_INC_STAT(
> +                       qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
> +                       1);
> +                       DLB2_LOG_DBG("ldb credits exhausted\n");
> +                       return 1; /* credits exhausted */
> +               }
> +       }
> +
> +       return 0;
> +}
> +
> +static inline int
> +dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
> +{
> +       if (unlikely(qm_port->cached_dir_credits == 0)) {
> +               qm_port->cached_dir_credits =
> +                       dlb2_port_credits_get(qm_port,
> +                                             DLB2_DIR_QUEUE);
> +               if (unlikely(qm_port->cached_dir_credits == 0)) {
> +                       DLB2_INC_STAT(
> +                       qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
> +                       1);
> +                       DLB2_LOG_DBG("dir credits exhausted\n");
> +                       return 1; /* credits exhausted */
> +               }
> +       }
> +
> +       return 0;
> +}
> +
> +static __rte_always_inline void
> +dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
> +             struct process_local_port_data *port_data)
> +{
> +       dlb2_movdir64b(port_data->pp_addr, qe4);
> +}
> +
> +static inline int
> +dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
> +{
> +       struct process_local_port_data *port_data;
> +       struct dlb2_cq_pop_qe *qe;
> +
> +       RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
> +
> +       qe = qm_port->consume_qe;
> +
> +       qe->tokens = num - 1;
> +
> +       /* No store fence needed since no pointer is being sent, and CQ token
> +        * pops can be safely reordered with other HCWs.
> +        */
> +       port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
> +
> +       dlb2_movntdq_single(port_data->pp_addr, qe);
> +
> +       DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
> +
> +       qm_port->owed_tokens = 0;
> +
> +       return 0;
> +}
> +
> +static inline void
> +dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
> +                  bool do_sfence,
> +                  struct process_local_port_data *port_data)
> +{
> +       /* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
> +        * application writes complete before enqueueing the QE.
> +        */
> +       if (do_sfence)
> +               rte_wmb();
> +
> +       dlb2_pp_write(qm_port->qe4, port_data);
> +}
> +
> +static inline void
> +dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
> +{
> +       struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
> +       int num = qm_port->owed_tokens;
> +
> +       qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
> +       qe[idx].tokens = num - 1;
> +
> +       qm_port->owed_tokens = 0;
> +}
> +
> +static inline void
> +dlb2_event_build_hcws(struct dlb2_port *qm_port,
> +                     const struct rte_event ev[],
> +                     int num,
> +                     uint8_t *sched_type,
> +                     uint8_t *queue_id)
> +{
> +       struct dlb2_enqueue_qe *qe;
> +       uint16_t sched_word[4];
> +       __m128i sse_qe[2];
> +       int i;
> +
> +       qe = qm_port->qe4;
> +
> +       sse_qe[0] = _mm_setzero_si128();
> +       sse_qe[1] = _mm_setzero_si128();
> +
> +       switch (num) {
> +       case 4:
> +               /* Construct the metadata portion of two HCWs in one 128b SSE
> +                * register. HCW metadata is constructed in the SSE registers
> +                * like so:
> +                * sse_qe[0][63:0]:   qe[0]'s metadata
> +                * sse_qe[0][127:64]: qe[1]'s metadata
> +                * sse_qe[1][63:0]:   qe[2]'s metadata
> +                * sse_qe[1][127:64]: qe[3]'s metadata
> +                */
> +
> +               /* Convert the event operation into a command byte and store it
> +                * in the metadata:
> +                * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
> +                * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
> +                * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
> +                * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
> +                */
> +#define DLB2_QE_CMD_BYTE 7
> +               sse_qe[0] = _mm_insert_epi8(sse_qe[0],
> +                               cmd_byte_map[qm_port->is_directed][ev[0].op],
> +                               DLB2_QE_CMD_BYTE);
> +               sse_qe[0] = _mm_insert_epi8(sse_qe[0],
> +                               cmd_byte_map[qm_port->is_directed][ev[1].op],
> +                               DLB2_QE_CMD_BYTE + 8);
> +               sse_qe[1] = _mm_insert_epi8(sse_qe[1],
> +                               cmd_byte_map[qm_port->is_directed][ev[2].op],
> +                               DLB2_QE_CMD_BYTE);
> +               sse_qe[1] = _mm_insert_epi8(sse_qe[1],
> +                               cmd_byte_map[qm_port->is_directed][ev[3].op],
> +                               DLB2_QE_CMD_BYTE + 8);
> +
> +               /* Store priority, scheduling type, and queue ID in the sched
> +                * word array because these values are re-used when the
> +                * destination is a directed queue.
> +                */
> +               sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
> +                               sched_type[0] << 8 |
> +                               queue_id[0];
> +               sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
> +                               sched_type[1] << 8 |
> +                               queue_id[1];
> +               sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
> +                               sched_type[2] << 8 |
> +                               queue_id[2];
> +               sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
> +                               sched_type[3] << 8 |
> +                               queue_id[3];
> +
> +               /* Store the event priority, scheduling type, and queue ID in
> +                * the metadata:
> +                * sse_qe[0][31:16] = sched_word[0]
> +                * sse_qe[0][95:80] = sched_word[1]
> +                * sse_qe[1][31:16] = sched_word[2]
> +                * sse_qe[1][95:80] = sched_word[3]
> +                */
> +#define DLB2_QE_QID_SCHED_WORD 1
> +               sse_qe[0] = _mm_insert_epi16(sse_qe[0],
> +                                            sched_word[0],
> +                                            DLB2_QE_QID_SCHED_WORD);
> +               sse_qe[0] = _mm_insert_epi16(sse_qe[0],
> +                                            sched_word[1],
> +                                            DLB2_QE_QID_SCHED_WORD + 4);
> +               sse_qe[1] = _mm_insert_epi16(sse_qe[1],
> +                                            sched_word[2],
> +                                            DLB2_QE_QID_SCHED_WORD);
> +               sse_qe[1] = _mm_insert_epi16(sse_qe[1],
> +                                            sched_word[3],
> +                                            DLB2_QE_QID_SCHED_WORD + 4);
> +
> +               /* If the destination is a load-balanced queue, store the lock
> +                * ID. If it is a directed queue, DLB places this field in
> +                * bytes 10-11 of the received QE, so we format it accordingly:
> +                * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
> +                * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
> +                * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
> +                * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
> +                */
> +#define DLB2_QE_LOCK_ID_WORD 2
> +               sse_qe[0] = _mm_insert_epi16(sse_qe[0],
> +                               (sched_type[0] == DLB2_SCHED_DIRECTED) ?
> +                                       sched_word[0] : ev[0].flow_id,
> +                               DLB2_QE_LOCK_ID_WORD);
> +               sse_qe[0] = _mm_insert_epi16(sse_qe[0],
> +                               (sched_type[1] == DLB2_SCHED_DIRECTED) ?
> +                                       sched_word[1] : ev[1].flow_id,
> +                               DLB2_QE_LOCK_ID_WORD + 4);
> +               sse_qe[1] = _mm_insert_epi16(sse_qe[1],
> +                               (sched_type[2] == DLB2_SCHED_DIRECTED) ?
> +                                       sched_word[2] : ev[2].flow_id,
> +                               DLB2_QE_LOCK_ID_WORD);
> +               sse_qe[1] = _mm_insert_epi16(sse_qe[1],
> +                               (sched_type[3] == DLB2_SCHED_DIRECTED) ?
> +                                       sched_word[3] : ev[3].flow_id,
> +                               DLB2_QE_LOCK_ID_WORD + 4);
> +
> +               /* Store the event type and sub event type in the metadata:
> +                * sse_qe[0][15:0]  = flow_id[0]
> +                * sse_qe[0][79:64] = flow_id[1]
> +                * sse_qe[1][15:0]  = flow_id[2]
> +                * sse_qe[1][79:64] = flow_id[3]
> +                */
> +#define DLB2_QE_EV_TYPE_WORD 0
> +               sse_qe[0] = _mm_insert_epi16(sse_qe[0],
> +                                            ev[0].sub_event_type << 8 |
> +                                               ev[0].event_type,
> +                                            DLB2_QE_EV_TYPE_WORD);
> +               sse_qe[0] = _mm_insert_epi16(sse_qe[0],
> +                                            ev[1].sub_event_type << 8 |
> +                                               ev[1].event_type,
> +                                            DLB2_QE_EV_TYPE_WORD + 4);
> +               sse_qe[1] = _mm_insert_epi16(sse_qe[1],
> +                                            ev[2].sub_event_type << 8 |
> +                                               ev[2].event_type,
> +                                            DLB2_QE_EV_TYPE_WORD);
> +               sse_qe[1] = _mm_insert_epi16(sse_qe[1],
> +                                            ev[3].sub_event_type << 8 |
> +                                               ev[3].event_type,
> +                                            DLB2_QE_EV_TYPE_WORD + 4);
> +
> +               /* Store the metadata to memory (use the double-precision
> +                * _mm_storeh_pd because there is no integer function for
> +                * storing the upper 64b):
> +                * qe[0] metadata = sse_qe[0][63:0]
> +                * qe[1] metadata = sse_qe[0][127:64]
> +                * qe[2] metadata = sse_qe[1][63:0]
> +                * qe[3] metadata = sse_qe[1][127:64]
> +                */
> +               _mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
> +               _mm_storeh_pd((double *)&qe[1].u.opaque_data,
> +                             (__m128d)sse_qe[0]);
> +               _mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
> +               _mm_storeh_pd((double *)&qe[3].u.opaque_data,
> +                             (__m128d)sse_qe[1]);
> +
> +               qe[0].data = ev[0].u64;
> +               qe[1].data = ev[1].u64;
> +               qe[2].data = ev[2].u64;
> +               qe[3].data = ev[3].u64;
> +
> +               break;
> +       case 3:
> +       case 2:
> +       case 1:
> +               /* At least one QE will be valid, so only zero out three */
> +               qe[1].cmd_byte = 0;
> +               qe[2].cmd_byte = 0;
> +               qe[3].cmd_byte = 0;
> +
> +               for (i = 0; i < num; i++) {
> +                       qe[i].cmd_byte =
> +                               cmd_byte_map[qm_port->is_directed][ev[i].op];
> +                       qe[i].sched_type = sched_type[i];
> +                       qe[i].data = ev[i].u64;
> +                       qe[i].qid = queue_id[i];
> +                       qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
> +                       qe[i].lock_id = ev[i].flow_id;
> +                       if (sched_type[i] == DLB2_SCHED_DIRECTED) {
> +                               struct dlb2_msg_info *info =
> +                                       (struct dlb2_msg_info *)&qe[i].lock_id;
> +
> +                               info->qid = queue_id[i];
> +                               info->sched_type = DLB2_SCHED_DIRECTED;
> +                               info->priority = qe[i].priority;
> +                       }
> +                       qe[i].u.event_type.major = ev[i].event_type;
> +                       qe[i].u.event_type.sub = ev[i].sub_event_type;
> +               }
> +               break;
> +       }
> +}
> +
> +static inline int
> +dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
> +                       struct dlb2_port *qm_port,
> +                       const struct rte_event ev[],
> +                       uint8_t *sched_type,
> +                       uint8_t *queue_id)
> +{
> +       struct dlb2_eventdev *dlb2 = ev_port->dlb2;
> +       struct dlb2_eventdev_queue *ev_queue;
> +       uint16_t *cached_credits = NULL;
> +       struct dlb2_queue *qm_queue;
> +
> +       ev_queue = &dlb2->ev_queues[ev->queue_id];
> +       qm_queue = &ev_queue->qm_queue;
> +       *queue_id = qm_queue->id;
> +
> +       /* Ignore sched_type and hardware credits on release events */
> +       if (ev->op == RTE_EVENT_OP_RELEASE)
> +               goto op_check;
> +
> +       if (!qm_queue->is_directed) {
> +               /* Load balanced destination queue */
> +
> +               if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
> +                       rte_errno = -ENOSPC;
> +                       return 1;
> +               }
> +               cached_credits = &qm_port->cached_ldb_credits;
> +
> +               switch (ev->sched_type) {
> +               case RTE_SCHED_TYPE_ORDERED:
> +                       DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
> +                       if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
> +                               DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
> +                                            *queue_id);
> +                               rte_errno = -EINVAL;
> +                               return 1;
> +                       }
> +                       *sched_type = DLB2_SCHED_ORDERED;
> +                       break;
> +               case RTE_SCHED_TYPE_ATOMIC:
> +                       DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
> +                       *sched_type = DLB2_SCHED_ATOMIC;
> +                       break;
> +               case RTE_SCHED_TYPE_PARALLEL:
> +                       DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
> +                       if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
> +                               *sched_type = DLB2_SCHED_ORDERED;
> +                       else
> +                               *sched_type = DLB2_SCHED_UNORDERED;
> +                       break;
> +               default:
> +                       DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
> +                       DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
> +                       rte_errno = -EINVAL;
> +                       return 1;
> +               }
> +       } else {
> +               /* Directed destination queue */
> +
> +               if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
> +                       rte_errno = -ENOSPC;
> +                       return 1;
> +               }
> +               cached_credits = &qm_port->cached_dir_credits;
> +
> +               DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
> +
> +               *sched_type = DLB2_SCHED_DIRECTED;
> +       }
> +
> +op_check:
> +       switch (ev->op) {
> +       case RTE_EVENT_OP_NEW:
> +               /* Check that a sw credit is available */
> +               if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
> +                       rte_errno = -ENOSPC;
> +                       return 1;
> +               }
> +               ev_port->inflight_credits--;
> +               (*cached_credits)--;
> +               break;
> +       case RTE_EVENT_OP_FORWARD:
> +               /* Check for outstanding_releases underflow. If this occurs,
> +                * the application is not using the EVENT_OPs correctly; for
> +                * example, forwarding or releasing events that were not
> +                * dequeued.
> +                */
> +               RTE_ASSERT(ev_port->outstanding_releases > 0);
> +               ev_port->outstanding_releases--;
> +               qm_port->issued_releases++;
> +               (*cached_credits)--;
> +               break;
> +       case RTE_EVENT_OP_RELEASE:
> +               ev_port->inflight_credits++;
> +               /* Check for outstanding_releases underflow. If this occurs,
> +                * the application is not using the EVENT_OPs correctly; for
> +                * example, forwarding or releasing events that were not
> +                * dequeued.
> +                */
> +               RTE_ASSERT(ev_port->outstanding_releases > 0);
> +               ev_port->outstanding_releases--;
> +               qm_port->issued_releases++;
> +
> +               /* Replenish s/w credits if enough are cached */
> +               dlb2_replenish_sw_credits(dlb2, ev_port);
> +               break;
> +       }
> +
> +       DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
> +       DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
> +
> +#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
> +       if (ev->op != RTE_EVENT_OP_RELEASE) {
> +               DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
> +               DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
> +       }
> +#endif
> +
> +       return 0;
> +}
> +
> +static inline uint16_t
> +dlb2_event_enqueue_burst(void *event_port,
> +                        const struct rte_event events[],
> +                        uint16_t num)
> +{
> +       struct dlb2_eventdev_port *ev_port = event_port;
> +       struct dlb2_port *qm_port = &ev_port->qm_port;
> +       struct process_local_port_data *port_data;
> +       int i, cnt;
> +
> +       RTE_ASSERT(ev_port->enq_configured);
> +       RTE_ASSERT(events != NULL);
> +
> +       cnt = 0;
> +
> +       port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
> +
> +       for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
> +               uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
> +               uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
> +               int j = 0;
> +
> +               for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
> +                       const struct rte_event *ev = &events[i + j];
> +
> +                       if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
> +                                                   &sched_types[j],
> +                                                   &queue_ids[j]))
> +                               break;
> +               }
> +
> +               if (j == 0)
> +                       break;
> +
> +               dlb2_event_build_hcws(qm_port, &events[i], j,
> +                                     sched_types, queue_ids);
> +
> +               dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
> +
> +               cnt += j;
> +
> +               if (j < DLB2_NUM_QES_PER_CACHE_LINE)
> +                       break;
> +       }
> +
> +       return cnt;
> +}
> +
> +static inline uint16_t
> +dlb2_event_enqueue(void *event_port,
> +                  const struct rte_event events[])
> +{
> +       return dlb2_event_enqueue_burst(event_port, events, 1);
> +}
> +
> +static uint16_t
> +dlb2_event_enqueue_new_burst(void *event_port,
> +                            const struct rte_event events[],
> +                            uint16_t num)
> +{
> +       return dlb2_event_enqueue_burst(event_port, events, num);
> +}
> +
> +static uint16_t
> +dlb2_event_enqueue_forward_burst(void *event_port,
> +                                const struct rte_event events[],
> +                                uint16_t num)
> +{
> +       return dlb2_event_enqueue_burst(event_port, events, num);
> +}
> +
>  static void
>  dlb2_entry_points_init(struct rte_eventdev *dev)
>  {
> @@ -2112,7 +2684,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
>                 .xstats_reset       = dlb2_eventdev_xstats_reset,
>         };
>
> +       /* Expose PMD's eventdev interface */
> +
>         dev->dev_ops = &dlb2_eventdev_entry_ops;
> +       dev->enqueue = dlb2_event_enqueue;
> +       dev->enqueue_burst = dlb2_event_enqueue_burst;
> +       dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
> +       dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
>  }
>
>  int
> --
> 2.6.4
>

Re: [dpdk-dev] [PATCH v8 17/23] event/dlb2: add enqueue and its burst variants

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Saturday, October 31, 2020 4:52 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <Erik.G.Carrillo@intel.com>;
> Eads, Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>; Thomas
> Monjalon <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH v8 17/23] event/dlb2: add enqueue and its burst
> variants
> 
> On Sat, Oct 31, 2020 at 7:36 AM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > Add support for enqueue and its variants.
> >
> > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > ---
> >  doc/guides/eventdevs/dlb2.rst | 118 +++++++++
> >  drivers/event/dlb2/dlb2.c     | 578
> ++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 696 insertions(+)
> 
> 
> 
> Found following build error at this patch with clang
> 
> [2262/2494] Linking target drivers/librte_bus_vmbus.so.21.0
> [2263/2494] Compiling C object
> drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_dlb2.c.o
> FAILED: drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_dlb2.c.o
> ccache clang -Idrivers/libtmp_rte_event_dlb2.a.p -Idrivers
> -I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
> -Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
> -I../config -Ilib/librte_eal/include -I../lib/librte_eal/i
> nclude -Ilib/librte_eal/linux/include
> -I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
> -I../lib/librte_eal/x86/include -Ilib/librte_eal/common
> -I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
> -Ilib/librte_kvargs
>  -I../lib/librte_kvargs -Ilib/librte_metrics -I../lib/librte_metrics
> -Ilib/librte_telemetry -I../lib/librte_telemetry -Ilib/librte_ring
> -I../lib/librte_ring -Ilib/librte_ethdev -I../lib/librte_ethdev
> -Ilib/librte_net -I../lib/librte_net -Il
> ib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
> -I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
> -Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_rcu
> -I../lib/librte_rcu -Ilib/librte_timer -I../lib/librte_timer -
> Ilib/librte_cryptodev -I../lib/librte_cryptodev -Ilib/librte_pci
> -I../lib/librte_pci -Idrivers/bus/pci -I../drivers/bus/pci
> -I../drivers/bus/pci/linux -Xclang -fcolor-diagnostics -pipe
> -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2
> -g -include rte_config.h -Wextra -Wcast-qual -Wdeprecated
> -Wformat-nonliteral -Wformat-security -Wmissing-declarations
> -Wmissing-prototypes -Wnested-externs -Wold-style-definition
> -Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef -
> Wwrite-strings -Wno-address-of-packed-member
> -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=native
> -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
> drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_dlb2.c.o -MF
> drivers/libtmp_rt
> e_event_dlb2.a.p/event_dlb2_dlb2.c.o.d -o
> drivers/libtmp_rte_event_dlb2.a.p/event_dlb2_dlb2.c.o -c
> ../drivers/event/dlb2/dlb2.c
> ../drivers/event/dlb2/dlb2.c:2231:1: error: unused function
> 'dlb2_consume_qe_immediate' [-Werror,-Wunused-function]
> dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
> ^
> ../drivers/event/dlb2/dlb2.c:2271:1: error: unused function
> 'dlb2_construct_token_pop_qe' [-Werror,-Wunused-function]
> dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
> ^
> 2 errors generated.
> [2264/2494] Generating rte_event_octeontx.sym_chk with a meson_exe.py
> custom command
> [2265/2494] Generating rte_baseband_fpga_lte_fec.sym_chk with a
> meson_exe.py custom command
> [2266/2494] Generating rte_event_octeontx2.sym_chk with a meson_exe.py
> custom command
> [2267/2494] Generating rte_event_sw.sym_chk with a meson_exe.py custom
> command
> [2268/2494] Linking target lib/librte_stack.so.21.0
> [2269/2494] Linking target lib/librte_jobstats.so.21.0
> [2270/2494] Linking target lib/librte_cfgfile.so.21.0
> 

Fixed in local repo. Will be present in next patch-set.

Sorry that I can't seem to clean these all up in a single commit, and instead have been
fixing them incrementally. I have built every patch of every patch-set, and no
errors are reported with gcc. Unfortunately, we do not have clang installed on our servers, and I have not yet
been able to find a way to get gcc to flag these errors.  I have tried setting
cflags += '-Wunsued' in my meson.build file, but that did not help.

[dpdk-dev] [PATCH v8 18/23] event/dlb2: add dequeue and its burst variants

[Timothy McDaniel]

Add support for dequeue, dequeue_burst, ...

DLB2 does not currently support interrupts, but instead use
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB2 supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst |  21 ++
 drivers/event/dlb2/dlb2.c     | 757 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 778 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index aa8bf01..b9f57fd 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -314,6 +314,27 @@ The PMD does not support the following configuration sequences:
 This sequence is not supported because the event device must be reconfigured
 before its ports or queues can be.
 
+Deferred Scheduling
+~~~~~~~~~~~~~~~~~~~
+
+The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
+dequeued event before returning from rte_event_dequeue_burst(). This frees the
+corresponding entries in the CQ, which enables the DLB2 to schedule more events
+to it.
+
+To support applications seeking finer-grained scheduling control -- for example
+deferring scheduling to get the best possible priority scheduling and
+load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
+the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
+call. This mode only applies to load-balanced event ports with dequeue depth of
+1.
+
+To enable deferred scheduling, use the defer_sched vdev argument like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,defer_sched=on
+
 Atomic Inflights Allocation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7d2cf76..b9082bc 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -26,6 +26,7 @@
 #include <rte_log.h>
 #include <rte_malloc.h>
 #include <rte_mbuf.h>
+#include <rte_power_intrinsics.h>
 #include <rte_prefetch.h>
 #include <rte_ring.h>
 #include <rte_string_fns.h>
@@ -2661,9 +2662,756 @@ dlb2_event_enqueue_forward_burst(void *event_port,
 	return dlb2_event_enqueue_burst(event_port, events, num);
 }
 
+static inline void
+dlb2_port_credits_inc(struct dlb2_port *qm_port, int num)
+{
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	/* increment port credits, and return to pool if exceeds threshold */
+	if (!qm_port->is_directed) {
+		qm_port->cached_ldb_credits += num;
+		if (qm_port->cached_ldb_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_LDB_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_ldb_credits -= batch_size;
+		}
+	} else {
+		qm_port->cached_dir_credits += num;
+		if (qm_port->cached_dir_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_DIR_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_dir_credits -= batch_size;
+		}
+	}
+}
+
+static inline int
+dlb2_dequeue_wait(struct dlb2_eventdev *dlb2,
+		  struct dlb2_eventdev_port *ev_port,
+		  struct dlb2_port *qm_port,
+		  uint64_t timeout,
+		  uint64_t start_ticks)
+{
+	struct process_local_port_data *port_data;
+	uint64_t elapsed_ticks;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	elapsed_ticks = rte_get_timer_cycles() - start_ticks;
+
+	/* Wait/poll time expired */
+	if (elapsed_ticks >= timeout) {
+		return 1;
+	} else if (dlb2->umwait_allowed) {
+		volatile struct dlb2_dequeue_qe *cq_base;
+		union {
+			uint64_t raw_qe[2];
+			struct dlb2_dequeue_qe qe;
+		} qe_mask;
+		uint64_t expected_value;
+		volatile uint64_t *monitor_addr;
+
+		qe_mask.qe.cq_gen = 1; /* set mask */
+
+		cq_base = port_data->cq_base;
+		monitor_addr = (volatile uint64_t *)(volatile void *)
+			&cq_base[qm_port->cq_idx];
+		monitor_addr++; /* cq_gen bit is in second 64bit location */
+
+		if (qm_port->gen_bit)
+			expected_value = qe_mask.raw_qe[1];
+		else
+			expected_value = 0;
+
+		rte_power_monitor(monitor_addr, expected_value,
+				  qe_mask.raw_qe[1], timeout + start_ticks,
+				  sizeof(uint64_t));
+
+		DLB2_INC_STAT(ev_port->stats.traffic.rx_umonitor_umwait, 1);
+	} else {
+		uint64_t poll_interval = RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL;
+		uint64_t curr_ticks = rte_get_timer_cycles();
+		uint64_t init_ticks = curr_ticks;
+
+		while ((curr_ticks - start_ticks < timeout) &&
+		       (curr_ticks - init_ticks < poll_interval))
+			curr_ticks = rte_get_timer_cycles();
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
+			 struct dlb2_port *qm_port,
+			 struct rte_event *events,
+			 struct dlb2_dequeue_qe *qes,
+			 int cnt)
+{
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	int i, num, evq_id;
+
+	for (i = 0, num = 0; i < cnt; i++) {
+		struct dlb2_dequeue_qe *qe = &qes[i];
+		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
+			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+			[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+		};
+
+		/* Fill in event information.
+		 * Note that flow_id must be embedded in the data by
+		 * the app, such as the mbuf RSS hash field if the data
+		 * buffer is a mbuf.
+		 */
+		if (unlikely(qe->error)) {
+			DLB2_LOG_ERR("QE error bit ON\n");
+			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
+			dlb2_consume_qe_immediate(qm_port, 1);
+			continue; /* Ignore */
+		}
+
+		events[num].u64 = qe->data;
+		events[num].flow_id = qe->flow_id;
+		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
+		events[num].event_type = qe->u.event_type.major;
+		events[num].sub_event_type = qe->u.event_type.sub;
+		events[num].sched_type = sched_type_map[qe->sched_type];
+		events[num].impl_opaque = qe->qid_depth;
+
+		/* qid not preserved for directed queues */
+		if (qm_port->is_directed)
+			evq_id = ev_port->link[0].queue_id;
+		else
+			evq_id = qid_mappings[qe->qid];
+
+		events[num].queue_id = evq_id;
+		DLB2_INC_STAT(
+			ev_port->stats.queue[evq_id].qid_depth[qe->qid_depth],
+			1);
+		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type], 1);
+		num++;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num);
+
+	return num;
+}
+
+static inline int
+dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
+			      struct dlb2_port *qm_port,
+			      struct rte_event *events,
+			      struct dlb2_dequeue_qe *qes)
+{
+	int sched_type_map[] = {
+		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+	};
+	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	__m128i sse_evt[2];
+
+	/* In the unlikely case that any of the QE error bits are set, process
+	 * them one at a time.
+	 */
+	if (unlikely(qes[0].error || qes[1].error ||
+		     qes[2].error || qes[3].error))
+		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
+						 qes, num_events);
+
+	events[0].u64 = qes[0].data;
+	events[1].u64 = qes[1].data;
+	events[2].u64 = qes[2].data;
+	events[3].u64 = qes[3].data;
+
+	/* Construct the metadata portion of two struct rte_events
+	 * in one 128b SSE register. Event metadata is constructed in the SSE
+	 * registers like so:
+	 * sse_evt[0][63:0]:   event[0]'s metadata
+	 * sse_evt[0][127:64]: event[1]'s metadata
+	 * sse_evt[1][63:0]:   event[2]'s metadata
+	 * sse_evt[1][127:64]: event[3]'s metadata
+	 */
+	sse_evt[0] = _mm_setzero_si128();
+	sse_evt[1] = _mm_setzero_si128();
+
+	/* Convert the hardware queue ID to an event queue ID and store it in
+	 * the metadata:
+	 * sse_evt[0][47:40]   = qid_mappings[qes[0].qid]
+	 * sse_evt[0][111:104] = qid_mappings[qes[1].qid]
+	 * sse_evt[1][47:40]   = qid_mappings[qes[2].qid]
+	 * sse_evt[1][111:104] = qid_mappings[qes[3].qid]
+	 */
+#define DLB_EVENT_QUEUE_ID_BYTE 5
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[0].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[1].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[2].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[3].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+
+	/* Convert the hardware priority to an event priority and store it in
+	 * the metadata, while also returning the queue depth status
+	 * value captured by the hardware, storing it in impl_opaque, which can
+	 * be read by the application but not modified
+	 * sse_evt[0][55:48]   = DLB2_TO_EV_PRIO(qes[0].priority)
+	 * sse_evt[0][63:56]   = qes[0].qid_depth
+	 * sse_evt[0][119:112] = DLB2_TO_EV_PRIO(qes[1].priority)
+	 * sse_evt[0][127:120] = qes[1].qid_depth
+	 * sse_evt[1][55:48]   = DLB2_TO_EV_PRIO(qes[2].priority)
+	 * sse_evt[1][63:56]   = qes[2].qid_depth
+	 * sse_evt[1][119:112] = DLB2_TO_EV_PRIO(qes[3].priority)
+	 * sse_evt[1][127:120] = qes[3].qid_depth
+	 */
+#define DLB_EVENT_PRIO_IMPL_OPAQUE_WORD 3
+#define DLB_BYTE_SHIFT 8
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[0].priority) |
+			(qes[0].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[1].priority) |
+			(qes[1].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[2].priority) |
+			(qes[2].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[3].priority) |
+			(qes[3].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+
+	/* Write the event type, sub event type, and flow_id to the event
+	 * metadata.
+	 * sse_evt[0][31:0]   = qes[0].flow_id |
+	 *			qes[0].u.event_type.major << 28 |
+	 *			qes[0].u.event_type.sub << 20;
+	 * sse_evt[0][95:64]  = qes[1].flow_id |
+	 *			qes[1].u.event_type.major << 28 |
+	 *			qes[1].u.event_type.sub << 20;
+	 * sse_evt[1][31:0]   = qes[2].flow_id |
+	 *			qes[2].u.event_type.major << 28 |
+	 *			qes[2].u.event_type.sub << 20;
+	 * sse_evt[1][95:64]  = qes[3].flow_id |
+	 *			qes[3].u.event_type.major << 28 |
+	 *			qes[3].u.event_type.sub << 20;
+	 */
+#define DLB_EVENT_EV_TYPE_DW 0
+#define DLB_EVENT_EV_TYPE_SHIFT 28
+#define DLB_EVENT_SUB_EV_TYPE_SHIFT 20
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[0].flow_id |
+			qes[0].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[0].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[1].flow_id |
+			qes[1].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[1].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[2].flow_id |
+			qes[2].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[2].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[3].flow_id |
+			qes[3].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT  |
+			qes[3].u.event_type.sub << DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+
+	/* Write the sched type to the event metadata. 'op' and 'rsvd' are not
+	 * set:
+	 * sse_evt[0][39:32]  = sched_type_map[qes[0].sched_type] << 6
+	 * sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6
+	 * sse_evt[1][39:32]  = sched_type_map[qes[2].sched_type] << 6
+	 * sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6
+	 */
+#define DLB_EVENT_SCHED_TYPE_BYTE 4
+#define DLB_EVENT_SCHED_TYPE_SHIFT 6
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[0].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[1].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[2].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[3].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+
+	/* Store the metadata to the event (use the double-precision
+	 * _mm_storeh_pd because there is no integer function for storing the
+	 * upper 64b):
+	 * events[0].event = sse_evt[0][63:0]
+	 * events[1].event = sse_evt[0][127:64]
+	 * events[2].event = sse_evt[1][63:0]
+	 * events[3].event = sse_evt[1][127:64]
+	 */
+	_mm_storel_epi64((__m128i *)&events[0].event, sse_evt[0]);
+	_mm_storeh_pd((double *)&events[1].event, (__m128d) sse_evt[0]);
+	_mm_storel_epi64((__m128i *)&events[2].event, sse_evt[1]);
+	_mm_storeh_pd((double *)&events[3].event, (__m128d) sse_evt[1]);
+
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[0].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[1].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[2].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[3].sched_type], 1);
+
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[0].queue_id].
+			qid_depth[qes[0].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[1].queue_id].
+			qid_depth[qes[1].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[2].queue_id].
+			qid_depth[qes[2].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[3].queue_id].
+			qid_depth[qes[3].qid_depth],
+		1);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num_events);
+
+	return num_events;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe_sparse(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe)
+{
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	uint8_t xor_mask[2] = {0x0F, 0x00};
+	const uint8_t and_mask = 0x0F;
+	__m128i *qes = (__m128i *)qe;
+	uint8_t gen_bits, gen_bit;
+	uintptr_t addr[4];
+	uint16_t idx;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+
+	idx = qm_port->cq_idx;
+
+	/* Load the next 4 QEs */
+	addr[0] = (uintptr_t)&cq_addr[idx];
+	addr[1] = (uintptr_t)&cq_addr[(idx +  4) & qm_port->cq_depth_mask];
+	addr[2] = (uintptr_t)&cq_addr[(idx +  8) & qm_port->cq_depth_mask];
+	addr[3] = (uintptr_t)&cq_addr[(idx + 12) & qm_port->cq_depth_mask];
+
+	/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */
+	rte_prefetch0(&cq_addr[(idx + 16) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 20) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 24) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 28) & qm_port->cq_depth_mask]);
+
+	/* Correct the xor_mask for wrap-around QEs */
+	gen_bit = qm_port->gen_bit;
+	xor_mask[gen_bit] ^= !!((idx +  4) > qm_port->cq_depth_mask) << 1;
+	xor_mask[gen_bit] ^= !!((idx +  8) > qm_port->cq_depth_mask) << 2;
+	xor_mask[gen_bit] ^= !!((idx + 12) > qm_port->cq_depth_mask) << 3;
+
+	/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is
+	 * valid, then QEs[0:N-1] are too.
+	 */
+	qes[3] = _mm_load_si128((__m128i *)(void *)addr[3]);
+	rte_compiler_barrier();
+	qes[2] = _mm_load_si128((__m128i *)(void *)addr[2]);
+	rte_compiler_barrier();
+	qes[1] = _mm_load_si128((__m128i *)(void *)addr[1]);
+	rte_compiler_barrier();
+	qes[0] = _mm_load_si128((__m128i *)(void *)addr[0]);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[gen_bit];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask;
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline void
+dlb2_inc_cq_idx(struct dlb2_port *qm_port, int cnt)
+{
+	uint16_t idx = qm_port->cq_idx_unmasked + cnt;
+
+	qm_port->cq_idx_unmasked = idx;
+	qm_port->cq_idx = idx & qm_port->cq_depth_mask;
+	qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) & 0x1;
+}
+
+static int
+dlb2_event_release(struct dlb2_eventdev *dlb2,
+		   uint8_t port_id,
+		   int n)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_eventdev_port *ev_port;
+	struct dlb2_port *qm_port;
+	int i, cnt;
+
+	if (port_id > dlb2->num_ports) {
+		DLB2_LOG_ERR("Invalid port id %d in dlb2-event_release\n",
+			     port_id);
+		rte_errno = -EINVAL;
+		return rte_errno;
+	}
+
+	ev_port = &dlb2->ev_ports[port_id];
+	qm_port = &ev_port->qm_port;
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	cnt = 0;
+
+	if (qm_port->is_directed) {
+		cnt = n;
+		goto sw_credit_update;
+	}
+
+	for (i = 0; i < n; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		int j;
+
+		/* Zero-out QEs */
+		qm_port->qe4[0].cmd_byte = 0;
+		qm_port->qe4[1].cmd_byte = 0;
+		qm_port->qe4[2].cmd_byte = 0;
+		qm_port->qe4[3].cmd_byte = 0;
+
+		for (j = 0; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++)
+			qm_port->qe4[j].cmd_byte = DLB2_COMP_CMD_BYTE;
+
+		qm_port->issued_releases += j;
+
+		if (j == 0)
+			break;
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+	}
+
+sw_credit_update:
+	/* each release returns one credit */
+	if (!ev_port->outstanding_releases) {
+		DLB2_LOG_ERR("Unrecoverable application error. Outstanding releases underflowed.\n");
+		rte_errno = -ENOTRECOVERABLE;
+		return rte_errno;
+	}
+
+	ev_port->outstanding_releases -= cnt;
+	ev_port->inflight_credits += cnt;
+
+	/* Replenish s/w credits if enough releases are performed */
+	dlb2_replenish_sw_credits(dlb2, ev_port);
+	return 0;
+}
+
+static inline int16_t
+dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
+		       struct dlb2_eventdev_port *ev_port,
+		       struct rte_event *events,
+		       uint16_t max_num,
+		       uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe_sparse(qm_port, qes);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail << 2);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							      qm_port,
+							      &events[num],
+							      &qes[0]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							 qm_port,
+							 &events[num],
+							 &qes[0],
+							 num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe,
+	     uint8_t *offset)
+{
+	uint8_t xor_mask[2][4] = { {0x0F, 0x0E, 0x0C, 0x08},
+				   {0x00, 0x01, 0x03, 0x07} };
+	uint8_t and_mask[4] = {0x0F, 0x0E, 0x0C, 0x08};
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	__m128i *qes = (__m128i *)qe;
+	uint64_t *cache_line_base;
+	uint8_t gen_bits;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	cq_addr = &cq_addr[qm_port->cq_idx];
+
+	cache_line_base = (void *)(((uintptr_t)cq_addr) & ~0x3F);
+	*offset = ((uintptr_t)cq_addr & 0x30) >> 4;
+
+	/* Load the next CQ cache line from memory. Pack these reads as tight
+	 * as possible to reduce the chance that DLB invalidates the line while
+	 * the CPU is reading it. Read the cache line backwards to ensure that
+	 * if QE[N] (N > 0) is valid, then QEs[0:N-1] are too.
+	 *
+	 * (Valid QEs start at &qe[offset])
+	 */
+	qes[3] = _mm_load_si128((__m128i *)&cache_line_base[6]);
+	qes[2] = _mm_load_si128((__m128i *)&cache_line_base[4]);
+	qes[1] = _mm_load_si128((__m128i *)&cache_line_base[2]);
+	qes[0] = _mm_load_si128((__m128i *)&cache_line_base[0]);
+
+	/* Evict the cache line ASAP */
+	dlb2_cldemote(cache_line_base);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[qm_port->gen_bit][*offset];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask[*offset];
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline int16_t
+dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
+		struct dlb2_eventdev_port *ev_port,
+		struct rte_event *events,
+		uint16_t max_num,
+		uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t offset;
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe(qm_port, qes, &offset);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							     qm_port,
+							     &events[num],
+							     &qes[offset]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							qm_port,
+							&events[num],
+							&qes[offset],
+							num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static uint16_t
+dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
+			 uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue(void *event_port, struct rte_event *ev, uint64_t wait)
+{
+	return dlb2_event_dequeue_burst(event_port, ev, 1, wait);
+}
+
+static uint16_t
+dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
+				uint16_t num, uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
+			  uint64_t wait)
+{
+	return dlb2_event_dequeue_burst_sparse(event_port, ev, 1, wait);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
+	struct dlb2_eventdev *dlb2;
+
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
@@ -2691,6 +3439,15 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	dev->enqueue_burst = dlb2_event_enqueue_burst;
 	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
 	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
+
+	dlb2 = dev->data->dev_private;
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE) {
+		dev->dequeue = dlb2_event_dequeue_sparse;
+		dev->dequeue_burst = dlb2_event_dequeue_burst_sparse;
+	} else {
+		dev->dequeue = dlb2_event_dequeue;
+		dev->dequeue_burst = dlb2_event_dequeue_burst;
+	}
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v8 19/23] event/dlb2: add eventdev stop and close

[Timothy McDaniel]

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 257 +++++++++++++++++++++++++++--
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c |  91 ++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  52 ++++++
 5 files changed, 396 insertions(+), 16 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index b9082bc..82108b6 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -71,21 +71,6 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
-/*
- * DUMMY - added so that xstats path will compile/link.
- * Will be replaced by real version in a subsequent
- * patch.
- */
-uint32_t
-dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
-		     struct dlb2_eventdev_queue *queue)
-{
-	RTE_SET_USED(dlb2);
-	RTE_SET_USED(queue);
-
-	return 0;
-}
-
 static void
 dlb2_free_qe_mem(struct dlb2_port *qm_port)
 {
@@ -1878,7 +1863,6 @@ dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
 		return 0; /* Ignore and return success */
 	}
 
-	/* FIXME: How to handle unlink on a directed port? */
 	if (ev_port->qm_port.is_directed) {
 		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
 			     ev_port->id);
@@ -3408,6 +3392,245 @@ dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
 }
 
 static void
+dlb2_flush_port(struct rte_eventdev *dev, int port_id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	eventdev_stop_flush_t flush;
+	struct rte_event ev;
+	uint8_t dev_id;
+	void *arg;
+	int i;
+
+	flush = dev->dev_ops->dev_stop_flush;
+	dev_id = dev->data->dev_id;
+	arg = dev->data->dev_stop_flush_arg;
+
+	while (rte_event_dequeue_burst(dev_id, port_id, &ev, 1, 0)) {
+		if (flush)
+			flush(dev_id, ev, arg);
+
+		if (dlb2->ev_ports[port_id].qm_port.is_directed)
+			continue;
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+	}
+
+	/* Enqueue any additional outstanding releases */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = dlb2->ev_ports[port_id].outstanding_releases; i > 0; i--)
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+}
+
+static uint32_t
+dlb2_get_ldb_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_ldb_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_ldb_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_ldb_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static uint32_t
+dlb2_get_dir_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_dir_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_dir_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_dir_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	if (queue->qm_queue.is_directed)
+		return dlb2_get_dir_queue_depth(dlb2, queue);
+	else
+		return dlb2_get_ldb_queue_depth(dlb2, queue);
+}
+
+static bool
+dlb2_queue_is_empty(struct dlb2_eventdev *dlb2,
+		    struct dlb2_eventdev_queue *queue)
+{
+	return dlb2_get_queue_depth(dlb2, queue) == 0;
+}
+
+static bool
+dlb2_linked_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0)
+			continue;
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static bool
+dlb2_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static void
+dlb2_drain(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_port *ev_port = NULL;
+	uint8_t dev_id;
+	int i;
+
+	dev_id = dev->data->dev_id;
+
+	while (!dlb2_linked_queues_empty(dlb2)) {
+		/* Flush all the ev_ports, which will drain all their connected
+		 * queues.
+		 */
+		for (i = 0; i < dlb2->num_ports; i++)
+			dlb2_flush_port(dev, i);
+	}
+
+	/* The queues are empty, but there may be events left in the ports. */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_flush_port(dev, i);
+
+	/* If the domain's queues are empty, we're done. */
+	if (dlb2_queues_empty(dlb2))
+		return;
+
+	/* Else, there must be at least one unlinked load-balanced queue.
+	 * Select a load-balanced port with which to drain the unlinked
+	 * queue(s).
+	 */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		if (!ev_port->qm_port.is_directed)
+			break;
+	}
+
+	if (i == dlb2->num_ports) {
+		DLB2_LOG_ERR("internal error: no LDB ev_ports\n");
+		return;
+	}
+
+	rte_errno = 0;
+	rte_event_port_unlink(dev_id, ev_port->id, NULL, 0);
+
+	if (rte_errno) {
+		DLB2_LOG_ERR("internal error: failed to unlink ev_port %d\n",
+			     ev_port->id);
+		return;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		uint8_t qid, prio;
+		int ret;
+
+		if (dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			continue;
+
+		qid = i;
+		prio = 0;
+
+		/* Link the ev_port to the queue */
+		ret = rte_event_port_link(dev_id, ev_port->id, &qid, &prio, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to link ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+
+		/* Flush the queue */
+		while (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			dlb2_flush_port(dev, ev_port->id);
+
+		/* Drain any extant events in the ev_port. */
+		dlb2_flush_port(dev, ev_port->id);
+
+		/* Unlink the ev_port from the queue */
+		ret = rte_event_port_unlink(dev_id, ev_port->id, &qid, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to unlink ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+	}
+}
+
+static void
+dlb2_eventdev_stop(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_DBG("Internal error: already stopped\n");
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	} else if (dlb2->run_state != DLB2_RUN_STATE_STARTED) {
+		DLB2_LOG_ERR("Internal error: bad state %d for dev_stop\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPING;
+
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	dlb2_drain(dev);
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPED;
+}
+
+static int
+dlb2_eventdev_close(struct rte_eventdev *dev)
+{
+	dlb2_hw_reset_sched_domain(dev, false);
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3417,6 +3640,8 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.dev_start        = dlb2_eventdev_start,
+		.dev_stop         = dlb2_eventdev_stop,
+		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a86191d..5471dd8 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -66,3 +66,9 @@ int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				     struct dlb2_start_domain_args *cfg);
+
+int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 7ead374..b508eb0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -65,4 +65,10 @@ extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				struct dlb2_start_domain_args *cfg);
+
+extern int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+extern int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index c14a2f3..ae5ef2f 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5934,3 +5934,94 @@ dlb2_hw_start_domain(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_dir_queue_depth(hw, queue);
+
+	return 0;
+}
+
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+
+	dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index fff2e57..632c4e0 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -527,6 +527,56 @@ dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_ldb_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_ldb_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_dir_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret = 0;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_dir_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -544,6 +594,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth;
+	dlb2_iface_get_dir_queue_depth = dlb2_pf_get_dir_queue_depth;
 	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v8 20/23] event/dlb2: add PMD's token pop public interface

[Timothy McDaniel]

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/api/doxy-api-index.md         |  1 +
 drivers/event/dlb2/dlb2.c         | 53 ++++++++++++++++++++++++----
 drivers/event/dlb2/dlb2_priv.h    |  3 ++
 drivers/event/dlb2/meson.build    |  5 ++-
 drivers/event/dlb2/rte_pmd_dlb2.c | 39 +++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2.h | 72 +++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/version.map    |  6 ++++
 7 files changed, 172 insertions(+), 7 deletions(-)
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index a9c12d1..00f13e2 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -52,6 +52,7 @@ The public API headers are grouped by topics:
   [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
   [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
+  [dlb2]	       (@ref rte_pmd_dlb2.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 82108b6..7a22231 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1197,7 +1197,7 @@ dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
 	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
 
 	qm_port->dequeue_depth = dequeue_depth;
-
+	qm_port->token_pop_thresh = dequeue_depth;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -1365,6 +1365,8 @@ dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
 
 	qm_port->dequeue_depth = dequeue_depth;
 
+	/* Directed ports are auto-pop, by default. */
+	qm_port->token_pop_mode = AUTO_POP;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -2612,6 +2614,14 @@ dlb2_event_enqueue_burst(void *event_port,
 		dlb2_event_build_hcws(qm_port, &events[i], j,
 				      sched_types, queue_ids);
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
@@ -2620,6 +2630,11 @@ dlb2_event_enqueue_burst(void *event_port,
 			break;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
 	return cnt;
 }
 
@@ -3097,11 +3112,25 @@ dlb2_event_release(struct dlb2_eventdev *dlb2,
 		if (j == 0)
 			break;
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
 sw_credit_update:
 	/* each release returns one credit */
 	if (!ev_port->outstanding_releases) {
@@ -3185,8 +3214,8 @@ dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3312,8 +3341,8 @@ dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3328,6 +3357,7 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 			 uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3343,6 +3373,9 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3362,6 +3395,7 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 				uint16_t num, uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3377,6 +3411,9 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3681,7 +3718,7 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    struct dlb2_devargs *dlb2_args)
 {
 	struct dlb2_eventdev *dlb2;
-	int err;
+	int err, i;
 
 	dlb2 = dev->data->dev_private;
 
@@ -3731,6 +3768,10 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Initialize each port's token pop mode */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
+		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 61567a6..b73cf3f 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -12,6 +12,7 @@
 #include <rte_config.h>
 #include "dlb2_user.h"
 #include "dlb2_log.h"
+#include "rte_pmd_dlb2.h"
 
 #ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
 #define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
@@ -290,6 +291,7 @@ struct dlb2_port {
 	bool gen_bit;
 	uint16_t dir_credits;
 	uint32_t dequeue_depth;
+	enum dlb2_token_pop_mode token_pop_mode;
 	union dlb2_port_config cfg;
 	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
 	uint16_t cached_ldb_credits;
@@ -298,6 +300,7 @@ struct dlb2_port {
 	bool int_armed;
 	uint16_t owed_tokens;
 	int16_t issued_releases;
+	int16_t token_pop_thresh;
 	int cq_depth;
 	uint16_t cq_idx;
 	uint16_t cq_idx_unmasked;
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index dc16c96..b734ff6 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,3 +1,4 @@
+
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
@@ -12,7 +13,9 @@ sources = files('dlb2.c',
 		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
-		'pf/base/dlb2_resource.c'
+		'pf/base/dlb2_resource.c',
+		'rte_pmd_dlb2.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
+install_headers('rte_pmd_dlb2.h')
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
new file mode 100644
index 0000000..b09b585
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "rte_pmd_dlb2.h"
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode)
+{
+	struct dlb2_eventdev *dlb2;
+	struct rte_eventdev *dev;
+
+	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+	dev = &rte_eventdevs[dev_id];
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (mode >= NUM_TOKEN_POP_MODES)
+		return -EINVAL;
+
+	/* The event device must be configured, but not yet started */
+	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
+		return -EINVAL;
+
+	/* The token pop mode must be set before configuring the port */
+	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
+		return -EINVAL;
+
+	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
new file mode 100644
index 0000000..74399db
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+/*!
+ *  @file      rte_pmd_dlb2.h
+ *
+ *  @brief     DLB PMD-specific functions
+ */
+
+#ifndef _RTE_PMD_DLB2_H_
+#define _RTE_PMD_DLB2_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Selects the token pop mode for a DLB2 port.
+ */
+enum dlb2_token_pop_mode {
+	/* Pop the CQ tokens immediately after dequeueing. */
+	AUTO_POP,
+	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
+	 * Supported on load-balanced ports only.
+	 */
+	DELAYED_POP,
+	/* Pop the CQ tokens during next dequeue operation. */
+	DEFERRED_POP,
+
+	/* NUM_TOKEN_POP_MODES must be last */
+	NUM_TOKEN_POP_MODES
+};
+
+/*!
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Configure the token pop mode for a DLB2 port. By default, all ports use
+ * AUTO_POP. This function must be called before calling rte_event_port_setup()
+ * for the port, but after calling rte_event_dev_configure().
+ *
+ * @param dev_id
+ *    The identifier of the event device.
+ * @param port_id
+ *    The identifier of the event port.
+ * @param mode
+ *    The token pop mode.
+ *
+ * @return
+ * - 0: Success
+ * - EINVAL: Invalid dev_id, port_id, or mode
+ * - EINVAL: The DLB2 is not configured, is already running, or the port is
+ *   already setup
+ */
+
+__rte_experimental
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_DLB2_H_ */
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
index 4a76d1d..b1e4dff 100644
--- a/drivers/event/dlb2/version.map
+++ b/drivers/event/dlb2/version.map
@@ -1,3 +1,9 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmd_dlb2_set_token_pop_mode;
+};
-- 
2.6.4

Re: [dpdk-dev] [PATCH v8 20/23] event/dlb2: add PMD's token pop public interface

[David Marchand]

On Sat, Oct 31, 2020 at 3:07 AM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
> diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
> index a9c12d1..00f13e2 100644
> --- a/doc/api/doxy-api-index.md
> +++ b/doc/api/doxy-api-index.md
> @@ -52,6 +52,7 @@ The public API headers are grouped by topics:
>    [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
>    [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
>    [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
> +  [dlb2]              (@ref rte_pmd_dlb2.h)

This comment applies to the dlb series too:

Those lines should be aligned with spaces, not tabs.
And the crypto_scheduler line is missing a comma.


>
>  - **memory**:
>    [memseg]             (@ref rte_memory.h),


--
David Marchand

Re: [dpdk-dev] [PATCH v8 20/23] event/dlb2: add PMD's token pop public interface

[McDaniel, Timothy]

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Saturday, October 31, 2020 5:52 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: Ray Kinsella <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>;
> dev <dev@dpdk.org>; Carrillo, Erik G <Erik.G.Carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Thomas Monjalon
> <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH v8 20/23] event/dlb2: add PMD's token pop
> public interface
> 
> On Sat, Oct 31, 2020 at 3:07 AM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> > diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
> > index a9c12d1..00f13e2 100644
> > --- a/doc/api/doxy-api-index.md
> > +++ b/doc/api/doxy-api-index.md
> > @@ -52,6 +52,7 @@ The public API headers are grouped by topics:
> >    [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
> >    [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
> >    [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
> > +  [dlb2]              (@ref rte_pmd_dlb2.h)
> 
> This comment applies to the dlb series too:
> 
> Those lines should be aligned with spaces, not tabs.
> And the crypto_scheduler line is missing a comma.
> 
> 
> >
> >  - **memory**:
> >    [memseg]             (@ref rte_memory.h),
> 
> 
> --
> David Marchand

Fixed in local repos for both DLB and DLB2. Will be included in next rev of patch-sets.

Tim

Re: [dpdk-dev] [PATCH v8 20/23] event/dlb2: add PMD's token pop public interface

[McDaniel, Timothy]

> -----Original Message-----
> From: McDaniel, Timothy
> Sent: Saturday, October 31, 2020 11:38 AM
> To: David Marchand <david.marchand@redhat.com>
> Cc: Ray Kinsella <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>;
> dev <dev@dpdk.org>; Carrillo, Erik G <Erik.G.Carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Thomas Monjalon
> <thomas@monjalon.net>
> Subject: RE: [dpdk-dev] [PATCH v8 20/23] event/dlb2: add PMD's token pop
> public interface
> 
> 
> 
> > -----Original Message-----
> > From: David Marchand <david.marchand@redhat.com>
> > Sent: Saturday, October 31, 2020 5:52 AM
> > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > Cc: Ray Kinsella <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>;
> > dev <dev@dpdk.org>; Carrillo, Erik G <Erik.G.Carrillo@intel.com>; Eads, Gage
> > <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> > Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Thomas Monjalon
> > <thomas@monjalon.net>
> > Subject: Re: [dpdk-dev] [PATCH v8 20/23] event/dlb2: add PMD's token pop
> > public interface
> >
> > On Sat, Oct 31, 2020 at 3:07 AM Timothy McDaniel
> > <timothy.mcdaniel@intel.com> wrote:
> > > diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
> > > index a9c12d1..00f13e2 100644
> > > --- a/doc/api/doxy-api-index.md
> > > +++ b/doc/api/doxy-api-index.md
> > > @@ -52,6 +52,7 @@ The public API headers are grouped by topics:
> > >    [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
> > >    [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
> > >    [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
> > > +  [dlb2]              (@ref rte_pmd_dlb2.h)
> >
> > This comment applies to the dlb series too:
> >
> > Those lines should be aligned with spaces, not tabs.
> > And the crypto_scheduler line is missing a comma.
> >
> >
> > >
> > >  - **memory**:
> > >    [memseg]             (@ref rte_memory.h),
> >
> >
> > --
> > David Marchand
> 
> Fixed in local repos for both DLB and DLB2. Will be included in next rev of patch-
> sets.
> 
> Tim

I have made the requested changes to doc/api/doxy-api-index.md in DLB and DLB2 repos, but I still see this error:
/home/travis/build/ovsrobot/dpdk/doc/api/doxy-api-index.md:54: error: unable to resolve reference to `rte_pmd_dlb.h' for \ref command (warning treated as error, aborting

I was hoping that the formatting issues were the culprit, but apparently not.  Any thoughts?

Thanks,
Tim

Re: [dpdk-dev] [PATCH v8 20/23] event/dlb2: add PMD's token pop public interface

[David Marchand]

On Sat, Oct 31, 2020 at 8:19 PM McDaniel, Timothy
<timothy.mcdaniel@intel.com> wrote:
> I have made the requested changes to doc/api/doxy-api-index.md in DLB and DLB2 repos, but I still see this error:
> /home/travis/build/ovsrobot/dpdk/doc/api/doxy-api-index.md:54: error: unable to resolve reference to `rte_pmd_dlb.h' for \ref command (warning treated as error, aborting
>
> I was hoping that the formatting issues were the culprit, but apparently not.  Any thoughts?

You must instruct where to find this header via doxy-api.conf.in

Applied following diff, it works:
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 2ff3f74413..1400fde9c0 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -52,7 +52,7 @@ The public API headers are grouped by topics:
   [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
   [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h),
-  [dlb]                (@ref rte_pmd_dlb.h),
+  [dlb]                (@ref rte_pmd_dlb.h)

 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index e37f8c2e80..176e7d7857 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -7,6 +7,7 @@ USE_MDFILE_AS_MAINPAGE  = @TOPDIR@/doc/api/doxy-api-index.md
 INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/drivers/bus/vdev \
                           @TOPDIR@/drivers/crypto/scheduler \
+                          @TOPDIR@/drivers/event/dlb \
                           @TOPDIR@/drivers/mempool/dpaa2 \
                           @TOPDIR@/drivers/net/bnxt \
                           @TOPDIR@/drivers/net/bonding \


I spotted more issues, I'll reply on the patches.



-- 
David Marchand

Re: [dpdk-dev] [PATCH v8 20/23] event/dlb2: add PMD's token pop public interface

[McDaniel, Timothy]

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Saturday, October 31, 2020 4:39 PM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: Ray Kinsella <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>;
> dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Thomas Monjalon
> <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH v8 20/23] event/dlb2: add PMD's token pop
> public interface
> 
> On Sat, Oct 31, 2020 at 8:19 PM McDaniel, Timothy
> <timothy.mcdaniel@intel.com> wrote:
> > I have made the requested changes to doc/api/doxy-api-index.md in DLB and
> DLB2 repos, but I still see this error:
> > /home/travis/build/ovsrobot/dpdk/doc/api/doxy-api-index.md:54: error:
> unable to resolve reference to `rte_pmd_dlb.h' for \ref command (warning
> treated as error, aborting
> >
> > I was hoping that the formatting issues were the culprit, but apparently not.
> Any thoughts?
> 
> You must instruct where to find this header via doxy-api.conf.in
> 
> Applied following diff, it works:
> diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
> index 2ff3f74413..1400fde9c0 100644
> --- a/doc/api/doxy-api-index.md
> +++ b/doc/api/doxy-api-index.md
> @@ -52,7 +52,7 @@ The public API headers are grouped by topics:
>    [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
>    [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
>    [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h),
> -  [dlb]                (@ref rte_pmd_dlb.h),
> +  [dlb]                (@ref rte_pmd_dlb.h)
> 
>  - **memory**:
>    [memseg]             (@ref rte_memory.h),
> diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
> index e37f8c2e80..176e7d7857 100644
> --- a/doc/api/doxy-api.conf.in
> +++ b/doc/api/doxy-api.conf.in
> @@ -7,6 +7,7 @@ USE_MDFILE_AS_MAINPAGE  = @TOPDIR@/doc/api/doxy-
> api-index.md
>  INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
>                            @TOPDIR@/drivers/bus/vdev \
>                            @TOPDIR@/drivers/crypto/scheduler \
> +                          @TOPDIR@/drivers/event/dlb \
>                            @TOPDIR@/drivers/mempool/dpaa2 \
>                            @TOPDIR@/drivers/net/bnxt \
>                            @TOPDIR@/drivers/net/bonding \
> 
> 
> I spotted more issues, I'll reply on the patches.
> 
> 
> 
> --
> David Marchand

Thank you, David. Much appreciated.

[dpdk-dev] [PATCH v8 21/23] event/dlb2: add PMD self-tests

[Timothy McDaniel]

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 app/test/test_eventdev.c           |    7 +
 drivers/event/dlb2/dlb2.c          |    1 +
 drivers/event/dlb2/dlb2_selftest.c | 1524 ++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build     |    3 +-
 4 files changed, 1534 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c

diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
index 62019c1..bcfaa53 100644
--- a/app/test/test_eventdev.c
+++ b/app/test/test_eventdev.c
@@ -1030,6 +1030,12 @@ test_eventdev_selftest_dpaa2(void)
 	return test_eventdev_selftest_impl("event_dpaa2", "");
 }
 
+static int
+test_eventdev_selftest_dlb2(void)
+{
+	return test_eventdev_selftest_impl("dlb2_event", "");
+}
+
 REGISTER_TEST_COMMAND(eventdev_common_autotest, test_eventdev_common);
 REGISTER_TEST_COMMAND(eventdev_selftest_sw, test_eventdev_selftest_sw);
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
@@ -1037,3 +1043,4 @@ REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
 		test_eventdev_selftest_octeontx2);
 REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2, test_eventdev_selftest_dpaa2);
+REGISTER_TEST_COMMAND(eventdev_selftest_dlb2, test_eventdev_selftest_dlb2);
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7a22231..99a22e6 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3692,6 +3692,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
 		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+		.dev_selftest     = test_dlb2_eventdev,
 	};
 
 	/* Expose PMD's eventdev interface */
diff --git a/drivers/event/dlb2/dlb2_selftest.c b/drivers/event/dlb2/dlb2_selftest.c
new file mode 100644
index 0000000..e300067
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_selftest.c
@@ -0,0 +1,1524 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <unistd.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_eventdev.h>
+#include <rte_pause.h>
+
+#include "dlb2_priv.h"
+#include "rte_pmd_dlb2.h"
+
+#define MAX_PORTS 32
+#define MAX_QIDS 32
+#define DEFAULT_NUM_SEQ_NUMS 64
+
+static struct rte_mempool *eventdev_func_mempool;
+static int evdev;
+
+struct test {
+	struct rte_mempool *mbuf_pool;
+	int nb_qids;
+};
+
+/* initialization and config */
+static inline int
+init(struct test *t, int nb_queues, int nb_ports)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	memset(t, 0, sizeof(*t));
+
+	t->mbuf_pool = eventdev_func_mempool;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = nb_queues;
+	config.nb_event_ports = nb_ports;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0)
+		printf("%d: Error configuring device\n", __LINE__);
+
+	return ret;
+}
+
+static inline int
+create_ports(int num_ports)
+{
+	int i;
+
+	if (num_ports > MAX_PORTS)
+		return -1;
+
+	for (i = 0; i < num_ports; i++) {
+		struct rte_event_port_conf conf;
+
+		if (rte_event_port_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default port conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		if (rte_event_port_setup(evdev, i, &conf) < 0) {
+			printf("%d: Error setting up port %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+create_lb_qids(struct test *t, int num_qids, uint32_t flags)
+{
+	int i;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		struct rte_event_queue_conf conf;
+
+		if (rte_event_queue_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default queue conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		conf.schedule_type = flags;
+
+		if (conf.schedule_type == RTE_SCHED_TYPE_PARALLEL)
+			conf.nb_atomic_order_sequences = 0;
+		else
+			conf.nb_atomic_order_sequences = DEFAULT_NUM_SEQ_NUMS;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+static inline int
+create_atomic_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ATOMIC);
+}
+
+/* destruction */
+static inline void
+cleanup(void)
+{
+	int ret = 0;
+
+	rte_event_dev_stop(evdev);
+	ret = rte_event_dev_close(evdev);
+
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+};
+
+static inline int
+enqueue_timeout(uint8_t port_id, struct rte_event *ev, uint64_t tmo_us)
+{
+	const uint64_t start = rte_get_timer_cycles();
+	const uint64_t ticks = (tmo_us * rte_get_timer_hz()) / 1E6;
+
+	while ((rte_get_timer_cycles() - start) < ticks) {
+		if (rte_event_enqueue_burst(evdev, port_id, ev, 1) == 1)
+			return 0;
+
+		if (rte_errno != -ENOSPC) {
+			printf("enqueue_burst returned rte_errno %d\n",
+			       rte_errno);
+			return -1;
+		}
+	}
+	printf("%s time out\n", __func__);
+	return -1;
+}
+
+static void
+flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused)
+{
+	rte_pktmbuf_free(event.mbuf);
+}
+
+static int
+test_stop_flush(struct test *t) /* test to check we can properly flush events */
+{
+	struct rte_event ev;
+	uint32_t dequeue_depth;
+	unsigned int i, count;
+	uint8_t queue_id;
+
+	ev.op = RTE_EVENT_OP_NEW;
+
+	if (init(t, 2, 1) < 0 ||
+	    create_ports(1) < 0 ||
+	    create_atomic_qids(t, 2) < 0) {
+		printf("%d: Error initializing device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_port_link(evdev, 0, NULL, NULL, 0) != 2) {
+		printf("%d: Error linking queues to the port\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: Error with start call\n", __LINE__);
+		goto err;
+	}
+
+	/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked
+	 * queue.
+	 */
+	queue_id = 1;
+
+	if (rte_event_port_unlink(evdev, 0, &queue_id, 1) != 1) {
+		printf("%d: Error unlinking queue 1 from port\n", __LINE__);
+		goto err;
+	}
+
+	count = rte_mempool_avail_count(t->mbuf_pool);
+
+	if (rte_event_port_attr_get(evdev,
+				    0,
+				    RTE_EVENT_PORT_ATTR_DEQ_DEPTH,
+				    &dequeue_depth)) {
+		printf("%d: Error retrieveing dequeue depth\n", __LINE__);
+		goto err;
+	}
+
+	/* Send QEs to queue 0 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 0;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Send QEs to queue 1 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 1;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Now the DLB is scheduling events from the port to the IQ, and at
+	 * least one event should be remaining in each queue.
+	 */
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, flush, NULL)) {
+		printf("%d: Error installing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+
+	if (count != rte_mempool_avail_count(t->mbuf_pool)) {
+		printf("%d: Error executing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, NULL, NULL)) {
+		printf("%d: Error uninstalling the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	return 0;
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_single_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Create a directed port */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) == 0) {
+		printf("%d: port 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Create a load-balanced port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a directed queue */
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed queue */
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) == 0) {
+		printf("%d: queue 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a load-balanced queue */
+	queue_conf.event_queue_cfg = 0;
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to link directed and load-balanced resources */
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 0 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 1 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Link ports to queues */
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+#define NUM_LDB_PORTS 64
+#define NUM_LDB_QUEUES 32
+
+static int
+test_info_get(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	config.nb_event_ports = info.max_event_ports;
+	config.nb_event_queues = NUM_LDB_QUEUES + info.max_event_ports / 2;
+	config.nb_single_link_event_port_queues = info.max_event_ports / 2;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		goto err;
+	}
+
+	/* The DLB2 PMD only reports load-balanced ports and queues in its
+	 * info_get function. Confirm that these values don't include the
+	 * directed port or queue counts.
+	 */
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret) {
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+		return -1;
+	}
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+static int
+test_reconfiguration_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 2 LDB queues */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+			printf("%d: P0->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P1->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+			printf("%d: P1->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Configure device with 2 DIR ports and 2 DIR queues */
+	config.nb_single_link_event_port_queues = 2;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q1 */
+	queue_id = 1;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: P0->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Link P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: P1->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_queues = 5;
+	config.nb_event_ports = 5;
+	config.nb_single_link_event_port_queues = 1;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < config.nb_event_queues - 1; i++) {
+		port_conf.event_port_cfg = 0;
+		queue_conf.event_queue_cfg = 0;
+
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: P%d->Q%d link expected to succeed\n",
+			       __LINE__, i, i);
+			goto err;
+		}
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+		printf("%d: port %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+		printf("%d: queue %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	queue_id = i;
+
+	if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+		printf("%d: P%d->Q%d link expected to succeed\n",
+		       __LINE__, i, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_ports += 1;
+
+	/* Reconfigure device with 1 more load-balanced port */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Configure the new port */
+	if (rte_event_port_setup(evdev, config.nb_event_ports - 1,
+				 &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_load_balanced_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_directed_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 DIR port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (ev.queue_id != 0) {
+		printf("%d: invalid dequeued event queue ID (%d)\n",
+		       __LINE__, ev.queue_id);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_deferred_sched(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	const int num_events = 128;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 1 queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 1, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.dequeue_depth = 1;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.schedule_type = RTE_SCHED_TYPE_PARALLEL;
+	queue_conf.nb_atomic_order_sequences = 0;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 128 NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_PARALLEL;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue one event from port 0 */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue (and release) all other events from port 1. Deferred
+	 * scheduling ensures no other events are scheduled to port 0 without a
+	 * subsequent rte_event_dequeue_burst() call.
+	 */
+	for (i = 0; i < num_events - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 1, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		if (rte_event_enqueue_burst(evdev, 1, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_delayed_pop(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	int ret, i, num_events;
+	struct rte_event ev;
+	uint8_t queue_id;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DELAYED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	num_events = 2 * port_conf.dequeue_depth;
+
+	/* Enqueue 2 * dequeue_depth NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue dequeue_depth events but only release dequeue_depth - 2.
+	 * Delayed pop won't perform the pop and no more events will be
+	 * scheduled.
+	 */
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = 0; i < port_conf.dequeue_depth - 2; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	ret = rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout);
+	if (ret != 0) {
+		printf("%d: event dequeue expected to fail (ret = %d)\n",
+		       __LINE__, ret);
+		goto err;
+	}
+
+	/* Release one more event. This will trigger the token pop, and
+	 * dequeue_depth - 1 more events will be scheduled to the device.
+	 */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: RELEASE enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 0) {
+		printf("%d: event dequeue expected to fail\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+do_selftest(void)
+{
+	struct test t;
+	int ret;
+
+	/* Only create mbuf pool once, reuse for each test run */
+	if (!eventdev_func_mempool) {
+		eventdev_func_mempool =
+			rte_pktmbuf_pool_create("EVENTDEV_DLB2_ST_POOL",
+						(1 << 12), /* 4k buffers */
+						32 /*MBUF_CACHE_SIZE*/,
+						0,
+						512, /* use very small mbufs */
+						rte_socket_id());
+		if (!eventdev_func_mempool) {
+			printf("ERROR creating mempool\n");
+			goto test_fail;
+		}
+	}
+	t.mbuf_pool = eventdev_func_mempool;
+
+	printf("*** Running Stop Flush test...\n");
+	ret = test_stop_flush(&t);
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Single Link test...\n");
+	ret = test_single_link();
+	if (ret != 0) {
+		printf("ERROR - Single Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Info Get test...\n");
+	ret = test_info_get();
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Reconfiguration Link test...\n");
+	ret = test_reconfiguration_link();
+	if (ret != 0) {
+		printf("ERROR - Reconfiguration Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Load-Balanced Traffic test...\n");
+	ret = test_load_balanced_traffic();
+	if (ret != 0) {
+		printf("ERROR - Load-Balanced Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Directed Traffic test...\n");
+	ret = test_directed_traffic();
+	if (ret != 0) {
+		printf("ERROR - Directed Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Deferred Scheduling test...\n");
+	ret = test_deferred_sched();
+	if (ret != 0) {
+		printf("ERROR - Deferred Scheduling test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Delayed Pop test...\n");
+	ret = test_delayed_pop();
+	if (ret != 0) {
+		printf("ERROR - Delayed Pop test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	return 0;
+
+test_fail:
+	return -1;
+}
+
+int
+test_dlb2_eventdev(void)
+{
+	const char *dlb2_eventdev_name = "dlb2_event";
+	uint8_t num_evdevs = rte_event_dev_count();
+	int i, ret = 0;
+	int found = 0, skipped = 0, passed = 0, failed = 0;
+	struct rte_event_dev_info info;
+
+	for (i = 0; found + skipped < num_evdevs && i < RTE_EVENT_MAX_DEVS;
+	     i++) {
+		ret = rte_event_dev_info_get(i, &info);
+		if (ret < 0)
+			continue;
+
+		/* skip non-dlb2 event devices */
+		if (strncmp(info.driver_name, dlb2_eventdev_name,
+			    sizeof(*info.driver_name)) != 0) {
+			skipped++;
+			continue;
+		}
+
+		evdev = rte_event_dev_get_dev_id(info.driver_name);
+		if (evdev < 0) {
+			printf("Could not get dev_id for eventdev with name %s, i=%d\n",
+			       info.driver_name, i);
+			skipped++;
+			continue;
+		}
+		found++;
+		printf("Running selftest on eventdev %s\n", info.driver_name);
+		ret = do_selftest();
+		if (ret == 0) {
+			passed++;
+			printf("Selftest passed for eventdev %s\n",
+			       info.driver_name);
+		} else {
+			failed++;
+			printf("Selftest failed for eventdev %s, err=%d\n",
+			       info.driver_name, ret);
+		}
+	}
+
+	printf("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n",
+	       found, skipped, passed, failed);
+	return ret;
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index b734ff6..340bc57 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -14,7 +14,8 @@ sources = files('dlb2.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c',
-		'rte_pmd_dlb2.c'
+		'rte_pmd_dlb2.c',
+		'dlb2_selftest.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v8 22/23] event/dlb2: add queue and port release

[Timothy McDaniel]

DLB does not support reconfiguring individual queues
or ports on the fly. The entire device must be reconfigured.
Previously allocated port QE and memzone memory
is freed in this patch.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 99a22e6..d9b13a1 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3668,6 +3668,28 @@ dlb2_eventdev_close(struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(id);
+
+	/* This function intentionally left blank. */
+}
+
+static void
+dlb2_eventdev_port_release(void *port)
+{
+	struct dlb2_eventdev_port *ev_port = port;
+	struct dlb2_port *qm_port;
+
+	if (ev_port) {
+		qm_port = &ev_port->qm_port;
+		if (qm_port->config_state == DLB2_CONFIGURED)
+			dlb2_free_qe_mem(qm_port);
+	}
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3681,8 +3703,10 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
+		.queue_release    = dlb2_eventdev_queue_release,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_release     = dlb2_eventdev_port_release,
 		.port_link        = dlb2_eventdev_port_link,
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
-- 
2.6.4

[dpdk-dev] [PATCH v8 23/23] event/dlb2: add timeout ticks entry point

[Timothy McDaniel]

Adds the timeout ticks conversion function.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index d9b13a1..d12cbf7 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3689,6 +3689,18 @@ dlb2_eventdev_port_release(void *port)
 	}
 }
 
+static int
+dlb2_eventdev_timeout_ticks(struct rte_eventdev *dev, uint64_t ns,
+			    uint64_t *timeout_ticks)
+{
+	RTE_SET_USED(dev);
+	uint64_t cycles_per_ns = rte_get_timer_hz() / 1E9;
+
+	*timeout_ticks = ns * cycles_per_ns;
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -3711,6 +3723,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
 				    dlb2_eventdev_port_unlinks_in_progress,
+		.timeout_ticks    = dlb2_eventdev_timeout_ticks,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v9 00/23] Add DLB2 PMD

[Timothy McDaniel]

The following patch series adds support for a new eventdev PMD. The DLB2
PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
hardware.
The DLB2 is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB2 hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB2 ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a
directed queue. Additionally, even though LDB ports may link multiple
queues,
the number of queues that may be linked is limited by hardware.

While reviewing the code, please be aware that this PMD has full
control over the DLB2 hardware. Intel will be extending the DLB2 PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.
Note that the DLB2 hardware is a successor of the DLB hardware, and
as such is structured similarly, both in terms of code layout and
implementation.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Changes since V8:
=================
- Fixed format errors in doc/api/doxy-api-index.md
- Delayed introduction of dlb2_consume_qe_immediate until
  add-dequeue-and-its-burst-variants.patch
- Delayed introduction of dlb2_construct_token_pop_qe until 
  add-PMD-s-token-pop-public-interface.patch

Changes since V7:
================
- fix CENTOS build error: use __m128i instead of __v2di with
  _mm_stream_si128
- deleted unused dlb2_umwait and dlb2_umonitor functions

Changes since V6:
=================
- removed unused function, fixing build error
- fixed typo in port_setup commit message
- this patch series is based on dpdk-next-eventdev

Changes since V5:
================
- convert to use rte_power_monitor patches
- replace __builtin_ia32_movntdq() with _mm_stream_si128()
- remove unused functions in dlb_selftest.c
- workaround for RHEL 7 gcc brace bug

Changes since V4:
================
- moved introduction of dlb in relnotes_20_11 to first patch in series
- fixed underlines in dlb.rst that were too short
- note that the code still uses its private byte-encoded versions of
  umonitor/umwait, rather than the new functions in the power
  patch that are built on top of those intrinsics. This is intentional.

Changes since V3:
================
- updated MAINTAINERS file to alphabetically insert DLB
- don't create RTE_ symbols in pmd
- converted to use version.map scheme
- converted to use .._master_lcore instead of .._main_lcore
- this patch set is based on dpdk-next-eventdev

Changes since V2:
================
- fixed meson conditional build. Moved test into driver’s meson.build
  file instead of event/meson.build
- documentation is populated as associated code is introduced
- add log_register in add dynamic logging patch
- rename RTE_xxx symbol(s) as DLB2_xxx
- replaced function ptr enqueue_four with direct call to movdir64b
- remove unused port_pages
- broke up probe patch into 3 smaller patches for easier review
- changed param order of movdir64b/movntdq to match intrinsics
- added self to MAINTAINERS files
- squashed announcement of availability into last patch in series
- correct spelling errors and delete repeated words
- DPDK_21.0 -> DPDK 21 in map file
- add experimental banner to public structs and APIs

Changes since V1:
=================
- implement changes requested in code reviews by Gage Eads and Mike
  Chen
- fix a memzone leak
- convert to use eal rte-cpuflags patch from Liang Ma

Known Issues:
- the documentation contained in dlb2.rst is not complete, and
  should be updated to include command line parameters (class of service,
  etc ...).

Depends-on: patch-82202 ("eventdev: increase MAX QUEUES PER DEV to 255")

Timothy McDaniel (23):
  event/dlb2: add documentation and meson build infrastructure
  event/dlb2: add dynamic logging
  event/dlb2: add private data structures and constants
  event/dlb2: add definitions shared with LKM or shared code
  event/dlb2: add inline functions
  event/dlb2: add eventdev probe
  event/dlb2: add flexible interface
  event/dlb2: add probe-time hardware init
  event/dlb2: add xstats
  event/dlb2: add infos get and configure
  event/dlb2: add queue and port default conf
  event/dlb2: add queue setup
  event/dlb2: add port setup
  event/dlb2: add port link
  event/dlb2: add port unlink and port unlinks in progress
  event/dlb2: add eventdev start
  event/dlb2: add enqueue and its burst variants
  event/dlb2: add dequeue and its burst variants
  event/dlb2: add eventdev stop and close
  event/dlb2: add PMD's token pop public interface
  event/dlb2: add PMD self-tests
  event/dlb2: add queue and port release
  event/dlb2: add timeout ticks entry point

 MAINTAINERS                                    |    6 +-
 app/test/test_eventdev.c                       |    7 +
 config/rte_config.h                            |    7 +
 doc/api/doxy-api-index.md                      |    3 +-
 doc/guides/eventdevs/dlb2.rst                  |  365 ++
 doc/guides/eventdevs/index.rst                 |    1 +
 doc/guides/rel_notes/release_20_11.rst         |    5 +
 drivers/event/dlb2/dlb2.c                      | 3947 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c                |   74 +
 drivers/event/dlb2/dlb2_iface.h                |   74 +
 drivers/event/dlb2/dlb2_inline_fns.h           |   42 +
 drivers/event/dlb2/dlb2_log.h                  |   25 +
 drivers/event/dlb2/dlb2_priv.h                 |  578 +++
 drivers/event/dlb2/dlb2_selftest.c             | 1524 ++++++
 drivers/event/dlb2/dlb2_user.h                 |  679 +++
 drivers/event/dlb2/dlb2_xstats.c               | 1240 +++++
 drivers/event/dlb2/meson.build                 |   22 +
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 +++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 +
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c     | 6027 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  673 +++
 drivers/event/dlb2/pf/dlb2_main.h              |   97 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  728 +++
 drivers/event/dlb2/rte_pmd_dlb2.c              |   39 +
 drivers/event/dlb2/rte_pmd_dlb2.h              |   72 +
 drivers/event/dlb2/version.map                 |    9 +
 drivers/event/meson.build                      |    3 +-
 33 files changed, 22479 insertions(+), 3 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
 create mode 100644 drivers/event/dlb2/dlb2_log.h
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c
 create mode 100644 drivers/event/dlb2/dlb2_user.h
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
 create mode 100644 drivers/event/dlb2/version.map

-- 
2.6.4

[dpdk-dev] [PATCH v9 01/23] event/dlb2: add documentation and meson build infrastructure

[Timothy McDaniel]

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux 64 bit X86 platforms at this time.

Adds announcement of availabililty for the new driver
for Intel Dynamic Load Balancer 2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 MAINTAINERS                            |  6 ++++-
 config/rte_config.h                    |  7 ++++++
 doc/guides/eventdevs/dlb2.rst          | 40 ++++++++++++++++++++++++++++++++++
 doc/guides/eventdevs/index.rst         |  1 +
 doc/guides/rel_notes/release_20_11.rst |  5 +++++
 drivers/event/dlb2/meson.build         | 13 +++++++++++
 drivers/event/dlb2/version.map         |  3 +++
 drivers/event/meson.build              |  3 ++-
 8 files changed, 76 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index a3d1927..727451b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1162,6 +1162,11 @@ Cavium OCTEON TX timvf
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 F: drivers/event/octeontx/timvf_*
 
+Intel DLB2
+M: Timothy McDaniel <timothy.mcdaniel@intel.com>
+F: drivers/event/dlb2/
+F: doc/guides/eventdevs/dlb2.rst
+
 Marvell OCTEON TX2
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 M: Jerin Jacob <jerinj@marvell.com>
@@ -1198,7 +1203,6 @@ M: Peter Mccarthy <peter.mccarthy@intel.com>
 F: drivers/event/opdl/
 F: doc/guides/eventdevs/opdl.rst
 
-
 Rawdev Drivers
 --------------
 
diff --git a/config/rte_config.h b/config/rte_config.h
index b78c6aa..64aa333 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -135,4 +135,11 @@
 /* QEDE PMD defines */
 #define RTE_LIBRTE_QEDE_FW ""
 
+/* DLB2 defines */
+#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
+#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
+#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
+#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
+
 #endif /* _RTE_CONFIG_H_ */
diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
new file mode 100644
index 0000000..e3091c9
--- /dev/null
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -0,0 +1,40 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Driver for the Intel® Dynamic Load Balancer (DLB2)
+==================================================
+
+The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
+
+Prerequisites
+-------------
+
+Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
+the basic DPDK environment.
+
+Configuration
+-------------
+
+The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
+device access. To use this operation mode, the PCIe PF device must be bound
+to a DPDK-compatible VFIO driver, such as vfio-pci.
+
+Eventdev API Notes
+------------------
+
+The DLB2 provides the functions of a DPDK event device; specifically, it
+supports atomic, ordered, and parallel scheduling events from queues to ports.
+However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
+features are abstracted by the PMD such as directed ports.
+
+In general the dlb PMD is designed for ease-of-use and does not require a
+detailed understanding of the hardware, but these details are important when
+writing high-performance code. This section describes the places where the
+eventdev API and DLB2 misalign.
+
+Flow ID
+~~~~~~~
+
+The flow ID field is preserved in the event when it is scheduled in the
+DLB2.
+
diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
index bb66a5e..738788d 100644
--- a/doc/guides/eventdevs/index.rst
+++ b/doc/guides/eventdevs/index.rst
@@ -11,6 +11,7 @@ application through the eventdev API.
     :maxdepth: 2
     :numbered:
 
+    dlb2
     dpaa
     dpaa2
     dsw
diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
index d8ac359..dbce3b1 100644
--- a/doc/guides/rel_notes/release_20_11.rst
+++ b/doc/guides/rel_notes/release_20_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
+
+  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
+  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
+
 * **Added write combining store APIs.**
 
   Added ``rte_write32_wc`` and ``rte_write32_wc_relaxed`` APIs
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
new file mode 100644
index 0000000..5324043
--- /dev/null
+++ b/drivers/event/dlb2/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2020 Intel Corporation
+
+if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
+        build = false
+        reason = 'only supported on ARCH_X86_64 Linux'
+        subdir_done()
+endif
+
+sources = files(
+)
+
+deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
new file mode 100644
index 0000000..4a76d1d
--- /dev/null
+++ b/drivers/event/dlb2/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index a7dac99..c433c10 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -5,7 +5,8 @@ if is_windows
 	subdir_done()
 endif
 
-drivers = ['dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw', 'dsw']
+drivers = ['dlb2', 'dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw',
+	   'dsw']
 if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
-- 
2.6.4

[dpdk-dev] [PATCH v9 02/23] event/dlb2: add dynamic logging

[Timothy McDaniel]

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c      |  7 +++++++
 drivers/event/dlb2/dlb2_log.h  | 25 +++++++++++++++++++++++++
 drivers/event/dlb2/meson.build |  3 +--
 3 files changed, 33 insertions(+), 2 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_log.h

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
new file mode 100644
index 0000000..2082dd6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2.c
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <rte_log.h>
+
+RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
new file mode 100644
index 0000000..dc1481e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_log.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_EVDEV_LOG_H_
+#define _DLB2_EVDEV_LOG_H_
+
+extern int eventdev_dlb2_log_level;
+
+/* Dynamic logging */
+#define DLB2_LOG_IMPL(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
+		__func__, ##args)
+
+#define DLB2_LOG_INFO(fmt, args...) \
+	DLB2_LOG_IMPL(INFO, fmt, ## args)
+
+#define DLB2_LOG_ERR(fmt, args...) \
+	DLB2_LOG_IMPL(ERR, fmt, ## args)
+
+/* remove debug logs at compile time unless actually debugging */
+#define DLB2_LOG_DBG(fmt, args...) \
+	RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
+
+#endif /* _DLB2_EVDEV_LOG_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 5324043..435c1ee 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,7 +7,6 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files(
-)
+sources = files('dlb2.c')
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v9 03/23] event/dlb2: add private data structures and constants

[Timothy McDaniel]

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_priv.h | 575 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 575 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_priv.h

diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
new file mode 100644
index 0000000..61567a6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -0,0 +1,575 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_PRIV_H_
+#define _DLB2_PRIV_H_
+
+#include <emmintrin.h>
+#include <stdbool.h>
+
+#include <rte_eventdev.h>
+#include <rte_config.h>
+#include "dlb2_user.h"
+#include "dlb2_log.h"
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
+#else
+#define DLB2_INC_STAT(_stat, _incr_val)
+#endif
+
+#define EVDEV_DLB2_NAME_PMD dlb2_event
+
+/*  command line arg strings */
+#define NUMA_NODE_ARG "numa_node"
+#define DLB2_MAX_NUM_EVENTS "max_num_events"
+#define DLB2_NUM_DIR_CREDITS "num_dir_credits"
+#define DEV_ID_ARG "dev_id"
+#define DLB2_DEFER_SCHED_ARG "defer_sched"
+#define DLB2_QID_DEPTH_THRESH_ARG "qid_depth_thresh"
+#define DLB2_COS_ARG "cos"
+
+/* Begin HW related defines and structs */
+
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_VFS 16
+#define DLB2_MAX_NUM_LDB_QUEUES 32
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS 64
+#define DLB2_MAX_NUM_DIR_QUEUES 64
+#define DLB2_MAX_NUM_FLOWS (64 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDITS (8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS (2 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_DIR_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQOS_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_MAX_DEVICE_PATH 32
+#define DLB2_MIN_DEQUEUE_TIMEOUT_NS 1
+/* Note: "- 1" here to support the timeout range check in eventdev_autotest */
+#define DLB2_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1)
+#define DLB2_SW_CREDIT_BATCH_SZ 32
+#define DLB2_NUM_SN_GROUPS 2
+#define DLB2_MAX_LDB_SN_ALLOC 1024
+#define DLB2_MAX_QUEUE_DEPTH_THRESHOLD 8191
+
+/* 2048 total hist list entries and 64 total ldb ports, which
+ * makes for 2048/64 == 32 hist list entries per port. However, CQ
+ * depth must be a power of 2 and must also be >= HIST LIST entries.
+ * As a result we just limit the maximum dequeue depth to 32.
+ */
+#define DLB2_MIN_CQ_DEPTH 1
+#define DLB2_MAX_CQ_DEPTH 32
+#define DLB2_MIN_HARDWARE_CQ_DEPTH 8
+#define DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \
+	DLB2_MAX_CQ_DEPTH
+
+/*
+ * Static per queue/port provisioning values
+ */
+#define DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 64
+
+#define CQ_BASE(is_dir) ((is_dir) ? DLB2_DIR_CQ_BASE : DLB2_LDB_CQ_BASE)
+#define CQ_SIZE(is_dir) ((is_dir) ? DLB2_DIR_CQ_MAX_SIZE : \
+				    DLB2_LDB_CQ_MAX_SIZE)
+#define PP_BASE(is_dir) ((is_dir) ? DLB2_DIR_PP_BASE : DLB2_LDB_PP_BASE)
+
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+
+#define DLB2_NUM_QES_PER_CACHE_LINE 4
+
+#define DLB2_MAX_ENQUEUE_DEPTH 64
+#define DLB2_MIN_ENQUEUE_DEPTH 4
+
+#define DLB2_NAME_SIZE 64
+
+#define DLB2_1K 1024
+#define DLB2_2K (2 * DLB2_1K)
+#define DLB2_4K (4 * DLB2_1K)
+#define DLB2_16K (16 * DLB2_1K)
+#define DLB2_32K (32 * DLB2_1K)
+#define DLB2_1MB (DLB2_1K * DLB2_1K)
+#define DLB2_16MB (16 * DLB2_1MB)
+
+/* Use the upper 3 bits of the event priority to select the DLB2 priority */
+#define EV_TO_DLB2_PRIO(x) ((x) >> 5)
+#define DLB2_TO_EV_PRIO(x) ((x) << 5)
+
+enum dlb2_hw_port_types {
+	DLB2_LDB_PORT,
+	DLB2_DIR_PORT,
+	DLB2_NUM_PORT_TYPES /* Must be last */
+};
+
+enum dlb2_hw_queue_types {
+	DLB2_LDB_QUEUE,
+	DLB2_DIR_QUEUE,
+	DLB2_NUM_QUEUE_TYPES /* Must be last */
+};
+
+#define PORT_TYPE(p) ((p)->is_directed ? DLB2_DIR_PORT : DLB2_LDB_PORT)
+
+/* Do not change - must match hardware! */
+enum dlb2_hw_sched_type {
+	DLB2_SCHED_ATOMIC = 0,
+	DLB2_SCHED_UNORDERED,
+	DLB2_SCHED_ORDERED,
+	DLB2_SCHED_DIRECTED,
+	/* DLB2_NUM_HW_SCHED_TYPES must be last */
+	DLB2_NUM_HW_SCHED_TYPES
+};
+
+struct dlb2_hw_rsrcs {
+	int32_t nb_events_limit;
+	uint32_t num_queues;		/* Total queues (lb + dir) */
+	uint32_t num_ldb_queues;	/* Number of available ldb queues */
+	uint32_t num_ldb_ports;         /* Number of load balanced ports */
+	uint32_t num_dir_ports;         /* Number of directed ports */
+	uint32_t num_ldb_credits;       /* Number of load balanced credits */
+	uint32_t num_dir_credits;       /* Number of directed credits */
+	uint32_t reorder_window_size;   /* Size of reorder window */
+};
+
+struct dlb2_hw_resource_info {
+	/**> Max resources that can be provided */
+	struct dlb2_hw_rsrcs hw_rsrc_max;
+	int num_sched_domains;
+	uint32_t socket_id;
+};
+
+enum dlb2_enqueue_type {
+	/**>
+	 * New : Used to inject a new packet into the QM.
+	 */
+	DLB2_ENQ_NEW,
+	/**>
+	 * Forward : Enqueues a packet, and
+	 *  - if atomic: release any lock it holds in the QM
+	 *  - if ordered: release the packet for egress re-ordering
+	 */
+	DLB2_ENQ_FWD,
+	/**>
+	 * Enqueue Drop : Release an inflight packet. Must be called with
+	 * event == NULL. Used to drop a packet.
+	 *
+	 * Note that all packets dequeued from a load-balanced port must be
+	 * released, either with DLB2_ENQ_DROP or DLB2_ENQ_FWD.
+	 */
+	DLB2_ENQ_DROP,
+
+	/* marker for array sizing etc. */
+	_DLB2_NB_ENQ_TYPES
+};
+
+/* hw-specific format - do not change */
+
+struct dlb2_event_type {
+	uint8_t major:4;
+	uint8_t unused:4;
+	uint8_t sub;
+};
+
+union dlb2_opaque_data {
+	uint16_t opaque_data;
+	struct dlb2_event_type event_type;
+};
+
+struct dlb2_msg_info {
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+};
+
+#define DLB2_NEW_CMD_BYTE 0x08
+#define DLB2_FWD_CMD_BYTE 0x0A
+#define DLB2_COMP_CMD_BYTE 0x02
+#define DLB2_POP_CMD_BYTE 0x01
+#define DLB2_NOOP_CMD_BYTE 0x00
+
+/* hw-specific format - do not change */
+struct dlb2_enqueue_qe {
+	uint64_t data;
+	/* Word 3 */
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	/* Word 4 */
+	uint16_t lock_id;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_cq_pop_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t tokens:10;
+	uint16_t rsvd2:6;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_dequeue_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t flow_id:16; /* was pp_id in v1 */
+	uint8_t debug;
+	uint8_t cq_gen:1;
+	uint8_t qid_depth:2; /* 2 bits in v2 */
+	uint8_t rsvd1:2;
+	uint8_t error:1;
+	uint8_t rsvd2:2;
+};
+
+union dlb2_port_config {
+	struct dlb2_create_ldb_port_args ldb;
+	struct dlb2_create_dir_port_args dir;
+};
+
+enum dlb2_port_state {
+	PORT_CLOSED,
+	PORT_STARTED,
+	PORT_STOPPED
+};
+
+enum dlb2_configuration_state {
+	/* The resource has not been configured */
+	DLB2_NOT_CONFIGURED,
+	/* The resource was configured, but the device was stopped */
+	DLB2_PREV_CONFIGURED,
+	/* The resource is currently configured */
+	DLB2_CONFIGURED
+};
+
+struct dlb2_port {
+	uint32_t id;
+	bool is_directed;
+	bool gen_bit;
+	uint16_t dir_credits;
+	uint32_t dequeue_depth;
+	union dlb2_port_config cfg;
+	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
+	uint16_t cached_ldb_credits;
+	uint16_t ldb_credits;
+	uint16_t cached_dir_credits;
+	bool int_armed;
+	uint16_t owed_tokens;
+	int16_t issued_releases;
+	int cq_depth;
+	uint16_t cq_idx;
+	uint16_t cq_idx_unmasked;
+	uint16_t cq_depth_mask;
+	uint16_t gen_bit_shift;
+	enum dlb2_port_state state;
+	enum dlb2_configuration_state config_state;
+	int num_mapped_qids;
+	uint8_t *qid_mappings;
+	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
+	struct dlb2_enqueue_qe *int_arm_qe;
+	struct dlb2_cq_pop_qe *consume_qe;
+	struct dlb2_eventdev *dlb2; /* back ptr */
+	struct dlb2_eventdev_port *ev_port; /* back ptr */
+};
+
+/* Per-process per-port mmio and memory pointers */
+struct process_local_port_data {
+	uint64_t *pp_addr;
+	struct dlb2_dequeue_qe *cq_base;
+	const struct rte_memzone *mz;
+	bool mmaped;
+};
+
+struct dlb2_eventdev;
+
+struct dlb2_config {
+	int configured;
+	int reserved;
+	uint32_t num_ldb_credits;
+	uint32_t num_dir_credits;
+	struct dlb2_create_sched_domain_args resources;
+};
+
+enum dlb2_cos {
+	DLB2_COS_DEFAULT = -1,
+	DLB2_COS_0 = 0,
+	DLB2_COS_1,
+	DLB2_COS_2,
+	DLB2_COS_3
+};
+
+struct dlb2_hw_dev {
+	struct dlb2_config cfg;
+	struct dlb2_hw_resource_info info;
+	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
+	uint32_t domain_id;
+	enum dlb2_cos cos_id;
+	rte_spinlock_t resource_lock; /* for MP support */
+} __rte_cache_aligned;
+
+/* End HW related defines and structs */
+
+/* Begin DLB2 PMD Eventdev related defines and structs */
+
+#define DLB2_MAX_NUM_QUEUES \
+	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
+
+#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS + DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
+
+/** Structure to hold the queue to port link establishment attributes */
+
+struct dlb2_event_queue_link {
+	uint8_t queue_id;
+	uint8_t priority;
+	bool mapped;
+	bool valid;
+};
+
+struct dlb2_traffic_stats {
+	uint64_t rx_ok;
+	uint64_t rx_drop;
+	uint64_t rx_interrupt_wait;
+	uint64_t rx_umonitor_umwait;
+	uint64_t tx_ok;
+	uint64_t total_polls;
+	uint64_t zero_polls;
+	uint64_t tx_nospc_ldb_hw_credits;
+	uint64_t tx_nospc_dir_hw_credits;
+	uint64_t tx_nospc_inflight_max;
+	uint64_t tx_nospc_new_event_limit;
+	uint64_t tx_nospc_inflight_credits;
+};
+
+/* DLB2 HW sets the 2bit qid_depth in rx QEs based on the programmable depth
+ * threshold. The global default value in config/common_base (or rte_config.h)
+ * can be overridden on a per-qid basis using a vdev command line parameter.
+ * 3: depth > threshold
+ * 2: threshold >= depth > 3/4 threshold
+ * 1: 3/4 threshold >= depth > 1/2 threshold
+ * 0: depth <= 1/2 threshold.
+ */
+#define DLB2_QID_DEPTH_LE50 0
+#define DLB2_QID_DEPTH_GT50_LE75 1
+#define DLB2_QID_DEPTH_GT75_LE100 2
+#define DLB2_QID_DEPTH_GT100 3
+#define DLB2_NUM_QID_DEPTH_STAT_VALS 4 /* 2 bits */
+
+struct dlb2_queue_stats {
+	uint64_t enq_ok;
+	uint64_t qid_depth[DLB2_NUM_QID_DEPTH_STAT_VALS];
+};
+
+struct dlb2_port_stats {
+	struct dlb2_traffic_stats traffic;
+	uint64_t tx_op_cnt[4]; /* indexed by rte_event.op */
+	uint64_t tx_implicit_rel;
+	uint64_t tx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t tx_invalid;
+	uint64_t rx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t rx_sched_invalid;
+	struct dlb2_queue_stats queue[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_eventdev_port {
+	struct dlb2_port qm_port; /* hw specific data structure */
+	struct rte_event_port_conf conf; /* user-supplied configuration */
+	uint16_t inflight_credits; /* num credits this port has right now */
+	uint16_t credit_update_quanta;
+	struct dlb2_eventdev *dlb2; /* backlink optimization */
+	struct dlb2_port_stats stats __rte_cache_aligned;
+	struct dlb2_event_queue_link link[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	int num_links;
+	uint32_t id; /* port id */
+	/* num releases yet to be completed on this port.
+	 * Only applies to load-balanced ports.
+	 */
+	uint16_t outstanding_releases;
+	uint16_t inflight_max; /* app requested max inflights for this port */
+	/* setup_done is set when the event port is setup */
+	bool setup_done;
+	/* enq_configured is set when the qm port is created */
+	bool enq_configured;
+	uint8_t implicit_release; /* release events before dequeueing */
+}  __rte_cache_aligned;
+
+struct dlb2_queue {
+	uint32_t num_qid_inflights; /* User config */
+	uint32_t num_atm_inflights; /* User config */
+	enum dlb2_configuration_state config_state;
+	int sched_type; /* LB queue only */
+	uint32_t id;
+	bool is_directed;
+};
+
+struct dlb2_eventdev_queue {
+	struct dlb2_queue qm_queue;
+	struct rte_event_queue_conf conf; /* User config */
+	int depth_threshold; /* use default if 0 */
+	uint32_t id;
+	bool setup_done;
+	uint8_t num_links;
+};
+
+enum dlb2_run_state {
+	DLB2_RUN_STATE_STOPPED = 0,
+	DLB2_RUN_STATE_STOPPING,
+	DLB2_RUN_STATE_STARTING,
+	DLB2_RUN_STATE_STARTED
+};
+
+struct dlb2_eventdev {
+	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
+	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each queue */
+	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
+	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each port */
+	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
+	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
+	struct dlb2_get_num_resources_args hw_rsrc_query_results;
+	uint32_t xstats_count_mode_queue;
+	struct dlb2_hw_dev qm_instance; /* strictly hw related */
+	uint64_t global_dequeue_wait_ticks;
+	struct dlb2_xstats_entry *xstats;
+	struct rte_eventdev *event_dev; /* backlink to dev */
+	uint32_t xstats_count_mode_dev;
+	uint32_t xstats_count_mode_port;
+	uint32_t xstats_count;
+	uint32_t inflights; /* use __atomic builtins */
+	uint32_t new_event_limit;
+	int max_num_events_override;
+	int num_dir_credits_override;
+	volatile enum dlb2_run_state run_state;
+	uint16_t num_dir_queues; /* total num of evdev dir queues requested */
+	uint16_t num_dir_credits;
+	uint16_t num_ldb_credits;
+	uint16_t num_queues; /* total queues */
+	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested */
+	uint16_t num_ports; /* total num of evdev ports requested */
+	uint16_t num_ldb_ports; /* total num of ldb ports requested */
+	uint16_t num_dir_ports; /* total num of dir ports requested */
+	bool umwait_allowed;
+	bool global_dequeue_wait; /* Not using per dequeue wait if true */
+	bool defer_sched;
+	enum dlb2_cq_poll_modes poll_mode;
+	uint8_t revision;
+	bool configured;
+	uint16_t max_ldb_credits;
+	uint16_t max_dir_credits;
+
+	/* force hw credit pool counters into exclusive cache lines */
+
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t ldb_credit_pool __rte_cache_aligned;
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t dir_credit_pool __rte_cache_aligned;
+};
+
+/* used for collecting and passing around the dev args */
+struct dlb2_qid_depth_thresholds {
+	int val[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_devargs {
+	int socket_id;
+	int max_num_events;
+	int num_dir_credits_override;
+	int dev_id;
+	int defer_sched;
+	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
+	enum dlb2_cos cos_id;
+};
+
+/* End Eventdev related defines and structs */
+
+/* Forwards for non-inlined functions */
+
+void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
+
+int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
+
+void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
+
+int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n);
+
+int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstat_names,
+		unsigned int *ids, unsigned int size);
+
+uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+					  const char *name, unsigned int *id);
+
+int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode,
+		int16_t queue_port_id,
+		const uint32_t ids[],
+		uint32_t nb_ids);
+
+int test_dlb2_eventdev(void);
+
+int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+				const char *name,
+				struct dlb2_devargs *dlb2_args);
+
+int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+				  const char *name);
+
+uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *queue);
+
+int dlb2_parse_params(const char *params,
+		      const char *name,
+		      struct dlb2_devargs *dlb2_args);
+
+/* Extern globals */
+extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
+
+#endif	/* _DLB2_PRIV_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v9 04/23] event/dlb2: add definitions shared with LKM or shared code

[Timothy McDaniel]

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_user.h | 679 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 679 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_user.h

diff --git a/drivers/event/dlb2/dlb2_user.h b/drivers/event/dlb2/dlb2_user.h
new file mode 100644
index 0000000..f4bda78
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_user.h
@@ -0,0 +1,679 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#define DLB2_MAX_NAME_LEN 64
+
+#include <linux/types.h>
+
+enum dlb2_error {
+	DLB2_ST_SUCCESS = 0,
+	DLB2_ST_NAME_EXISTS,
+	DLB2_ST_DOMAIN_UNAVAILABLE,
+	DLB2_ST_LDB_PORTS_UNAVAILABLE,
+	DLB2_ST_DIR_PORTS_UNAVAILABLE,
+	DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB2_ST_INVALID_DOMAIN_ID,
+	DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB2_ST_INVALID_LDB_QUEUE_ID,
+	DLB2_ST_INVALID_CQ_DEPTH,
+	DLB2_ST_INVALID_CQ_VIRT_ADDR,
+	DLB2_ST_INVALID_PORT_ID,
+	DLB2_ST_INVALID_QID,
+	DLB2_ST_INVALID_PRIORITY,
+	DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+	DLB2_ST_INVALID_DIR_QUEUE_ID,
+	DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB2_ST_DOMAIN_NOT_CONFIGURED,
+	DLB2_ST_INTERNAL_ERROR,
+	DLB2_ST_DOMAIN_IN_USE,
+	DLB2_ST_DOMAIN_NOT_FOUND,
+	DLB2_ST_QUEUE_NOT_FOUND,
+	DLB2_ST_DOMAIN_STARTED,
+	DLB2_ST_DOMAIN_NOT_STARTED,
+	DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB2_ST_DOMAIN_RESET_FAILED,
+	DLB2_ST_MBOX_ERROR,
+	DLB2_ST_INVALID_HIST_LIST_DEPTH,
+	DLB2_ST_NO_MEMORY,
+	DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB2_ST_INVALID_COS_ID,
+};
+
+static const char dlb2_error_strings[][128] = {
+	"DLB2_ST_SUCCESS",
+	"DLB2_ST_NAME_EXISTS",
+	"DLB2_ST_DOMAIN_UNAVAILABLE",
+	"DLB2_ST_LDB_PORTS_UNAVAILABLE",
+	"DLB2_ST_DIR_PORTS_UNAVAILABLE",
+	"DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+	"DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+	"DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+	"DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+	"DLB2_ST_INVALID_DOMAIN_ID",
+	"DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+	"DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+	"DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+	"DLB2_ST_INVALID_LDB_QUEUE_ID",
+	"DLB2_ST_INVALID_CQ_DEPTH",
+	"DLB2_ST_INVALID_CQ_VIRT_ADDR",
+	"DLB2_ST_INVALID_PORT_ID",
+	"DLB2_ST_INVALID_QID",
+	"DLB2_ST_INVALID_PRIORITY",
+	"DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+	"DLB2_ST_INVALID_DIR_QUEUE_ID",
+	"DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+	"DLB2_ST_DOMAIN_NOT_CONFIGURED",
+	"DLB2_ST_INTERNAL_ERROR",
+	"DLB2_ST_DOMAIN_IN_USE",
+	"DLB2_ST_DOMAIN_NOT_FOUND",
+	"DLB2_ST_QUEUE_NOT_FOUND",
+	"DLB2_ST_DOMAIN_STARTED",
+	"DLB2_ST_DOMAIN_NOT_STARTED",
+	"DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+	"DLB2_ST_DOMAIN_RESET_FAILED",
+	"DLB2_ST_MBOX_ERROR",
+	"DLB2_ST_INVALID_HIST_LIST_DEPTH",
+	"DLB2_ST_NO_MEMORY",
+	"DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+	"DLB2_ST_INVALID_COS_ID",
+};
+
+struct dlb2_cmd_response {
+	__u32 status; /* Interpret using enum dlb2_error */
+	__u32 id;
+};
+
+/*******************/
+/* 'dlb2' commands */
+/*******************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+	DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	This ioctl interface is the same in all driver versions and is always
+ *	the first ioctl.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+	/* Output parameters */
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ *	load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ *		percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ *	load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ *   ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+enum dlb2_cq_poll_modes {
+	DLB2_CQ_POLL_MODE_STD,
+	DLB2_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB2_CQ_POLL_MODE must be last */
+	NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/********************************/
+/* 'scheduling domain' commands */
+/********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_start_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
+#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
+
+
+#endif /* __DLB2_USER_H */
-- 
2.6.4

[dpdk-dev] [PATCH v9 05/23] event/dlb2: add inline functions

[Timothy McDaniel]

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_inline_fns.h | 42 ++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h

diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
new file mode 100644
index 0000000..cf7265c
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_inline_fns.h
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_INLINE_FNS_H_
+#define _DLB2_INLINE_FNS_H_
+
+/* Inline functions required in more than one source file. */
+
+static inline struct dlb2_eventdev *
+dlb2_pmd_priv(const struct rte_eventdev *eventdev)
+{
+	return eventdev->data->dev_private;
+}
+
+static inline void
+dlb2_movntdq_single(void *pp_addr, void *qe4)
+{
+	long long *_qe  = (long long *)qe4;
+	__m128i src_data0 = (__m128i){_qe[0], _qe[1]};
+
+	_mm_stream_si128(pp_addr, src_data0);
+}
+
+static inline void
+dlb2_cldemote(void *addr)
+{
+	/* Load addr into RSI, then demote the cache line of the address
+	 * contained in that register.
+	 */
+	asm volatile(".byte 0x0f, 0x1c, 0x06" :: "S" (addr));
+}
+
+static inline void
+dlb2_movdir64b(void *pp_addr, void *qe4)
+{
+	asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
+		     :
+		     : "a" (pp_addr), "d" (qe4));
+}
+
+#endif /* _DLB2_INLINE_FNS_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v9 06/23] event/dlb2: add eventdev probe

[Timothy McDaniel]

Add the eventdev portion of probe, and parse command line
options, but do not initialize hardware.

Changes since v2 patch-set probe:

Primary and secondary probe-time init has been removed, and
will be introduced in subsequent patches contained in
the v3 patch-set.

Hardware init has been moved to a subsequent patch in order to
minimize the patch size. The files dlb2/pf/dlb2_pf.c and
dlb2/pf/dlb_main.c include stubs of hardware init functions,
and these will be replaced with the real versions when the
hardware layer is committed.

Initialization of the flexible interface layer has been moved to
a subsequent patch in order to minimize patch size.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                      |  343 ++++
 drivers/event/dlb2/meson.build                 |    5 +-
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +++
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  621 ++++++
 drivers/event/dlb2/pf/dlb2_main.h              |   98 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  196 ++
 13 files changed, 7497 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2082dd6..1b11979 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2,6 +2,349 @@
  * Copyright(c) 2016-2020 Intel Corporation
  */
 
+#include <assert.h>
+#include <errno.h>
+#include <nmmintrin.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_cycles.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_io.h>
+#include <rte_kvargs.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_ring.h>
+#include <rte_string_fns.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+/*
+ * Resources exposed to eventdev. Some values overridden at runtime using
+ * values returned by the DLB kernel driver.
+ */
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
+#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
+#endif
+
+struct process_local_port_data
+dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
+
+#define DLB2_BASE_10 10
+
+static int
+dlb2_string_to_int(int *result, const char *str)
+{
+	long ret;
+	char *endptr;
+
+	if (str == NULL || result == NULL)
+		return -EINVAL;
+
+	errno = 0;
+	ret = strtol(str, &endptr, DLB2_BASE_10);
+	if (errno)
+		return -errno;
+
+	/* long int and int may be different width for some architectures */
+	if (ret < INT_MIN || ret > INT_MAX || endptr == str)
+		return -EINVAL;
+
+	*result = ret;
+	return 0;
+}
+
+static int
+set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
+{
+	int *socket_id = opaque;
+	int ret;
+
+	ret = dlb2_string_to_int(socket_id, value);
+	if (ret < 0)
+		return ret;
+
+	if (*socket_id > RTE_MAX_NUMA_NODES)
+		return -EINVAL;
+	return 0;
+}
+
+static int
+set_max_num_events(const char *key __rte_unused,
+		   const char *value,
+		   void *opaque)
+{
+	int *max_num_events = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(max_num_events, value);
+	if (ret < 0)
+		return ret;
+
+	if (*max_num_events < 0 || *max_num_events >
+			DLB2_MAX_NUM_LDB_CREDITS) {
+		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
+			     DLB2_MAX_NUM_LDB_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_num_dir_credits(const char *key __rte_unused,
+		    const char *value,
+		    void *opaque)
+{
+	int *num_dir_credits = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(num_dir_credits, value);
+	if (ret < 0)
+		return ret;
+
+	if (*num_dir_credits < 0 ||
+	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
+		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
+			     DLB2_MAX_NUM_DIR_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_dev_id(const char *key __rte_unused,
+	   const char *value,
+	   void *opaque)
+{
+	int *dev_id = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(dev_id, value);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+set_cos(const char *key __rte_unused,
+	const char *value,
+	void *opaque)
+{
+	enum dlb2_cos *cos_id = opaque;
+	int x = 0;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(&x, value);
+	if (ret < 0)
+		return ret;
+
+	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
+		DLB2_LOG_ERR(
+			"COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
+			x);
+		return -EINVAL;
+	}
+
+	*cos_id = x;
+
+	return 0;
+}
+
+
+static int
+set_qid_depth_thresh(const char *key __rte_unused,
+		     const char *value,
+		     void *opaque)
+{
+	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
+	int first, last, thresh, i;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	/* command line override may take one of the following 3 forms:
+	 * qid_depth_thresh=all:<threshold_value> ... all queues
+	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
+	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
+	 */
+	if (sscanf(value, "all:%d", &thresh) == 1) {
+		first = 0;
+		last = DLB2_MAX_NUM_QUEUES - 1;
+	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
+		/* we have everything we need */
+	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
+		last = first;
+	} else {
+		DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val, qid-qid:val, or qid:val\n");
+		return -EINVAL;
+	}
+
+	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
+		return -EINVAL;
+	}
+
+	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
+			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
+		return -EINVAL;
+	}
+
+	for (i = first; i <= last; i++)
+		qid_thresh->val[i] = thresh; /* indexed by qid */
+
+	return 0;
+}
+
+int
+dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+			    const char *name,
+			    struct dlb2_devargs *dlb2_args)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+	RTE_SET_USED(dlb2_args);
+
+	return 0;
+}
+
+int
+dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+			      const char *name)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+int
+dlb2_parse_params(const char *params,
+		  const char *name,
+		  struct dlb2_devargs *dlb2_args)
+{
+	int ret = 0;
+	static const char * const args[] = { NUMA_NODE_ARG,
+					     DLB2_MAX_NUM_EVENTS,
+					     DLB2_NUM_DIR_CREDITS,
+					     DEV_ID_ARG,
+					     DLB2_QID_DEPTH_THRESH_ARG,
+					     DLB2_COS_ARG,
+					     NULL };
+
+	if (params != NULL && params[0] != '\0') {
+		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
+
+		if (kvlist == NULL) {
+			RTE_LOG(INFO, PMD,
+				"Ignoring unsupported parameters when creating device '%s'\n",
+				name);
+		} else {
+			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
+						     set_numa_node,
+						     &dlb2_args->socket_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing numa node parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
+						 set_max_num_events,
+						 &dlb2_args->max_num_events);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist,
+					DLB2_NUM_DIR_CREDITS,
+					set_num_dir_credits,
+					&dlb2_args->num_dir_credits_override);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
+						 set_dev_id,
+						 &dlb2_args->dev_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(
+					kvlist,
+					DLB2_QID_DEPTH_THRESH_ARG,
+					set_qid_depth_thresh,
+					&dlb2_args->qid_depth_thresholds);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
+						 set_cos,
+						 &dlb2_args->cos_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing cos parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
 
+			rte_kvargs_free(kvlist);
+		}
+	}
+	return ret;
+}
 RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 435c1ee..1eab9b9 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,6 +7,9 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files('dlb2.c')
+sources = files('dlb2.c',
+		'pf/dlb2_main.c',
+		'pf/dlb2_pf.c'
+)
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
new file mode 100644
index 0000000..1d99f1e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_osdep_list.h"
+#include "dlb2_osdep_types.h"
+
+#define DLB2_MAX_NUM_VDEVS			16
+#define DLB2_MAX_NUM_DOMAINS			32
+#define DLB2_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS			64
+#define DLB2_MAX_NUM_DIR_PORTS			64
+#define DLB2_MAX_NUM_LDB_CREDITS		(8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS		(2 * 1024)
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB2_MAX_NUM_AQED_ENTRIES		2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB2_QID_PRIORITIES			8
+#define DLB2_NUM_ARB_WEIGHTS			8
+#define DLB2_MAX_WEIGHT				255
+#define DLB2_NUM_COS_DOMAINS			4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#ifdef FPGA
+#define DLB2_HZ					2000000
+#else
+#define DLB2_HZ					800000000
+#endif
+
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
+
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+	DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+	DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID		3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID		4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW		5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ	1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
+#define DLB2_VF_BASE_CQ_VECTOR_ID	     0
+#define DLB2_VF_LAST_CQ_VECTOR_ID	     30
+#define DLB2_VF_MBOX_VECTOR_ID		     31
+#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
+
+#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
+					     DLB2_MAX_NUM_DIR_PORTS + 1)
+
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE   0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
+				DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE   0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
+				DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE       0x2100000
+#define DLB2_LDB_PP_STRIDE     0x1000
+#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
+				DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE       0x2000000
+#define DLB2_DIR_PP_STRIDE     0x1000
+#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
+				DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
+struct dlb2_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb2_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb2_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
+struct dlb2_ldb_queue {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB2_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB2_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB2_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB2_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+	enum dlb2_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb2_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+	u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+	u32 bound[6] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & (1 << i))) {
+			group->slot_use_bitmap |= 1 << i;
+			return i;
+		}
+	}
+
+	return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+	int i, cnt = 0;
+
+	for (i = 0; i < 32; i++)
+		cnt += !!(group->slot_use_bitmap & (1 << i));
+
+	return cnt;
+}
+
+struct dlb2_hw_domain {
+	struct dlb2_function_resources *parent_func;
+	struct dlb2_list_entry func_list;
+	struct dlb2_list_head used_ldb_queues;
+	struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head used_dir_pq_pairs;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb2_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb2_bitmap;
+
+struct dlb2_function_resources {
+	struct dlb2_list_head avail_domains;
+	struct dlb2_list_head used_domains;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	struct dlb2_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+	struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+	struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+	struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
+struct dlb2_mbox {
+	u32 *mbox;
+	u32 *isr_in_progress;
+};
+
+struct dlb2_sw_mbox {
+	struct dlb2_mbox vdev_to_pf;
+	struct dlb2_mbox pf_to_vdev;
+	void (*pf_to_vdev_inject)(void *arg);
+	void *pf_to_vdev_inject_arg;
+};
+
+struct dlb2_hw {
+	/* BAR 0 address */
+	void  *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void  *func_kva;
+	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb2_hw_resources rsrcs;
+	struct dlb2_function_resources pf;
+	struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+	struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+	/* Virtualization */
+	int virt_mode;
+	struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
+	unsigned int pasid[DLB2_MAX_NUM_VDEVS];
+};
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
new file mode 100644
index 0000000..ce462c0
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
@@ -0,0 +1,596 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_BASE_DLB2_MBOX_H
+#define __DLB2_BASE_DLB2_MBOX_H
+
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+
+#define DLB2_MBOX_INTERFACE_VERSION 1
+
+/*
+ * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
+ * to send requests of its own (e.g. notifying a VF of an impending FLR).
+ * To avoid communication race conditions, e.g. the PF sends a response and then
+ * sends a request before the VF reads the response, the PF->VF mailbox is
+ * divided into two sections:
+ * - Bytes 0-47: PF responses
+ * - Bytes 48-63: PF requests
+ *
+ * Partitioning the PF->VF mailbox allows responses and requests to occupy the
+ * mailbox simultaneously.
+ */
+#define DLB2_PF2VF_RESP_BYTES	  48
+#define DLB2_PF2VF_RESP_BASE	  0
+#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
+
+#define DLB2_PF2VF_REQ_BYTES	  16
+#define DLB2_PF2VF_REQ_BASE	  (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
+#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
+
+/*
+ * Similarly, the VF->PF mailbox is divided into two sections:
+ * - Bytes 0-239: VF requests
+ * -- (Bytes 0-3 are unused due to a hardware errata)
+ * - Bytes 240-255: VF responses
+ */
+#define DLB2_VF2PF_REQ_BYTES	 236
+#define DLB2_VF2PF_REQ_BASE	 4
+#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
+
+#define DLB2_VF2PF_RESP_BYTES	  16
+#define DLB2_VF2PF_RESP_BASE	  (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
+#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
+
+/* VF-initiated commands */
+enum dlb2_mbox_cmd_type {
+	DLB2_MBOX_CMD_REGISTER,
+	DLB2_MBOX_CMD_UNREGISTER,
+	DLB2_MBOX_CMD_GET_NUM_RESOURCES,
+	DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
+	DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
+	DLB2_MBOX_CMD_CREATE_LDB_PORT,
+	DLB2_MBOX_CMD_CREATE_DIR_PORT,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT,
+	DLB2_MBOX_CMD_DISABLE_LDB_PORT,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT,
+	DLB2_MBOX_CMD_DISABLE_DIR_PORT,
+	DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_MAP_QID,
+	DLB2_MBOX_CMD_UNMAP_QID,
+	DLB2_MBOX_CMD_START_DOMAIN,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
+	DLB2_MBOX_CMD_ARM_CQ_INTR,
+	DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
+	DLB2_MBOX_CMD_GET_SN_ALLOCATION,
+	DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
+	DLB2_MBOX_CMD_GET_COS_BW,
+	DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
+	DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_QE_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_CMD_TYPES,
+};
+
+static const char dlb2_mbox_cmd_type_strings[][128] = {
+	"DLB2_MBOX_CMD_REGISTER",
+	"DLB2_MBOX_CMD_UNREGISTER",
+	"DLB2_MBOX_CMD_GET_NUM_RESOURCES",
+	"DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_LDB_PORT",
+	"DLB2_MBOX_CMD_CREATE_DIR_PORT",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_DISABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_DISABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_MAP_QID",
+	"DLB2_MBOX_CMD_UNMAP_QID",
+	"DLB2_MBOX_CMD_START_DOMAIN",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
+	"DLB2_MBOX_CMD_ARM_CQ_INTR",
+	"DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
+	"DLB2_MBOX_CMD_GET_SN_ALLOCATION",
+	"DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
+	"DLB2_MBOX_CMD_GET_COS_BW",
+	"DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
+	"DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
+};
+
+/* PF-initiated commands */
+enum dlb2_mbox_vf_cmd_type {
+	DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
+	DLB2_MBOX_VF_CMD_NOTIFICATION,
+	DLB2_MBOX_VF_CMD_IN_USE,
+
+	/* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_VF_CMD_TYPES,
+};
+
+static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
+	"DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
+	"DLB2_MBOX_VF_CMD_NOTIFICATION",
+	"DLB2_MBOX_VF_CMD_IN_USE",
+};
+
+#define DLB2_MBOX_CMD_TYPE(hdr) \
+	(((struct dlb2_mbox_req_hdr *)hdr)->type)
+#define DLB2_MBOX_CMD_STRING(hdr) \
+	dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
+
+enum dlb2_mbox_status_type {
+	DLB2_MBOX_ST_SUCCESS,
+	DLB2_MBOX_ST_INVALID_CMD_TYPE,
+	DLB2_MBOX_ST_VERSION_MISMATCH,
+	DLB2_MBOX_ST_INVALID_OWNER_VF,
+};
+
+static const char dlb2_mbox_status_type_strings[][128] = {
+	"DLB2_MBOX_ST_SUCCESS",
+	"DLB2_MBOX_ST_INVALID_CMD_TYPE",
+	"DLB2_MBOX_ST_VERSION_MISMATCH",
+	"DLB2_MBOX_ST_INVALID_OWNER_VF",
+};
+
+#define DLB2_MBOX_ST_TYPE(hdr) \
+	(((struct dlb2_mbox_resp_hdr *)hdr)->status)
+#define DLB2_MBOX_ST_STRING(hdr) \
+	dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
+
+/* This structure is always the first field in a request structure */
+struct dlb2_mbox_req_hdr {
+	u32 type;
+};
+
+/* This structure is always the first field in a response structure */
+struct dlb2_mbox_resp_hdr {
+	u32 status;
+};
+
+struct dlb2_mbox_register_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 min_interface_version;
+	u16 max_interface_version;
+};
+
+struct dlb2_mbox_register_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 interface_version;
+	u8 pf_id;
+	u8 vf_id;
+	u8 is_auxiliary_vf;
+	u8 primary_vf_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u16 num_sched_domains;
+	u16 num_ldb_queues;
+	u16 num_ldb_ports;
+	u16 num_cos_ldb_ports[4];
+	u16 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 max_contiguous_hist_list_entries;
+	u16 num_ldb_credits;
+	u16 num_dir_credits;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 num_ldb_queues;
+	u32 num_ldb_ports;
+	u32 num_cos_ldb_ports[4];
+	u32 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u8 cos_strict;
+	u8 padding0[3];
+	u32 padding1;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 num_sequence_numbers;
+	u32 num_qid_inflights;
+	u32 num_atomic_inflights;
+	u32 lock_id_comp_level;
+	u32 depth_threshold;
+	u32 padding;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 depth_threshold;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u16 cq_depth;
+	u16 cq_history_list_size;
+	u8 cos_id;
+	u8 cos_strict;
+	u16 padding1;
+	u64 cq_base_address;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u64 cq_base_address;
+	u16 cq_depth;
+	u16 padding0;
+	s32 queue_id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_map_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+	u32 priority;
+	u32 padding0;
+};
+
+struct dlb2_mbox_map_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_start_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+};
+
+struct dlb2_mbox_start_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 is_ldb;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding0;
+};
+
+/*
+ * The alert_id and aux_alert_data follows the format of the alerts defined in
+ * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
+ * the aux_alert_data value varies depending on the alert.
+ */
+struct dlb2_mbox_vf_alert_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 alert_id;
+	u32 aux_alert_data;
+};
+
+enum dlb2_mbox_vf_notification_type {
+	DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
+	DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
+
+	/* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
+	NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
+};
+
+struct dlb2_mbox_vf_notification_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 notification;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 in_use;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 num;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 cos_id;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 mode;
+};
+
+#endif /* __DLB2_BASE_DLB2_MBOX_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
new file mode 100644
index 0000000..c4c34eb
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
@@ -0,0 +1,230 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_H
+#define __DLB2_OSDEP_H
+
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <rte_string_fns.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include "../dlb2_main.h"
+#include "dlb2_resource.h"
+#include "../../dlb2_log.h"
+#include "../../dlb2_user.h"
+
+
+#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
+#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
+#define DLB2_CSR_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
+#define DLB2_FUNC_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
+/* Map to PMDs logging interface */
+#define DLB2_ERR(dev, fmt, args...) \
+	DLB2_LOG_ERR(fmt, ## args)
+
+#define DLB2_INFO(dev, fmt, args...) \
+	DLB2_LOG_INFO(fmt, ## args)
+
+#define DLB2_DEBUG(dev, fmt, args...) \
+	DLB2_LOG_DBG(fmt, ## args)
+
+/**
+ * os_udelay() - busy-wait for a number of microseconds
+ * @usecs: delay duration.
+ */
+static inline void os_udelay(int usecs)
+{
+	rte_delay_us(usecs);
+}
+
+/**
+ * os_msleep() - sleep for a number of milliseconds
+ * @usecs: delay duration.
+ */
+static inline void os_msleep(int msecs)
+{
+	rte_delay_ms(msecs);
+}
+
+#define DLB2_PP_BASE(__is_ldb) \
+	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
+
+/**
+ * os_map_producer_port() - map a producer port into the caller's address space
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ *
+ * This function maps the requested producer port memory into the caller's
+ * address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void *os_map_producer_port(struct dlb2_hw *hw,
+					 u8 port_id,
+					 bool is_ldb)
+{
+	uint64_t addr;
+	uint64_t pp_dma_base;
+
+	pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
+	addr = (pp_dma_base + (PAGE_SIZE * port_id));
+
+	return (void *)(uintptr_t)addr;
+}
+
+/**
+ * os_unmap_producer_port() - unmap a producer port
+ * @addr: mapped producer port address
+ *
+ * This function undoes os_map_producer_port() by unmapping the producer port
+ * memory from the caller's address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(addr);
+}
+
+/**
+ * os_fence_hcw() - fence an HCW to ensure it arrives at the device
+ * @hw: dlb2_hw handle for a particular device.
+ * @pp_addr: producer port address
+ */
+static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
+{
+	RTE_SET_USED(hw);
+
+	/* To ensure outstanding HCWs reach the device, read the PP address. IA
+	 * memory ordering prevents reads from passing older writes, and the
+	 * mfence also ensures this.
+	 */
+	rte_mb();
+
+	*(volatile u64 *)pp_addr;
+}
+
+/**
+ * DLB2_HW_ERR() - log an error message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_ERR(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_ERR(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/**
+ * DLB2_HW_DBG() - log an info message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_DBG(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_DEBUG(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/* The callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void *dlb2_complete_queue_map_unmap(void *__args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
+	int ret;
+
+	while (1) {
+		rte_spinlock_lock(&dlb2_dev->resource_mutex);
+
+		ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+		ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+		if (ret != 0) {
+			rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+			/* Relinquish the CPU so the application can process
+			 * its CQs, so this function doesn't deadlock.
+			 */
+			sched_yield();
+		} else {
+			break;
+		}
+	}
+
+	dlb2_dev->worker_launched = false;
+
+	rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+
+	return NULL;
+}
+
+
+/**
+ * os_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static inline void os_schedule_work(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+	pthread_t complete_queue_map_unmap_thread;
+	int ret;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
+				     "dlb_queue_unmap_waiter",
+				     NULL,
+				     dlb2_complete_queue_map_unmap,
+				     dlb2_dev);
+	if (ret)
+		DLB2_ERR(dlb2_dev,
+			 "Could not create queue complete map/unmap thread, err=%d\n",
+			 ret);
+	else
+		dlb2_dev->worker_launched = true;
+}
+
+/**
+ * os_worker_active() - query whether the map/unmap worker thread is active
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether a thread (launched by
+ * os_schedule_work()) is active. This function is used to determine
+ * whether or not to launch a worker thread.
+ */
+static inline bool os_worker_active(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	return dlb2_dev->worker_launched;
+}
+
+#endif /*  __DLB2_OSDEP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
new file mode 100644
index 0000000..423233b
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <rte_bitmap.h>
+#include <rte_string_fns.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include "../dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+	struct rte_bitmap *map;
+	unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+				    unsigned int len)
+{
+	struct dlb2_bitmap *bm;
+	void *mem;
+	uint32_t alloc_size;
+	uint32_t nbits = (uint32_t)len;
+
+	if (bitmap == NULL || nbits == 0)
+		return -EINVAL;
+
+	/* Allocate DLB2 bitmap control struct */
+	bm = rte_malloc("DLB2_PF",
+			sizeof(struct dlb2_bitmap),
+			RTE_CACHE_LINE_SIZE);
+
+	if (bm == NULL)
+		return -ENOMEM;
+
+	/* Allocate bitmap memory */
+	alloc_size = rte_bitmap_get_memory_footprint(nbits);
+	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
+	if (mem == NULL) {
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->map = rte_bitmap_init(len, mem, alloc_size);
+	if (bm->map == NULL) {
+		rte_free(mem);
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap == NULL)
+		return;
+
+	rte_free(bitmap->map);
+	rte_free(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++)
+		rte_bitmap_set(bitmap->map, i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	rte_bitmap_reset(bitmap->map);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set() - set a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
+				  unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_set(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+					unsigned int bit,
+					unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_set(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear() - clear a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
+				    unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_clear(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+					  unsigned int bit,
+					  unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_clear(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+						 unsigned int len)
+{
+	unsigned int i, j = 0;
+
+	if (bitmap  == NULL || bitmap->map == NULL || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			if (++j == len)
+				return i - j + 1;
+		} else {
+			j = 0;
+		}
+	}
+
+	/* No set bit range of length len? */
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			return i;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+	int weight = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			weight++;
+	}
+	return weight;
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				      bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+	int max_len = 0, len = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			len++;
+		} else {
+			if (len > max_len)
+				max_len = len;
+			len = 0;
+		}
+	}
+
+	if (len > max_len)
+		max_len = len;
+
+	return max_len;
+}
+
+/**
+ * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
+ * @dest: pointer to dlb2_bitmap structure, which will contain the results of
+ *	  the 'or' of src1 and src2.
+ * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
+ * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
+ *
+ * This function 'or's two bitmaps together and stores the result in a third
+ * bitmap. The source and destination bitmaps can be the same.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - One of the bitmaps is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
+				 struct dlb2_bitmap *src1,
+				 struct dlb2_bitmap *src2)
+{
+	unsigned int i, min;
+	int numset = 0;
+
+	if (dest  == NULL || dest->map == NULL ||
+	    src1  == NULL || src1->map == NULL ||
+	    src2  == NULL || src2->map == NULL)
+		return -EINVAL;
+
+	min = dest->len;
+	min = (min > src1->len) ? src1->len : min;
+	min = (min > src2->len) ? src2->len : min;
+
+	for (i = 0; i != min; i++) {
+		if  (rte_bitmap_get(src1->map, i) ||
+		     rte_bitmap_get(src2->map, i)) {
+			rte_bitmap_set(dest->map, i);
+			numset++;
+		} else {
+			rte_bitmap_clear(dest->map, i);
+		}
+	}
+
+	return numset;
+}
+
+#endif /*  __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
new file mode 100644
index 0000000..5531739
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_LIST_H
+#define __DLB2_OSDEP_LIST_H
+
+#include <rte_tailq.h>
+
+struct dlb2_list_entry {
+	TAILQ_ENTRY(dlb2_list_entry) node;
+};
+
+/* Dummy - just a struct definition */
+TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
+
+/* =================
+ * TAILQ Supplements
+ * =================
+ */
+
+#ifndef TAILQ_FOREACH_ENTRY
+#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)		\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter)						\
+		&& (ptr = container_of(iter, typeof(*(ptr)), name)); \
+	    (iter) = TAILQ_NEXT((iter), node))
+#endif
+
+#ifndef TAILQ_FOREACH_ENTRY_SAFE
+#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)	\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter) &&						\
+		(ptr = container_of(iter, typeof(*(ptr)), name)) &&\
+		((tvar) = TAILQ_NEXT((iter), node), 1);	\
+	    (iter) = (tvar))
+#endif
+
+/***********************/
+/*** List operations ***/
+/***********************/
+
+/**
+ * dlb2_list_init_head() - initialize the head of a list
+ * @head: list head
+ */
+static inline void dlb2_list_init_head(struct dlb2_list_head *head)
+{
+	TAILQ_INIT(head);
+}
+
+/**
+ * dlb2_list_add() - add an entry to a list
+ * @head: list head
+ * @entry: new list entry
+ */
+static inline void
+dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
+{
+	TAILQ_INSERT_TAIL(head, entry, node);
+}
+
+/**
+ * dlb2_list_del() - delete an entry from a list
+ * @entry: list entry
+ * @head: list head
+ */
+static inline void dlb2_list_del(struct dlb2_list_head *head,
+				 struct dlb2_list_entry *entry)
+{
+	TAILQ_REMOVE(head, entry, node);
+}
+
+/**
+ * dlb2_list_empty() - check if a list is empty
+ * @head: list head
+ *
+ * Return:
+ * Returns 1 if empty, 0 if not.
+ */
+static inline int dlb2_list_empty(struct dlb2_list_head *head)
+{
+	return TAILQ_EMPTY(head);
+}
+
+/**
+ * dlb2_list_splice() - splice a list
+ * @src_head: list to be added
+ * @ head: where src_head will be inserted
+ */
+static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
+				    struct dlb2_list_head *head)
+{
+	TAILQ_CONCAT(head, src_head, node);
+}
+
+/**
+ * DLB2_LIST_HEAD() - retrieve the head of the list
+ * @head: list head
+ * @type: type of the list variable
+ * @name: name of the list field within the containing struct
+ */
+#define DLB2_LIST_HEAD(head, type, name)                       \
+	(TAILQ_FIRST(&head) ?					\
+		container_of(TAILQ_FIRST(&head), type, name) :	\
+		NULL)
+
+/**
+ * DLB2_LIST_FOR_EACH() - iterate over a list
+ * @head: list head
+ * @ptr: pointer to struct containing a struct list
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ */
+#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
+	TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
+
+/**
+ * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
+ * an element is removed from the list while processing it.
+ * @ptr: pointer to struct containing a struct list
+ * @ptr_tmp: pointer to struct containing a struct list (temporary)
+ * @head: list head
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ * @iter_tmp: iterator variable (temporary)
+ */
+#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
+	TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
+
+#endif /*  __DLB2_OSDEP_LIST_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
new file mode 100644
index 0000000..0a48f7e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_TYPES_H
+#define __DLB2_OSDEP_TYPES_H
+
+#include <linux/types.h>
+
+#include <inttypes.h>
+#include <ctype.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Types for user mode PF PMD */
+typedef uint8_t         u8;
+typedef int8_t          s8;
+typedef uint16_t        u16;
+typedef int16_t         s16;
+typedef uint32_t        u32;
+typedef int32_t         s32;
+typedef uint64_t        u64;
+
+#define __iomem
+
+/* END types for user mode PF PMD */
+
+#endif /* __DLB2_OSDEP_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
new file mode 100644
index 0000000..43ecad4
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
@@ -0,0 +1,2527 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "dlb2_osdep_types.h"
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+	struct {
+		u32 isr_pend : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+	struct {
+		u32 vf0_reset_in_progress : 1;
+		u32 vf1_reset_in_progress : 1;
+		u32 vf2_reset_in_progress : 1;
+		u32 vf3_reset_in_progress : 1;
+		u32 vf4_reset_in_progress : 1;
+		u32 vf5_reset_in_progress : 1;
+		u32 vf6_reset_in_progress : 1;
+		u32 vf7_reset_in_progress : 1;
+		u32 vf8_reset_in_progress : 1;
+		u32 vf9_reset_in_progress : 1;
+		u32 vf10_reset_in_progress : 1;
+		u32 vf11_reset_in_progress : 1;
+		u32 vf12_reset_in_progress : 1;
+		u32 vf13_reset_in_progress : 1;
+		u32 vf14_reset_in_progress : 1;
+		u32 vf15_reset_in_progress : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+	struct {
+		u32 vec_mask : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+	struct {
+		u32 func_vf_bar_dis : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+	struct {
+		u32 total_vas : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+	struct {
+		u32 total_dir_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+	struct {
+		u32 total_ldb_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+	struct {
+		u32 total_dir_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+	struct {
+		u32 total_ldb_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+	struct {
+		u32 total_dir_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+	struct {
+		u32 total_ldb_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 meas : 1;
+		u32 debug : 7;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 cq_int_rearm : 1;
+		u32 dsi_error : 1;
+		u32 rsvd0 : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 rsvd0 : 3;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+	struct {
+		u32 qid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+	struct {
+		u32 vqid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+	struct {
+		u32 qid : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 debug : 8;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 isz : 1;
+		u32 dsi_error : 1;
+		u32 dlbrsvd : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 vf_synd0_parity : 1;
+		u32 vf_synd1_parity : 1;
+		u32 vf_synd2_parity : 1;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+	struct {
+		u32 sn_cfg_v : 1;
+		u32 fid_cfg_v : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS  0
+#define DLB2_CQ_ISR_MODE_MSI  1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI  3
+union dlb2_sys_ldb_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+	struct {
+		u32 keep_pf_ppid : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+	struct {
+		u32 illegal_hcw : 1;
+		u32 illegal_pp : 1;
+		u32 illegal_pasid : 1;
+		u32 illegal_qid : 1;
+		u32 disabled_qid : 1;
+		u32 illegal_ldb_qid_cfg : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+	struct {
+		u32 msix_0_ack : 1;
+		u32 msix_1_ack : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+	struct {
+		u32 msix_0_passthru : 1;
+		u32 msix_1_passthru : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED     0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+	struct {
+		u32 mode : 1;
+		u32 poll_mode : 1;
+		u32 poll_mask : 1;
+		u32 poll_lock : 1;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+	struct {
+		u32 cq : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 alarm : 1;
+		u32 cwd : 1;
+		u32 vf_pf_mb : 1;
+		u32 rsvd0 : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+	struct {
+		u32 qid_fid_limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+	struct {
+		u32 compress_code : 3;
+		u32 rsvd0 : 29;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+	(DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+	struct {
+		u32 count : 15;
+		u32 rsvd0 : 17;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+	struct {
+		u32 sn : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+	struct {
+		u32 mode : 3;
+		u32 slot : 4;
+		u32 rsvz0 : 1;
+		u32 grp : 1;
+		u32 rsvz1 : 1;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+	struct {
+		u32 en : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+	struct {
+		u32 write_pointer : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+	struct {
+		u32 base : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+	struct {
+		u32 limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+	struct {
+		u32 pop_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+	struct {
+		u32 push_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+	struct {
+		u32 depth : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+	struct {
+		u32 write_pointer : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 dlb_cor_alarm_enable : 1;
+		u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+		u32 cfg_64bytes_qe_dir_cq_mode : 1;
+		u32 pad_write_ldb : 1;
+		u32 pad_write_dir : 1;
+		u32 pad_first_write_ldb : 1;
+		u32 pad_first_write_dir : 1;
+		u32 rsvz0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+	struct {
+		u32 egress_credit_status_empty : 1;
+		u32 egress_credit_status_afull : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+		u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+		u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+		u32 chp_rop_pipe_credit_status_empty : 1;
+		u32 chp_rop_pipe_credit_status_afull : 1;
+		u32 qed_to_cq_pipe_credit_status_empty : 1;
+		u32 qed_to_cq_pipe_credit_status_afull : 1;
+		u32 egress_lsp_token_credit_status_empty : 1;
+		u32 egress_lsp_token_credit_status_afull : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 rsvz0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+	struct {
+		u32 sn_mode_0 : 3;
+		u32 rszv0 : 5;
+		u32 sn_mode_1 : 3;
+		u32 rszv1 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+	struct {
+		u32 unit_single_step_mode : 1;
+		u32 rr_en : 1;
+		u32 rszv0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+	struct {
+		u32 prio : 24;
+		u32 v : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+	struct {
+		u32 qid_p0 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p1 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p2 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p3 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+	struct {
+		u32 qid_p4 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p5 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p6 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p7 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+	struct {
+		u32 token_depth_select : 4;
+		u32 disable_wb_opt : 1;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+	struct {
+		u32 token_count : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+	struct {
+		u32 thresh : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+	(DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+	(DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+	struct {
+		u32 depth : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+	struct {
+		u32 cq : 8;
+		u32 qidix : 3;
+		u32 value : 1;
+		u32 nalb_haswork_v : 1;
+		u32 rlist_haswork_v : 1;
+		u32 slist_haswork_v : 1;
+		u32 inflight_ok_v : 1;
+		u32 aqed_nfull_v : 1;
+		u32 rsvz0 : 15;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+	struct {
+		u32 transfer : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+	struct {
+		u32 bw_range : 9;
+		u32 rsvz0 : 22;
+		u32 no_extra_credit : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+	struct {
+		u32 disab_atq_empty_arb : 1;
+		u32 inc_tok_unit_idle : 1;
+		u32 disab_rlist_pri : 1;
+		u32 inc_cmp_unit_idle : 1;
+		u32 rsvz0 : 2;
+		u32 dir_single_op : 1;
+		u32 dir_half_bw : 1;
+		u32 dir_single_out : 1;
+		u32 dir_disab_multi : 1;
+		u32 atq_single_op : 1;
+		u32 atq_half_bw : 1;
+		u32 atq_single_out : 1;
+		u32 atq_disab_multi : 1;
+		u32 dirrpl_single_op : 1;
+		u32 dirrpl_half_bw : 1;
+		u32 dirrpl_single_out : 1;
+		u32 lbrpl_single_op : 1;
+		u32 lbrpl_half_bw : 1;
+		u32 lbrpl_single_out : 1;
+		u32 ldb_single_op : 1;
+		u32 ldb_half_bw : 1;
+		u32 ldb_disab_multi : 1;
+		u32 atm_single_sch : 1;
+		u32 atm_single_cmp : 1;
+		u32 ldb_ce_tog_arb : 1;
+		u32 rsvz1 : 1;
+		u32 smon0_valid_sel : 2;
+		u32 smon0_value_sel : 1;
+		u32 smon0_compare_sel : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+	struct {
+		u32 chp_pf_reset_done : 1;
+		u32 rop_pf_reset_done : 1;
+		u32 lsp_pf_reset_done : 1;
+		u32 nalb_pf_reset_done : 1;
+		u32 ap_pf_reset_done : 1;
+		u32 dp_pf_reset_done : 1;
+		u32 qed_pf_reset_done : 1;
+		u32 dqed_pf_reset_done : 1;
+		u32 aqed_pf_reset_done : 1;
+		u32 sys_pf_reset_done : 1;
+		u32 pf_reset_active : 1;
+		u32 flrsm_state : 7;
+		u32 rsvd0 : 13;
+		u32 dlb_proc_reset_done : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+	struct {
+		u32 chp_pipeidle : 1;
+		u32 rop_pipeidle : 1;
+		u32 lsp_pipeidle : 1;
+		u32 nalb_pipeidle : 1;
+		u32 ap_pipeidle : 1;
+		u32 dp_pipeidle : 1;
+		u32 qed_pipeidle : 1;
+		u32 dqed_pipeidle : 1;
+		u32 aqed_pipeidle : 1;
+		u32 sys_pipeidle : 1;
+		u32 chp_unit_idle : 1;
+		u32 rop_unit_idle : 1;
+		u32 lsp_unit_idle : 1;
+		u32 nalb_unit_idle : 1;
+		u32 ap_unit_idle : 1;
+		u32 dp_unit_idle : 1;
+		u32 qed_unit_idle : 1;
+		u32 dqed_unit_idle : 1;
+		u32 aqed_unit_idle : 1;
+		u32 sys_unit_idle : 1;
+		u32 rsvd1 : 4;
+		u32 mstr_cfg_ring_idle : 1;
+		u32 mstr_cfg_mstr_idle : 1;
+		u32 mstr_flr_clkreq_b : 1;
+		u32 mstr_proc_idle : 1;
+		u32 mstr_proc_idle_masked : 1;
+		u32 rsvd0 : 2;
+		u32 dlb_func_idle : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+	struct {
+		u32 prochot : 1;
+		u32 pgcb_dlb_idle : 1;
+		u32 pgcb_dlb_pg_rdy_ack_b : 1;
+		u32 pmsm_pgcb_req_b : 1;
+		u32 pgbc_pmc_pg_req_b : 1;
+		u32 pmc_pgcb_pg_ack_b : 1;
+		u32 pmc_pgcb_fet_en_b : 1;
+		u32 pgcb_fet_en_b : 1;
+		u32 rsvz0 : 1;
+		u32 rsvz1 : 1;
+		u32 fuse_force_on : 1;
+		u32 fuse_proc_disable : 1;
+		u32 rsvz2 : 1;
+		u32 rsvz3 : 1;
+		u32 pm_fsm_d0tod3_ok : 1;
+		u32 pm_fsm_d3tod0_ok : 1;
+		u32 dlb_in_d3 : 1;
+		u32 rsvz4 : 7;
+		u32 pmsm : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+	struct {
+		u32 disable : 1;
+		u32 rsvz0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+	struct {
+		u32 isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+	struct {
+		u32 pf_isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+	struct {
+		u32 isr_pend : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+	struct {
+		u32 reset_in_progress : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+	struct {
+		u32 vf_msi_isr : 32;
+	} field;
+	u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
new file mode 100644
index 0000000..503fdf3
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
@@ -0,0 +1,1913 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_osdep_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_start_domain() - start a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: start domain arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int dlb2_hw_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp,
+			 bool vdev_request,
+			 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_request,
+		    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding unmap procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_finish_map_qid_procedures() - finish any pending map procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding map procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ *					interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+			  int port_id,
+			  bool is_ldb,
+			  bool vdev_request,
+			  unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+					 u32 *ldb_interrupts,
+					 u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+				 u32 *ldb_interrupts,
+				 u32 *dir_interrupts);
+
+/**
+ * dlb2_read_vf_intr_status() - read the VF interrupt status register
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function can be called from a VF's interrupt handler to determine
+ * which interrupts have fired. The first 31 bits correspond to CQ interrupt
+ * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
+ *
+ * Return:
+ * Returns a bit vector indicating which interrupt vectors are active.
+ */
+u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_intr_status() - ack VF interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function ACKs a VF's interrupts. Its interrupts argument should be the
+ * value returned by dlb2_read_vf_intr_status().
+ */
+void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function clears the VF's MSI interrupt pending register. Its interrupts
+ * argument should be contain the MSI vectors to ACK. For example, if MSI MME
+ * is in mode 0, then one bit 0 should ever be set.
+ */
+void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When done processing the PF mailbox request, this function unsets
+ * the PF's mailbox ISR register.
+ */
+void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
+
+/**
+ * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
+ *					vdevs
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the vdev->PF ISR fires, this function can be called to determine which
+ * vdev(s) are requesting service. This bitvector must be passed to
+ * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
+ * vdevs.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested service.
+ */
+u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ *
+ * When done processing all VF mailbox requests, this function unsets the VF's
+ * mailbox ISR register.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
+ *				    FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the VF FLR ISR fires, this function can be called to determine which
+ * VF(s) are requesting FLRs. This bitvector must passed to
+ * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
+ */
+u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF FLR requests, this function unsets the VF's FLR
+ * ISR register.
+ */
+void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF requests, this function communicates to the
+ * hardware that processing is complete.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
+			     u32 mbox_bitvec,
+			     u32 flr_bitvec);
+
+/**
+ * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the watchdog timer interrupt cause registers to
+ * determine which port(s) had a watchdog timeout, and notifies the
+ * application(s) that own the port(s).
+ */
+void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads and logs the alarm syndrome, then acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_request,
+			      unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of resources in use by the PF or a VF. It
+ * fills in the fields that args points to, except the following:
+ * - max_contiguous_atomic_inflights
+ * - max_contiguous_hist_list_entries
+ * - max_contiguous_ldb_credits
+ * - max_contiguous_dir_credits
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
+				   struct dlb2_get_num_resources_args *arg,
+				   bool vdev_request,
+				   unsigned int vdev_id);
+
+/**
+ * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
+ *				       the PF
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function sends a VF->PF mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the PF has processed
+ * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
+ * when the PF has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
+ *				 request
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * processing a VF->PF mailbox request. It should only be called after sending
+ * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vf_flr_complete() - check the status of a VF FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * executing the VF FLR. It should only be called after setting the VF's FLR
+ * bit.
+ */
+bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
+ *					vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function sends a PF->vdev mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the vdev has
+ * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
+ * determine when the vdev has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
+ *			       request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function returns a boolean indicating whether the vdev has finished
+ * processing a PF->vdev mailbox request. It should only be called after
+ * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
+			     unsigned int vf_id,
+			     void *data,
+			     int len);
+
+/**
+ * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
+			       unsigned int vf_id,
+			       void *data,
+			       int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_reset_vdev() - reset the hardware owned by a virtual device
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function resets the hardware owned by a vdev, by resetting the vdev's
+ * domains one by one.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function returns whether or not the vdev's resource assignments are
+ * locked. If locked, no resources can be added to or subtracted from the
+ * group.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_lock_vdev() - lock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sets a flag indicating that the vdev is using its resources.
+ * When the vdev is locked, its resource assignment cannot be changed.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_unlock_vdev() - unlock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function unlocks the vdev's resource assignment, allowing it to be
+ * modified.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of scheduling domains to assign to this vdev
+ *
+ * This function assigns num scheduling domains to the specified vdev. If the
+ * vdev already has domains assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB queues to assign to this vdev
+ *
+ * This function assigns num LDB queues to the specified vdev. If the vdev
+ * already has LDB queues assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports to the specified vdev. If the vdev
+ * already has LDB ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @cos: class-of-service ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports from class-of-service cos to the
+ * specified vdev. If the vdev already has LDB ports from this class-of-service
+ * assigned, this existing assignment is adjusted accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
+				   u32 id,
+				   u32 cos,
+				   u32 num);
+
+/**
+ * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR ports to assign to this vdev
+ *
+ * This function assigns num DIR ports to the specified vdev. If the vdev
+ * already has DIR ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB credit credits to assign to this vdev
+ *
+ * This function assigns num LDB credit to the specified vdev. If the vdev
+ * already has LDB credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR credits to assign to this vdev
+ *
+ * This function assigns num DIR credit to the specified vdev. If the vdev
+ * already has DIR credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of history list entries to assign to this vdev
+ *
+ * This function assigns num history list entries to the specified vdev. If the
+ * vdev already has history list entries assigned, this existing assignment is
+ * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
+ * this function may fail if a sufficiently large contiguous chunk is not
+ * available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of atomic inflights to assign to this vdev
+ *
+ * This function assigns num atomic inflights to the specified vdev. If the vdev
+ * already has atomic inflights assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of entries, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function takes any resources currently assigned to the vdev and
+ * reassigns them to the PF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_notify_vf() - send an alarm to a VF
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID
+ * @notification: notification
+ *
+ * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
+ *
+ * Return:
+ * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
+ */
+int dlb2_notify_vf(struct dlb2_hw *hw,
+		   unsigned int vf_id,
+		   u32 notification);
+
+/**
+ * dlb2_vdev_in_use() - query whether a virtual device is in use
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sends a mailbox request to the vdev to query whether the vdev
+ * is in use.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
+ * an internal error occurs.
+ */
+int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+enum dlb2_virt_mode {
+	DLB2_VIRT_NONE,
+	DLB2_VIRT_SRIOV,
+	DLB2_VIRT_SIOV,
+
+	/* NUM_DLB2_VIRT_MODES must be last */
+	NUM_DLB2_VIRT_MODES,
+};
+
+/**
+ * dlb2_hw_set_virt_mode() - set the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: either none, SR-IOV, or Scalable IOV.
+ *
+ * This function sets the virtualization mode of the device. This controls
+ * whether the device uses a software or hardware mailbox.
+ *
+ * This should be called by the PF driver when either SR-IOV or Scalable IOV is
+ * selected as the virtualization mechanism, and by the VF/VDEV driver during
+ * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
+ *
+ * Errors:
+ * EINVAL - Invalid mode.
+ */
+int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
+
+/**
+ * dlb2_hw_get_virt_mode() - get the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function gets the virtualization mode of the device.
+ */
+enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_register_sw_mbox() - register a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
+ * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
+ * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
+ * @inject_arg: user argument for pf2vdev_inject callback.
+ *
+ * When Scalable IOV is enabled, the VDCM must register a software mailbox for
+ * every virtual device during vdev creation.
+ *
+ * This function notifies the driver to use a software mailbox using the
+ * provided pointers, instead of the device's hardware mailbox. When the driver
+ * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
+ * go to the software mailbox instead of the hardware one. This is used in
+ * Scalable IOV virtualization.
+ */
+void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
+			      unsigned int vdev_id,
+			      u32 *vdev2pf_mbox,
+			      u32 *pf2vdev_mbox,
+			      void (*pf2vdev_inject)(void *),
+			      void *inject_arg);
+
+/**
+ * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function notifies the driver to stop using a previously registered
+ * software mailbox.
+ */
+void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ * @addr_lo: least-significant 32 bits of address.
+ * @data: 32 data bits.
+ *
+ * This sets up the CQ's IMS entry with the provided address and data values.
+ * This function should only be called if the device is configured for Scalable
+ * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
+ * not needed.
+ */
+void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb,
+				u32 addr_lo,
+				u32 data);
+
+/**
+ * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ *
+ * This clears the CQ's IMS entry, reverting it to its reset state.
+ */
+void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb);
+
+/**
+ * dlb2_hw_register_pasid() - register a vdev's PASID
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @pasid: the vdev's PASID.
+ *
+ * This function stores the user-supplied PASID, and uses it when configuring
+ * the vdev's CQs.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+int dlb2_hw_register_pasid(struct dlb2_hw *hw,
+			   unsigned int vdev_id,
+			   unsigned int pasid);
+
+/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vf_request,
+				unsigned int vf_id);
+
+/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ *	to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ *	port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ *	    cause the total bandwidth across all classes of service to exceed
+ *	    100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+enum dlb2_wd_tmo {
+	/* 40s watchdog timeout */
+	DLB2_WD_TMO_40S,
+	/* 10s watchdog timeout */
+	DLB2_WD_TMO_10S,
+	/* 1s watchdog timeout */
+	DLB2_WD_TMO_1S,
+
+	/* Must be last */
+	NUM_DLB2_WD_TMOS,
+};
+
+/**
+ * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
+ *	caller-specified timeout.
+ * @hw: dlb2_hw handle for a particular device.
+ * @tmo: watchdog timeout.
+ *
+ * This function should be called during device initialization and after reset.
+ * The watchdog timer interrupt must also be enabled per-CQ, using either
+ * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid timeout.
+ */
+int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
+
+/**
+ * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid directed port ID.
+ */
+int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid load-balanced port ID.
+ */
+int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *	ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * load-balanced CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_ldb_cq_interrupt().
+ */
+void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical directed port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * directed CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_dir_cq_interrupt().
+ */
+void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+#endif /* __DLB2_RESOURCE_H */
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
new file mode 100644
index 0000000..3c51cab
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -0,0 +1,621 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "base/dlb2_resource.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_regs.h"
+#include "dlb2_main.h"
+#include "../dlb2_user.h"
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+
+#define PF_ID_ZERO 0	/* PF ONLY! */
+#define NO_OWNER_VF 0	/* PF ONLY! */
+#define NOT_VF_REQ false /* PF ONLY! */
+
+#define DLB2_PCI_CFG_SPACE_SIZE 256
+#define DLB2_PCI_CAP_POINTER 0x34
+#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
+#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
+#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
+#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
+#define DLB2_PCI_EXT_CAP_ID_ERR 1
+#define DLB2_PCI_ERR_UNCOR_MASK 8
+#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
+
+#define DLB2_PCI_EXP_DEVCTL 8
+#define DLB2_PCI_LNKCTL 16
+#define DLB2_PCI_SLTCTL 24
+#define DLB2_PCI_RTCTL 28
+#define DLB2_PCI_EXP_DEVCTL2 40
+#define DLB2_PCI_LNKCTL2 48
+#define DLB2_PCI_SLTCTL2 56
+#define DLB2_PCI_CMD 4
+#define DLB2_PCI_X_CMD 2
+#define DLB2_PCI_EXP_DEVSTA 10
+#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
+#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
+
+#define DLB2_PCI_CAP_ID_EXP       0x10
+#define DLB2_PCI_CAP_ID_MSIX      0x11
+#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
+#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
+#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
+
+#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
+#define DLB2_PCI_PRI_ALLOC_REQ           0xC
+#define DLB2_PCI_PRI_CTRL                0x4
+#define DLB2_PCI_MSIX_FLAGS              0x2
+#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
+#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
+#define DLB2_PCI_ERR_ROOT_STATUS         0x30
+#define DLB2_PCI_ERR_COR_STATUS          0x10
+#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
+#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
+#define DLB2_PCI_ACS_CAP                 0x4
+#define DLB2_PCI_ACS_CTRL                0x6
+#define DLB2_PCI_ACS_SV                  0x1
+#define DLB2_PCI_ACS_RR                  0x4
+#define DLB2_PCI_ACS_CR                  0x8
+#define DLB2_PCI_ACS_UF                  0x10
+#define DLB2_PCI_ACS_EC                  0x20
+
+/* Stubs: Allow building partial probe patch */
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	int ret = 0;
+	RTE_SET_USED(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+/* End stubs */
+
+static int
+dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint32_t hdr;
+	size_t sz;
+	int pos;
+
+	pos = DLB2_PCI_CFG_SPACE_SIZE;
+	sz = sizeof(hdr);
+
+	while (pos > 0xFF) {
+		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
+			return -1;
+
+		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
+			return pos;
+
+		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint8_t pos;
+	int ret;
+	uint16_t hdr;
+
+	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
+	pos &= 0xFC;
+
+	if (ret != 1)
+		return -1;
+
+	while (pos > 0x3F) {
+		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
+		if (ret != 2)
+			return -1;
+
+		if (DLB2_PCI_CAP_ID(hdr) == id)
+			return pos;
+
+		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
+			return -1;
+
+		pos = DLB2_PCI_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+	rte_spinlock_init(&dlb2_dev->resource_mutex);
+	rte_spinlock_init(&dlb2_dev->measurement_lock);
+
+	return 0;
+}
+
+static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
+{
+	u32 retries = 0;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
+		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+		union dlb2_cfg_mstr_cfg_pm_status pm_st;
+		u32 addr;
+
+		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+			break;
+
+		rte_delay_ms(1);
+	};
+
+	if (retries == DLB2_READY_RETRY_LIMIT) {
+		printf("[%s()] wait for device ready timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+struct dlb2_dev *
+dlb2_probe(struct rte_pci_device *pdev)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret = 0;
+
+	DLB2_INFO(dlb2_dev, "probe\n");
+
+	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
+			      RTE_CACHE_LINE_SIZE);
+
+	if (dlb2_dev == NULL) {
+		ret = -ENOMEM;
+		goto dlb2_dev_malloc_fail;
+	}
+
+	/* PCI Bus driver has already mapped bar space into process.
+	 * Save off our IO register and FUNC addresses.
+	 */
+
+	/* BAR 0 */
+	if (pdev->mem_resource[0].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
+	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.func_kva,
+		  (void *)dlb2_dev->hw.func_phys_addr,
+		  (void *)(pdev->mem_resource[0].len));
+
+	/* BAR 2 */
+	if (pdev->mem_resource[2].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
+	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.csr_kva,
+		  (void *)dlb2_dev->hw.csr_phys_addr,
+		  (void *)(pdev->mem_resource[2].len));
+
+	dlb2_dev->pdev = pdev;
+
+	/* PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb2_pf_enable_pm(dlb2_dev);
+
+	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb2_pf_reset(dlb2_dev);
+	if (ret)
+		goto dlb2_reset_fail;
+
+	ret = dlb2_pf_init_driver_state(dlb2_dev);
+	if (ret)
+		goto init_driver_state_fail;
+
+	ret = dlb2_resource_init(&dlb2_dev->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	return dlb2_dev;
+
+resource_init_fail:
+	dlb2_resource_free(&dlb2_dev->hw);
+init_driver_state_fail:
+dlb2_reset_fail:
+pci_mmap_bad_addr:
+wait_for_device_ready_fail:
+	rte_free(dlb2_dev);
+dlb2_dev_malloc_fail:
+	rte_errno = ret;
+	return NULL;
+}
+
+int
+dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
+{
+	int ret = 0;
+	int i = 0;
+	uint32_t dword[16];
+	uint16_t cmd;
+	off_t off;
+
+	uint16_t dev_ctl_word;
+	uint16_t dev_ctl2_word;
+	uint16_t lnk_word;
+	uint16_t lnk_word2;
+	uint16_t slt_word;
+	uint16_t slt_word2;
+	uint16_t rt_ctl_word;
+	uint32_t pri_reqs_dword;
+	uint16_t pri_ctrl_word;
+
+	int pcie_cap_offset;
+	int pri_cap_offset;
+	int msix_cap_offset;
+	int err_cap_offset;
+	int acs_cap_offset;
+	int wait_count;
+
+	uint16_t devsta_busy_word;
+	uint16_t devctl_word;
+
+	struct rte_pci_device *pdev = dlb2_dev->pdev;
+
+	/* Save PCI config state */
+
+	for (i = 0; i < 16; i++) {
+		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
+			return ret;
+	}
+
+	pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
+
+	if (pcie_cap_offset < 0) {
+		printf("[%s()] failed to find the pcie capability\n",
+		       __func__);
+		return pcie_cap_offset;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
+		dev_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
+		lnk_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
+		slt_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_RTCTL;
+	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
+		rt_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
+		dev_ctl2_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
+		lnk_word2 = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
+		slt_word2 = 0;
+
+	off = DLB2_PCI_EXT_CAP_ID_PRI;
+	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (pri_cap_offset >= 0) {
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
+			pri_reqs_dword = 0;
+	}
+
+	/* clear the PCI command register before issuing the FLR */
+
+	off = DLB2_PCI_CMD;
+	cmd = 0;
+	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+		printf("[%s()] failed to write the pci command\n",
+		       __func__);
+		return ret;
+	}
+
+	/* issue the FLR */
+	for (wait_count = 0; wait_count < 4; wait_count++) {
+		int sleep_time;
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
+		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to read the pci device status\n",
+			       __func__);
+			return ret;
+		}
+
+		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
+			break;
+
+		sleep_time = (1 << (wait_count)) * 100;
+		rte_delay_ms(sleep_time);
+	}
+
+	if (wait_count == 4) {
+		printf("[%s()] wait for pci pending transactions timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to read the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
+
+	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to write the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	rte_delay_ms(100);
+
+	/* Restore PCI config state */
+
+	if (pcie_cap_offset >= 0) {
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie device control at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_RTCTL;
+		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	if (pri_cap_offset >= 0) {
+		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
+
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
+		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ERR;
+	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (err_cap_offset >= 0) {
+		uint32_t tmp;
+
+		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	for (i = 16; i > 0; i--) {
+		off = (i - 1) * 4;
+		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_CMD;
+	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
+		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+			printf("[%s()] failed to write the pci command\n",
+			       __func__);
+			return ret;
+		}
+	}
+
+	msix_cap_offset = dlb2_pci_find_capability(pdev,
+						   DLB2_PCI_CAP_ID_MSIX);
+	if (msix_cap_offset >= 0) {
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
+			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ACS;
+	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (acs_cap_offset >= 0) {
+		uint16_t acs_cap, acs_ctrl, acs_mask;
+		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
+		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
+			acs_cap = 0;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
+		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
+		acs_ctrl |= (acs_cap & acs_mask);
+
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
+		acs_mask |= DLB2_PCI_ACS_EC;
+		acs_ctrl &= ~acs_mask;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
new file mode 100644
index 0000000..f082a94
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -0,0 +1,98 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+#endif
+
+#include "base/dlb2_hw_types.h"
+#include "../dlb2_user.h"
+
+#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
+
+struct dlb2_dev;
+
+struct dlb2_port_memory {
+	struct dlb2_list_head list;
+	void *cq_base;
+	bool valid;
+};
+
+struct dlb2_dev {
+	struct rte_pci_device *pdev;
+	struct dlb2_hw hw;
+	/* struct list_head list; */
+	struct device *dlb2_device;
+	bool domain_reset_failed;
+	/* The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	rte_spinlock_t resource_mutex;
+	rte_spinlock_t measurement_lock;
+	bool worker_launched;
+	u8 revision;
+};
+
+struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
+int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
+int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+
+#endif /* __DLB2_MAIN_H */
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
new file mode 100644
index 0000000..ae4d785
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -0,0 +1,196 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_devargs.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_pci.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+#include "dlb2_main.h"
+#include "base/dlb2_hw_types.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_resource.h"
+
+static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
+
+/* Stubs: Allow building partial probe patch */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(arg);
+	RTE_SET_USED(vdev_req);
+	RTE_SET_USED(vdev_id);
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+/* End stubs */
+
+static void
+dlb2_pf_iface_fn_ptrs_init(void)
+{
+/* flexible iface fn ptr assignments will go here */
+}
+
+/* PCI DEV HOOKS */
+static int
+dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
+{
+	int ret = 0;
+	struct rte_pci_device *pci_dev;
+	struct dlb2_devargs dlb2_args = {
+		.socket_id = rte_socket_id(),
+		.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+		.num_dir_credits_override = -1,
+		.qid_depth_thresholds = { {0} },
+		.cos_id = DLB2_COS_DEFAULT
+	};
+	struct dlb2_eventdev *dlb2;
+
+	DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
+		     eventdev->data->dev_id, eventdev->data->socket_id);
+
+	dlb2_pf_iface_fn_ptrs_init();
+
+	pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
+
+		/* Probe the DLB2 PF layer */
+		dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
+
+		if (dlb2->qm_instance.pf_dev == NULL) {
+			DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
+				     rte_errno);
+			ret = -rte_errno;
+			goto dlb2_probe_failed;
+		}
+
+		/* Were we invoked with runtime parameters? */
+		if (pci_dev->device.devargs) {
+			ret = dlb2_parse_params(pci_dev->device.devargs->args,
+						pci_dev->device.devargs->name,
+						&dlb2_args);
+			if (ret) {
+				DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
+					     ret, rte_errno);
+				goto dlb2_probe_failed;
+			}
+		}
+
+		ret = dlb2_primary_eventdev_probe(eventdev,
+						  event_dlb2_pf_name,
+						  &dlb2_args);
+	} else {
+		ret = dlb2_secondary_eventdev_probe(eventdev,
+						    event_dlb2_pf_name);
+	}
+	if (ret)
+		goto dlb2_probe_failed;
+
+	DLB2_LOG_INFO("DLB2 PF Probe success\n");
+
+	return 0;
+
+dlb2_probe_failed:
+
+	DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
+
+	return ret;
+}
+
+#define EVENTDEV_INTEL_VENDOR_ID 0x8086
+
+static const struct rte_pci_id pci_id_dlb2_map[] = {
+	{
+		RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
+			       PCI_DEVICE_ID_INTEL_DLB2_PF)
+	},
+	{
+		.vendor_id = 0,
+	},
+};
+
+static int
+event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
+		     struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
+					     sizeof(struct dlb2_eventdev),
+					     dlb2_eventdev_pci_init,
+					     event_dlb2_pf_name);
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+}
+
+static int
+event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_remove(pci_dev, NULL);
+
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+
+}
+
+static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
+	.id_table = pci_id_dlb2_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe = event_dlb2_pci_probe,
+	.remove = event_dlb2_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
-- 
2.6.4

[dpdk-dev] [PATCH v9 07/23] event/dlb2: add flexible interface

[Timothy McDaniel]

This commit introduces the flexible interface. This
interface allows the core code to operate in PF mode (direct
hardware access) or bifurcated mode (hardware configured via
kernel driver). This driver currently only supports PF mode
but bifurcated mode will be added in a future DPDK patch-set.
Note that the flexible interface is not used for data path
operations, and thus there are no performance concerns
related to the use of function pointers.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_iface.c | 28 ++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.h | 29 +++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build  |  1 +
 3 files changed, 58 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h

diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
new file mode 100644
index 0000000..0fc9991
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+
+#include "dlb2_priv.h"
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+void (*dlb2_iface_low_level_io_init)(void);
+
+int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+				     uint8_t *revision);
+
+void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+				   enum dlb2_cq_poll_modes *mode);
+
+int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
new file mode 100644
index 0000000..4fb416e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_IFACE_H_
+#define _DLB2_IFACE_H_
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+extern void (*dlb2_iface_low_level_io_init)(void);
+
+extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+					    uint8_t *revision);
+
+extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+					  enum dlb2_cq_poll_modes *mode);
+
+extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
+
+#endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 1eab9b9..491e76d 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -8,6 +8,7 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
 endif
 
 sources = files('dlb2.c',
+		'dlb2_iface.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c'
 )
-- 
2.6.4

[dpdk-dev] [PATCH v9 08/23] event/dlb2: add probe-time hardware init

[Timothy McDaniel]

This commit adds probe-time low level hardware
initialization.  It also adds probe-time init for both
primary and secondary DPDK processes.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 188 +++++++++++++++++++-
 drivers/event/dlb2/meson.build             |   3 +-
 drivers/event/dlb2/pf/base/dlb2_resource.c | 274 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  23 +--
 drivers/event/dlb2/pf/dlb2_main.h          |   1 -
 drivers/event/dlb2/pf/dlb2_pf.c            |  78 ++++++--
 6 files changed, 523 insertions(+), 44 deletions(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 1b11979..16653db 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -31,6 +31,7 @@
 #include <rte_string_fns.h>
 
 #include "dlb2_priv.h"
+#include "dlb2_iface.h"
 #include "dlb2_inline_fns.h"
 
 /*
@@ -40,10 +41,108 @@
 #if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
 #error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
 #endif
+static struct rte_event_dev_info evdev_dlb2_default_info = {
+	.driver_name = "", /* probe will set */
+	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
+	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
+	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
+#else
+	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
+#endif
+	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
+	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
+	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
+	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
+	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
+	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+	.max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
+	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
+			  RTE_EVENT_DEV_CAP_EVENT_QOS |
+			  RTE_EVENT_DEV_CAP_BURST_MODE |
+			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
+			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
+			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
+};
 
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/* override defaults with value(s) provided on command line */
+static void
+dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
+				 int *qid_depth_thresholds)
+{
+	int q;
+
+	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
+		if (qid_depth_thresholds[q] != 0)
+			dlb2->ev_queues[q].depth_threshold =
+				qid_depth_thresholds[q];
+	}
+}
+
+static int
+dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
+	int ret;
+
+	/* Query driver resources provisioned for this device */
+
+	ret = dlb2_iface_get_num_resources(handle,
+					   &dlb2->hw_rsrc_query_results);
+	if (ret) {
+		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
+		return ret;
+	}
+
+	/* Complete filling in device resource info returned to evdev app,
+	 * overriding any default values.
+	 * The capabilities (CAPs) were set at compile time.
+	 */
+
+	evdev_dlb2_default_info.max_event_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	evdev_dlb2_default_info.max_event_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	evdev_dlb2_default_info.max_num_events =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	/* Save off values used when creating the scheduling domain. */
+
+	handle->info.num_sched_domains =
+		dlb2->hw_rsrc_query_results.num_sched_domains;
+
+	handle->info.hw_rsrc_max.nb_events_limit =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	handle->info.hw_rsrc_max.num_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues +
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.num_ldb_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	handle->info.hw_rsrc_max.num_ldb_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	handle->info.hw_rsrc_max.num_dir_ports =
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.reorder_window_size =
+		dlb2->hw_rsrc_query_results.num_hist_list_entries;
+
+	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
+
+	return 0;
+}
+
 #define DLB2_BASE_10 10
 
 static int
@@ -235,14 +334,71 @@ set_qid_depth_thresh(const char *key __rte_unused,
 	return 0;
 }
 
+static void
+dlb2_entry_points_init(struct rte_eventdev *dev)
+{
+	RTE_SET_USED(dev);
+
+	/* Eventdev PMD entry points */
+}
+
 int
 dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    const char *name,
 			    struct dlb2_devargs *dlb2_args)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
-	RTE_SET_USED(dlb2_args);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	dlb2->event_dev = dev; /* backlink */
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	dlb2->max_num_events_override = dlb2_args->max_num_events;
+	dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
+	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
+					    &dlb2->revision);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_hardware_init(&dlb2->qm_instance);
+
+	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
+			     err);
+		return err;
+	}
+
+	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	dlb2_init_queue_depth_thresholds(dlb2,
+					 dlb2_args->qid_depth_thresholds.val);
 
 	return 0;
 }
@@ -251,8 +407,30 @@ int
 dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
 			      const char *name)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 491e76d..28fef3f 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -10,7 +10,8 @@ endif
 sources = files('dlb2.c',
 		'dlb2_iface.c',
 		'pf/dlb2_main.c',
-		'pf/dlb2_pf.c'
+		'pf/dlb2_pf.c',
+		'pf/base/dlb2_resource.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
new file mode 100644
index 0000000..6de8b95
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_mbox.h"
+#include "dlb2_osdep.h"
+#include "dlb2_osdep_bitmap.h"
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+	int i;
+
+	dlb2_list_init_head(&domain->used_ldb_queues);
+	dlb2_list_init_head(&domain->used_dir_pq_pairs);
+	dlb2_list_init_head(&domain->avail_ldb_queues);
+	dlb2_list_init_head(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+	int i;
+
+	dlb2_list_init_head(&rsrc->avail_domains);
+	dlb2_list_init_head(&rsrc->used_domains);
+	dlb2_list_init_head(&rsrc->avail_ldb_queues);
+	dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
+	arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
+	arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
+	arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb2_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	struct dlb2_list_entry *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. This is application
+	 * dependent, but the driver interleaves port IDs as much as possible
+	 * to reduce the likelihood of this. This initial allocation maximizes
+	 * the average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	/* Zero-out resource tracking data structures */
+	memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+	memset(&hw->pf, 0, sizeof(hw->pf));
+
+	dlb2_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+		dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+		dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_domains, list);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_ldb_queues, list);
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+		struct dlb2_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
+			      &port->func_list);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+				DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+	if (ret)
+		goto unwind;
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+					DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+		if (ret)
+			goto unwind;
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb2_resource_free(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+	r0.field.disable = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 3c51cab..210b3186 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -19,6 +19,7 @@
 #include "dlb2_main.h"
 #include "../dlb2_user.h"
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 
 #define PF_ID_ZERO 0	/* PF ONLY! */
@@ -72,27 +73,6 @@
 #define DLB2_PCI_ACS_UF                  0x10
 #define DLB2_PCI_ACS_EC                  0x20
 
-/* Stubs: Allow building partial probe patch */
-void dlb2_resource_free(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-int dlb2_resource_init(struct dlb2_hw *hw)
-{
-	int ret = 0;
-	RTE_SET_USED(hw);
-
-	return ret;
-}
-
-void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-/* End stubs */
-
 static int
 dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
 {
@@ -149,7 +129,6 @@ static int
 dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
 {
 	rte_spinlock_init(&dlb2_dev->resource_mutex);
-	rte_spinlock_init(&dlb2_dev->measurement_lock);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index f082a94..f3bee71 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -38,7 +38,6 @@ struct dlb2_dev {
 	 * hardware registers.
 	 */
 	rte_spinlock_t resource_mutex;
-	rte_spinlock_t measurement_lock;
 	bool worker_launched;
 	u8 revision;
 };
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index ae4d785..7d64309 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -32,6 +32,7 @@
 #include <rte_string_fns.h>
 
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 #include "dlb2_main.h"
 #include "base/dlb2_hw_types.h"
@@ -40,35 +41,82 @@
 
 static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
 
-/* Stubs: Allow building partial probe patch */
-int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
-			      struct dlb2_get_num_resources_args *arg,
-			      bool vdev_req,
-			      unsigned int vdev_id)
+static void
+dlb2_pf_low_level_io_init(void)
+{
+	int i;
+	/* Addresses will be initialized at port create */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		/* First directed ports */
+		dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
+
+		/* Now load balanced ports */
+		dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
+	}
+}
+
+static int
+dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
+{
+	RTE_SET_USED(handle);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+static int
+dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
+			   uint8_t *revision)
 {
-	RTE_SET_USED(hw);
-	RTE_SET_USED(arg);
-	RTE_SET_USED(vdev_req);
-	RTE_SET_USED(vdev_id);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	*revision = dlb2_dev->revision;
 
 	return 0;
 }
 
-void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+static void
+dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+	dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_num_resources_args *rsrcs)
 {
-	RTE_SET_USED(hw);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
 }
 
-void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+static int
+dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
+			 enum dlb2_cq_poll_modes *mode)
 {
-	RTE_SET_USED(hw);
+	RTE_SET_USED(handle);
+
+	*mode = DLB2_CQ_POLL_MODE_SPARSE;
+
+	return 0;
 }
-/* End stubs */
 
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
-/* flexible iface fn ptr assignments will go here */
+	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
+	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
+	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
+	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
+	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v9 09/23] event/dlb2: add xstats

[Timothy McDaniel]

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
Reviewed-by: Chen, Mike Ximing <mike.ximing.chen@intel.com>
---
 drivers/event/dlb2/dlb2.c        |   35 +-
 drivers/event/dlb2/dlb2_xstats.c | 1240 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build   |    1 +
 3 files changed, 1273 insertions(+), 3 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 16653db..925824e 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,6 +70,21 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/*
+ * DUMMY - added so that xstats path will compile/link.
+ * Will be replaced by real version in a subsequent
+ * patch.
+ */
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	RTE_SET_USED(dlb2);
+	RTE_SET_USED(queue);
+
+	return 0;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -337,9 +352,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
-	RTE_SET_USED(dev);
-
-	/* Eventdev PMD entry points */
+	/* Expose PMD's eventdev interface */
+	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dump             = dlb2_eventdev_dump,
+		.xstats_get       = dlb2_eventdev_xstats_get,
+		.xstats_get_names = dlb2_eventdev_xstats_get_names,
+		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
+		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+	};
+
+	dev->dev_ops = &dlb2_eventdev_entry_ops;
 }
 
 int
@@ -391,6 +413,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Complete xtstats runtime initialization */
+	err = dlb2_xstats_init(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
+		return err;
+	}
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
new file mode 100644
index 0000000..21391f7
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_xstats.c
@@ -0,0 +1,1240 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eventdev.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+enum dlb2_xstats_type {
+	/* common to device and port */
+	rx_ok,				/**< Receive an event */
+	rx_drop,                        /**< Error bit set in received QE */
+	rx_interrupt_wait,		/**< Wait on an interrupt */
+	rx_umonitor_umwait,		/**< Block using umwait */
+	tx_ok,				/**< Transmit an event */
+	total_polls,			/**< Call dequeue_burst */
+	zero_polls,			/**< Call dequeue burst and return 0 */
+	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
+	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
+	tx_nospc_inflight_max,		/**< Reach the new_event_threshold */
+	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
+	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
+	/* device specific */
+	nb_events_limit,
+	inflight_events,
+	ldb_pool_size,
+	dir_pool_size,
+	/* port specific */
+	tx_new,				/**< Send an OP_NEW event */
+	tx_fwd,				/**< Send an OP_FORWARD event */
+	tx_rel,				/**< Send an OP_RELEASE event */
+	tx_implicit_rel,		/**< Issue an implicit event release */
+	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED event */
+	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL event */
+	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event */
+	tx_sched_directed,		/**< Send a directed event */
+	tx_invalid,                     /**< Send an event with an invalid op */
+	outstanding_releases,		/**< # of releases a port owes */
+	max_outstanding_releases,	/**< max # of releases a port can owe */
+	rx_sched_ordered,		/**< Dequeue an ordered event */
+	rx_sched_unordered,		/**< Dequeue an unordered event */
+	rx_sched_atomic,		/**< Dequeue an atomic event */
+	rx_sched_directed,		/**< Dequeue an directed event */
+	rx_sched_invalid,               /**< Dequeue event sched type invalid */
+	/* common to port and queue */
+	is_configured,			/**< Port is configured */
+	is_load_balanced,		/**< Port is LDB */
+	hw_id,				/**< Hardware ID */
+	/* queue specific */
+	num_links,			/**< Number of ports linked */
+	sched_type,			/**< Queue sched type */
+	enq_ok,				/**< # events enqueued to the queue */
+	current_depth,			/**< Current queue depth */
+	depth_threshold,		/**< Programmed depth threshold */
+	depth_le50_threshold,
+	/**< Depth LE to 50% of the configured hardware threshold */
+	depth_gt50_le75_threshold,
+	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold */
+	depth_gt75_le100_threshold,
+	/**< Depth GT 75%. but LE to the configured hardware threshold */
+	depth_gt100_threshold
+	/**< Depth GT 100% of the configured hw threshold */
+};
+
+typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
+		uint16_t obj_idx, /* port or queue id */
+		enum dlb2_xstats_type stat, int extra_arg);
+
+enum dlb2_xstats_fn_type {
+	DLB2_XSTATS_FN_DEV,
+	DLB2_XSTATS_FN_PORT,
+	DLB2_XSTATS_FN_QUEUE
+};
+
+struct dlb2_xstats_entry {
+	struct rte_event_dev_xstats_name name;
+	uint64_t reset_value; /* an offset to be taken away to emulate resets */
+	enum dlb2_xstats_fn_type fn_id;
+	enum dlb2_xstats_type stat;
+	enum rte_event_dev_xstats_mode mode;
+	int extra_arg;
+	uint16_t obj_idx;
+	uint8_t reset_allowed; /* when set, this value can be reset */
+};
+
+/* Some device stats are simply a summation of the corresponding port values */
+static uint64_t
+dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
+			     int which_stat)
+{
+	int i;
+	uint64_t val = 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
+
+		if (!port->setup_done)
+			continue;
+
+		switch (which_stat) {
+		case rx_ok:
+			val += port->stats.traffic.rx_ok;
+			break;
+		case rx_drop:
+			val += port->stats.traffic.rx_drop;
+			break;
+		case rx_interrupt_wait:
+			val += port->stats.traffic.rx_interrupt_wait;
+			break;
+		case rx_umonitor_umwait:
+			val += port->stats.traffic.rx_umonitor_umwait;
+			break;
+		case tx_ok:
+			val += port->stats.traffic.tx_ok;
+			break;
+		case total_polls:
+			val += port->stats.traffic.total_polls;
+			break;
+		case zero_polls:
+			val += port->stats.traffic.zero_polls;
+			break;
+		case tx_nospc_ldb_hw_credits:
+			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
+			break;
+		case tx_nospc_dir_hw_credits:
+			val += port->stats.traffic.tx_nospc_dir_hw_credits;
+			break;
+		case tx_nospc_inflight_max:
+			val += port->stats.traffic.tx_nospc_inflight_max;
+			break;
+		case tx_nospc_new_event_limit:
+			val += port->stats.traffic.tx_nospc_new_event_limit;
+			break;
+		case tx_nospc_inflight_credits:
+			val += port->stats.traffic.tx_nospc_inflight_credits;
+			break;
+		default:
+			return -1;
+		}
+	}
+	return val;
+}
+
+static uint64_t
+get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
+	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	switch (type) {
+	case rx_ok:
+	case rx_drop:
+	case rx_interrupt_wait:
+	case rx_umonitor_umwait:
+	case tx_ok:
+	case total_polls:
+	case zero_polls:
+	case tx_nospc_ldb_hw_credits:
+	case tx_nospc_dir_hw_credits:
+	case tx_nospc_inflight_max:
+	case tx_nospc_new_event_limit:
+	case tx_nospc_inflight_credits:
+		return dlb2_device_traffic_stat_get(dlb2, type);
+	case nb_events_limit:
+		return dlb2->new_event_limit;
+	case inflight_events:
+		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
+	case ldb_pool_size:
+		return dlb2->num_ldb_credits;
+	case dir_pool_size:
+		return dlb2->num_dir_credits;
+	default: return -1;
+	}
+}
+
+static uint64_t
+get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
+
+	switch (type) {
+	case rx_ok: return ev_port->stats.traffic.rx_ok;
+
+	case rx_drop: return ev_port->stats.traffic.rx_drop;
+
+	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
+
+	case rx_umonitor_umwait:
+		return ev_port->stats.traffic.rx_umonitor_umwait;
+
+	case tx_ok: return ev_port->stats.traffic.tx_ok;
+
+	case total_polls: return ev_port->stats.traffic.total_polls;
+
+	case zero_polls: return ev_port->stats.traffic.zero_polls;
+
+	case tx_nospc_ldb_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
+
+	case tx_nospc_dir_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
+
+	case tx_nospc_inflight_max:
+		return ev_port->stats.traffic.tx_nospc_inflight_max;
+
+	case tx_nospc_new_event_limit:
+		return ev_port->stats.traffic.tx_nospc_new_event_limit;
+
+	case tx_nospc_inflight_credits:
+		return ev_port->stats.traffic.tx_nospc_inflight_credits;
+
+	case is_configured: return ev_port->setup_done;
+
+	case is_load_balanced: return !ev_port->qm_port.is_directed;
+
+	case hw_id: return ev_port->qm_port.id;
+
+	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
+
+	case tx_fwd: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
+
+	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
+
+	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
+
+	case tx_sched_ordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case tx_sched_unordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case tx_sched_atomic:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case tx_sched_directed:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case tx_invalid: return ev_port->stats.tx_invalid;
+
+	case outstanding_releases: return ev_port->outstanding_releases;
+
+	case max_outstanding_releases:
+		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	case rx_sched_ordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case rx_sched_unordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case rx_sched_atomic:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case rx_sched_directed:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
+
+	default: return -1;
+	}
+}
+
+static uint64_t
+dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
+{
+	int port = 0;
+	uint64_t tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
+
+	return tally;
+}
+
+static uint64_t
+dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
+{
+	int port = 0;
+	uint64_t enq_ok_tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
+
+	return enq_ok_tally;
+}
+
+static uint64_t
+get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_queue *ev_queue =
+		&dlb2->ev_queues[obj_idx];
+
+	switch (type) {
+	case is_configured: return ev_queue->setup_done;
+
+	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
+
+	case hw_id: return ev_queue->qm_queue.id;
+
+	case num_links: return ev_queue->num_links;
+
+	case sched_type: return ev_queue->qm_queue.sched_type;
+
+	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
+
+	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
+
+	case depth_threshold: return ev_queue->depth_threshold;
+
+	case depth_le50_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_LE50);
+
+	case depth_gt50_le75_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT50_LE75);
+
+	case depth_gt75_le100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT75_LE100);
+
+	case depth_gt100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT100);
+
+	default: return -1;
+	}
+}
+
+int
+dlb2_xstats_init(struct dlb2_eventdev *dlb2)
+{
+	/*
+	 * define the stats names and types. Used to build up the device
+	 * xstats array
+	 * There are multiple set of stats:
+	 *   - device-level,
+	 *   - per-port,
+	 *   - per-qid,
+	 *
+	 * For each of these sets, we have three parallel arrays, one for the
+	 * names, the other for the stat type parameter to be passed in the fn
+	 * call to get that stat. The third array allows resetting or not.
+	 * All these arrays must be kept in sync
+	 */
+	static const char * const dev_stats[] = {
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"nb_events_limit",
+		"inflight_events",
+		"ldb_pool_size",
+		"dir_pool_size",
+	};
+	static const enum dlb2_xstats_type dev_types[] = {
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		nb_events_limit,
+		inflight_events,
+		ldb_pool_size,
+		dir_pool_size,
+	};
+	/* Note: generated device stats are not allowed to be reset. */
+	static const uint8_t dev_reset_allowed[] = {
+		0, /* rx_ok */
+		0, /* rx_drop */
+		0, /* rx_interrupt_wait */
+		0, /* rx_umonitor_umwait */
+		0, /* tx_ok */
+		0, /* total_polls */
+		0, /* zero_polls */
+		0, /* tx_nospc_ldb_hw_credits */
+		0, /* tx_nospc_dir_hw_credits */
+		0, /* tx_nospc_inflight_max */
+		0, /* tx_nospc_new_event_limit */
+		0, /* tx_nospc_inflight_credits */
+		0, /* nb_events_limit */
+		0, /* inflight_events */
+		0, /* ldb_pool_size */
+		0, /* dir_pool_size */
+	};
+	static const char * const port_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"tx_new",
+		"tx_fwd",
+		"tx_rel",
+		"tx_implicit_rel",
+		"tx_sched_ordered",
+		"tx_sched_unordered",
+		"tx_sched_atomic",
+		"tx_sched_directed",
+		"tx_invalid",
+		"outstanding_releases",
+		"max_outstanding_releases",
+		"rx_sched_ordered",
+		"rx_sched_unordered",
+		"rx_sched_atomic",
+		"rx_sched_directed",
+		"rx_sched_invalid"
+	};
+	static const enum dlb2_xstats_type port_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		tx_new,
+		tx_fwd,
+		tx_rel,
+		tx_implicit_rel,
+		tx_sched_ordered,
+		tx_sched_unordered,
+		tx_sched_atomic,
+		tx_sched_directed,
+		tx_invalid,
+		outstanding_releases,
+		max_outstanding_releases,
+		rx_sched_ordered,
+		rx_sched_unordered,
+		rx_sched_atomic,
+		rx_sched_directed,
+		rx_sched_invalid
+	};
+	static const uint8_t port_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		1, /* rx_ok */
+		1, /* rx_drop */
+		1, /* rx_interrupt_wait */
+		1, /* rx_umonitor_umwait */
+		1, /* tx_ok */
+		1, /* total_polls */
+		1, /* zero_polls */
+		1, /* tx_nospc_ldb_hw_credits */
+		1, /* tx_nospc_dir_hw_credits */
+		1, /* tx_nospc_inflight_max */
+		1, /* tx_nospc_new_event_limit */
+		1, /* tx_nospc_inflight_credits */
+		1, /* tx_new */
+		1, /* tx_fwd */
+		1, /* tx_rel */
+		1, /* tx_implicit_rel */
+		1, /* tx_sched_ordered */
+		1, /* tx_sched_unordered */
+		1, /* tx_sched_atomic */
+		1, /* tx_sched_directed */
+		1, /* tx_invalid */
+		0, /* outstanding_releases */
+		0, /* max_outstanding_releases */
+		1, /* rx_sched_ordered */
+		1, /* rx_sched_unordered */
+		1, /* rx_sched_atomic */
+		1, /* rx_sched_directed */
+		1  /* rx_sched_invalid */
+	};
+
+	/* QID specific stats */
+	static const char * const qid_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"num_links",
+		"sched_type",
+		"enq_ok",
+		"current_depth",
+		"depth_threshold",
+		"depth_le50_threshold",
+		"depth_gt50_le75_threshold",
+		"depth_gt75_le100_threshold",
+		"depth_gt100_threshold",
+	};
+	static const enum dlb2_xstats_type qid_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		num_links,
+		sched_type,
+		enq_ok,
+		current_depth,
+		depth_threshold,
+		depth_le50_threshold,
+		depth_gt50_le75_threshold,
+		depth_gt75_le100_threshold,
+		depth_gt100_threshold,
+	};
+	static const uint8_t qid_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		0, /* num_links */
+		0, /* sched_type */
+		1, /* enq_ok */
+		0, /* current_depth */
+		0, /* depth_threshold */
+		1, /* depth_le50_threshold */
+		1, /* depth_gt50_le75_threshold */
+		1, /* depth_gt75_le100_threshold */
+		1, /* depth_gt100_threshold */
+	};
+
+	/* ---- end of stat definitions ---- */
+
+	/* check sizes, since a missed comma can lead to strings being
+	 * joined by the compiler.
+	 */
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
+
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_reset_allowed));
+
+	/* other vars */
+	const unsigned int count = RTE_DIM(dev_stats) +
+			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
+			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
+	unsigned int i, port, qid, stat_id = 0;
+
+	dlb2->xstats = rte_zmalloc_socket(NULL,
+			sizeof(dlb2->xstats[0]) * count, 0,
+			dlb2->qm_instance.info.socket_id);
+	if (dlb2->xstats == NULL)
+		return -ENOMEM;
+
+#define sname dlb2->xstats[stat_id].name.name
+	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
+		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
+			.fn_id = DLB2_XSTATS_FN_DEV,
+			.stat = dev_types[i],
+			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
+			.reset_allowed = dev_reset_allowed[i],
+		};
+		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
+	}
+	dlb2->xstats_count_mode_dev = stat_id;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
+		dlb2->xstats_offset_for_port[port] = stat_id;
+
+		uint32_t count_offset = stat_id;
+
+		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_PORT,
+				.obj_idx = port,
+				.stat = port_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_PORT,
+				.reset_allowed = port_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "port_%u_%s",
+				 port, port_stats[i]);
+		}
+
+		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_port = stat_id - dlb2->xstats_count_mode_dev;
+
+	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
+		uint32_t count_offset = stat_id;
+
+		dlb2->xstats_offset_for_qid[qid] = stat_id;
+
+		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_QUEUE,
+				.obj_idx = qid,
+				.stat = qid_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
+				.reset_allowed = qid_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "qid_%u_%s",
+				 qid, qid_stats[i]);
+		}
+
+		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_queue = stat_id -
+		(dlb2->xstats_count_mode_dev + dlb2->xstats_count_mode_port);
+#undef sname
+
+	dlb2->xstats_count = stat_id;
+
+	return 0;
+}
+
+void
+dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
+{
+	rte_free(dlb2->xstats);
+	dlb2->xstats_count = 0;
+}
+
+int
+dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstats_names,
+		unsigned int *ids, unsigned int size)
+{
+	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+	uint32_t start_offset = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			break;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			break;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
+		break;
+	default:
+		return -EINVAL;
+	};
+
+	if (xstats_mode_count > size || ids == NULL || xstats_names == NULL)
+		return xstats_mode_count;
+
+	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
+		if (dlb2->xstats[i].mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != dlb2->xstats[i].obj_idx)
+			continue;
+
+		xstats_names[xidx] = dlb2->xstats[i].name;
+		if (ids)
+			ids[xidx] = start_offset + xidx;
+		xidx++;
+	}
+	return xidx;
+}
+
+static int
+dlb2_xstats_update(struct dlb2_eventdev *dlb2,
+		enum rte_event_dev_xstats_mode mode,
+		uint8_t queue_port_id, const unsigned int ids[],
+		uint64_t values[], unsigned int n, const uint32_t reset)
+{
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			goto invalid_value;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			goto invalid_value;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		break;
+	default:
+		goto invalid_value;
+	};
+
+	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
+		dlb2_xstats_fn fn;
+
+		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != xs->obj_idx)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			goto invalid_value;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+
+		if (values)
+			values[xidx] = val;
+
+		if (xs->reset_allowed && reset)
+			xs->reset_value += val;
+
+		xidx++;
+	}
+
+	return xidx;
+
+invalid_value:
+	return -EINVAL;
+}
+
+int
+dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	const uint32_t reset = 0;
+
+	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
+				  reset);
+}
+
+uint64_t
+dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+				 const char *name, unsigned int *id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	dlb2_xstats_fn fn;
+
+	for (i = 0; i < dlb2->xstats_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (strncmp(xs->name.name, name,
+			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
+			if (id != NULL)
+				*id = i;
+
+			switch (xs->fn_id) {
+			case DLB2_XSTATS_FN_DEV:
+				fn = get_dev_stat;
+				break;
+			case DLB2_XSTATS_FN_PORT:
+				fn = get_port_stat;
+				break;
+			case DLB2_XSTATS_FN_QUEUE:
+				fn = get_queue_stat;
+				break;
+			default:
+				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
+					  xs->fn_id);
+				return (uint64_t)-1;
+			}
+
+			return fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+		}
+	}
+	if (id != NULL)
+		*id = (uint32_t)-1;
+	return (uint64_t)-1;
+}
+
+static void
+dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
+			uint32_t num)
+{
+	uint32_t i;
+	dlb2_xstats_fn fn;
+
+	for (i = start; i < start + num; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (!xs->reset_allowed)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			return;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
+		xs->reset_value = val;
+	}
+}
+
+static int
+dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
+			const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_QUEUE,
+					queue_id, ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	if (ids == NULL)
+		dlb2_xstats_reset_range(dlb2,
+			dlb2->xstats_offset_for_qid[queue_id],
+			dlb2->xstats_count_per_qid[queue_id]);
+
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
+		       const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+	int offset = dlb2->xstats_offset_for_port[port_id];
+	int nb_stat = dlb2->xstats_count_per_port[port_id];
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_PORT, port_id,
+					ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
+		      uint32_t nb_ids)
+{
+	uint32_t i;
+
+	if (ids) {
+		for (i = 0; i < nb_ids; i++) {
+			uint32_t id = ids[i];
+
+			if (id >= dlb2->xstats_count_mode_dev)
+				return -EINVAL;
+			dlb2_xstats_reset_range(dlb2, id, 1);
+		}
+	} else {
+		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
+			dlb2_xstats_reset_range(dlb2, i, 1);
+	}
+
+	return 0;
+}
+
+int
+dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+			   enum rte_event_dev_xstats_mode mode,
+			   int16_t queue_port_id,
+			   const uint32_t ids[],
+			   uint32_t nb_ids)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	uint32_t i;
+
+	/* handle -1 for queue_port_id here, looping over all ports/queues */
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
+			return -EINVAL;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+				if (dlb2_xstats_reset_port(dlb2, i,
+							   ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
+			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
+						   ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
+				if (dlb2_xstats_reset_queue(dlb2, i,
+							    ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
+			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
+						    ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	};
+
+	return 0;
+}
+
+void
+dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	int i;
+
+	if (f == NULL) {
+		printf("Invalid file pointer\n");
+		return;
+	}
+
+	if (dev == NULL) {
+		fprintf(f, "Invalid event device\n");
+		return;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (dlb2 == NULL) {
+		fprintf(f, "DLB2 Event device cannot be dumped!\n");
+		return;
+	}
+
+	if (!dlb2->configured)
+		fprintf(f, "DLB2 Event device is not configured\n");
+
+	handle = &dlb2->qm_instance;
+
+	fprintf(f, "================\n");
+	fprintf(f, "DLB2 Device Dump\n");
+	fprintf(f, "================\n");
+
+	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
+		dlb2->umwait_allowed ? "yes" : "no");
+
+	/* Generic top level device information */
+
+	fprintf(f, "device is configured and run state =");
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		fprintf(f, "STOPPED\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
+		fprintf(f, "STOPPING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
+		fprintf(f, "STARTING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
+		fprintf(f, "STARTED\n");
+	else
+		fprintf(f, "UNEXPECTED\n");
+
+	fprintf(f, "domain ID=%u, socket_id=%u, evdev=%p\n",
+		handle->domain_id, handle->info.socket_id, dlb2->event_dev);
+
+	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
+		dlb2->num_dir_ports, dlb2->num_dir_queues);
+
+	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
+		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
+
+	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
+		handle->cfg.resources.num_atomic_inflights,
+		handle->cfg.resources.num_hist_list_entries);
+
+	fprintf(f, "results from most recent hw resource query:\n");
+
+	fprintf(f, "\tnum_sched_domains = %u\n",
+		dlb2->hw_rsrc_query_results.num_sched_domains);
+
+	fprintf(f, "\tnum_ldb_queues = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_queues);
+
+	fprintf(f, "\tnum_ldb_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_ports);
+
+	fprintf(f, "\tnum_dir_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_ports);
+
+	fprintf(f, "\tnum_atomic_inflights = %u\n",
+		dlb2->hw_rsrc_query_results.num_atomic_inflights);
+
+	fprintf(f, "\tnum_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.num_hist_list_entries);
+
+	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
+
+	fprintf(f, "\tnum_ldb_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_credits);
+
+	fprintf(f, "\tnum_dir_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_credits);
+
+	/* Port level information */
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
+		int j;
+
+		if (!p->enq_configured)
+			fprintf(f, "Port_%d is not configured\n", i);
+
+		fprintf(f, "Port_%d\n", i);
+		fprintf(f, "=======\n");
+
+		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
+			p->id, p->setup_done);
+
+		fprintf(f, "\tconfig state=%d, port state=%d\n",
+			p->qm_port.config_state, p->qm_port.state);
+
+		fprintf(f, "\tport is %s\n",
+			p->qm_port.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\toutstanding releases=%u\n",
+			p->outstanding_releases);
+
+		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
+			p->inflight_max, p->inflight_credits);
+
+		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
+			p->credit_update_quanta, p->implicit_release);
+
+		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (p->link[j].valid)
+				fprintf(f, "id=%u prio=%u ",
+					p->link[j].queue_id,
+					p->link[j].priority);
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
+
+		fprintf(f, "\tcached_ldb_credits=%u\n",
+			p->qm_port.cached_ldb_credits);
+
+		fprintf(f, "\tldb_credits = %u\n",
+			p->qm_port.ldb_credits);
+
+		fprintf(f, "\tcached_dir_credits = %u\n",
+			p->qm_port.cached_dir_credits);
+
+		fprintf(f, "\tdir_credits = %u\n",
+			p->qm_port.dir_credits);
+
+		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
+			p->qm_port.gen_bit,
+			p->qm_port.cq_idx,
+			p->qm_port.cq_depth);
+
+		fprintf(f, "\tinterrupt armed=%d\n",
+			p->qm_port.int_armed);
+
+		fprintf(f, "\tPort statistics\n");
+
+		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
+			p->stats.traffic.rx_ok);
+
+		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
+			p->stats.traffic.rx_drop);
+
+		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
+			p->stats.traffic.rx_interrupt_wait);
+
+		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
+			p->stats.traffic.rx_umonitor_umwait);
+
+		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
+			p->stats.traffic.tx_ok);
+
+		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
+			p->stats.traffic.total_polls);
+
+		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
+			p->stats.traffic.zero_polls);
+
+		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_ldb_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_dir_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_max);
+
+		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_new_event_limit);
+
+		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_credits);
+
+		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
+
+		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
+
+		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
+
+		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
+			p->stats.tx_implicit_rel);
+
+		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
+			p->stats.tx_invalid);
+
+		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
+			p->stats.rx_sched_invalid);
+	}
+
+	/* Queue level information */
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
+		int j, k;
+
+		if (!q->setup_done)
+			fprintf(f, "Queue_%d is not configured\n", i);
+
+		fprintf(f, "Queue_%d\n", i);
+		fprintf(f, "========\n");
+
+		fprintf(f, "\tevqueue_%u is set up\n", q->id);
+
+		fprintf(f, "\tqueue is %s\n",
+			q->qm_queue.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
+
+		for (j = 0; j < dlb2->num_ports; j++) {
+			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
+
+			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++) {
+				if (p->link[k].valid &&
+				    p->link[k].queue_id == q->id)
+					fprintf(f, "id=%u prio=%u ",
+						p->id, p->link[k].priority);
+			}
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\tcurrent depth: %u events\n",
+			dlb2_get_queue_depth(dlb2, q));
+
+		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
+			q->qm_queue.num_qid_inflights, q->qm_queue.sched_type);
+	}
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 28fef3f..dc16c96 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files('dlb2.c',
 		'dlb2_iface.c',
+		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c'
-- 
2.6.4

[dpdk-dev] [PATCH v9 10/23] event/dlb2: add infos get and configure

[Timothy McDaniel]

Add support for configuring the DLB2 hardware.
In particular, this patch configures the DLB2
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  116 +
 drivers/event/dlb2/dlb2.c                  |  313 +++
 drivers/event/dlb2/dlb2_iface.c            |    5 +
 drivers/event/dlb2/dlb2_iface.h            |    4 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 3237 ++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |   15 +
 drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
 7 files changed, 3734 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index e3091c9..5f6c486 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -32,9 +32,125 @@ detailed understanding of the hardware, but these details are important when
 writing high-performance code. This section describes the places where the
 eventdev API and DLB2 misalign.
 
+Scheduling Domain Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are 32 scheduling domainis the DLB2.
+When one is configured, it allocates load-balanced and
+directed queues, ports, credits, and other hardware resources. Some
+resource allocations are user-controlled -- the number of queues, for example
+-- and others, like credit pools (one directed and one load-balanced pool per
+scheduling domain), are not.
+
+The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
+setup argument and the per-port ``new_event_threshold`` argument apply as
+defined in the eventdev header file. The limit is applied to all enqueues,
+regardless of whether it will consume a directed or load-balanced credit.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Reconfiguration
+~~~~~~~~~~~~~~~
+
+The Eventdev API allows one to reconfigure a device, its ports, and its queues
+by first stopping the device, calling the configuration function(s), then
+restarting the device. The DLB2 does not support configuring an individual queue
+or port without first reconfiguring the entire device, however, so there are
+certain reconfiguration sequences that are valid in the eventdev API but not
+supported by the PMD.
+
+Specifically, the PMD supports the following configuration sequence:
+1. Configure and start the device
+2. Stop the device
+3. (Optional) Reconfigure the device
+4. (Optional) If step 3 is run:
+
+   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
+   b. The reconfigured port(s) lose their previous queue links.
+
+5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
+6. Restart the device. If the device is reconfigured in step 3 but one or more
+   of its ports or queues are not, the PMD will apply their previous
+   configuration (including port->queue links) at this time.
+
+The PMD does not support the following configuration sequences:
+1. Configure and start the device
+2. Stop the device
+3. Setup queue or setup port
+4. Start the device
+
+This sequence is not supported because the event device must be reconfigured
+before its ports or queues can be.
+
+Atomic Inflights Allocation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
+inflight event in a temporary buffer that is divided among load-balanced
+queues. If a queue's atomic buffer storage fills up, this can result in
+head-of-line-blocking. For example:
+
+- An LDB queue allocated N atomic buffer entries
+- All N entries are filled with events from flow X, which is pinned to CQ 0.
+
+Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
+scheduled. The likelihood of this case depends on the eventdev configuration,
+traffic behavior, event processing latency, potential for a worker to be
+interrupted or otherwise delayed, etc.
+
+By default, the PMD allocates 16 buffer entries for each load-balanced queue,
+which provides an even division across all 128 queues but potentially wastes
+buffer space (e.g. if not all queues are used, or aren't used for atomic
+scheduling).
+
+The PMD provides a dev arg to override the default per-queue allocation. To
+increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,atm_inflights=64
+
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 925824e..6798e66 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -85,6 +85,25 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
 	return 0;
 }
 
+static void
+dlb2_free_qe_mem(struct dlb2_port *qm_port)
+{
+	if (qm_port == NULL)
+		return;
+
+	rte_free(qm_port->qe4);
+	qm_port->qe4 = NULL;
+
+	rte_free(qm_port->int_arm_qe);
+	qm_port->int_arm_qe = NULL;
+
+	rte_free(qm_port->consume_qe);
+	qm_port->consume_qe = NULL;
+
+	rte_memzone_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz);
+	dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz = NULL;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -350,10 +369,304 @@ set_qid_depth_thresh(const char *key __rte_unused,
 }
 
 static void
+dlb2_eventdev_info_get(struct rte_eventdev *dev,
+		       struct rte_event_dev_info *dev_info)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret;
+
+	ret = dlb2_hw_query_resources(dlb2);
+	if (ret) {
+		const struct rte_eventdev_data *data = dev->data;
+
+		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+			     ret, data->dev_id);
+		/* fn is void, so fall through and return values set up in
+		 * probe
+		 */
+	}
+
+	/* Add num resources currently owned by this domain.
+	 * These would become available if the scheduling domain were reset due
+	 * to the application recalling eventdev_configure to *reconfigure* the
+	 * domain.
+	 */
+	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
+	evdev_dlb2_default_info.max_event_queues += dlb2->num_ldb_queues;
+	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
+
+	evdev_dlb2_default_info.max_event_queues =
+		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
+			RTE_EVENT_MAX_QUEUES_PER_DEV);
+
+	evdev_dlb2_default_info.max_num_events =
+		RTE_MIN(evdev_dlb2_default_info.max_num_events,
+			dlb2->max_num_events_override);
+
+	*dev_info = evdev_dlb2_default_info;
+}
+
+static int
+dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
+			    const struct dlb2_hw_rsrcs *resources_asked)
+{
+	int ret = 0;
+	struct dlb2_create_sched_domain_args *cfg;
+
+	if (resources_asked == NULL) {
+		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
+		ret = EINVAL;
+		goto error_exit;
+	}
+
+	/* Map generic qm resources to dlb2 resources */
+	cfg = &handle->cfg.resources;
+
+	/* DIR ports and queues */
+
+	cfg->num_dir_ports = resources_asked->num_dir_ports;
+
+	cfg->num_dir_credits = resources_asked->num_dir_credits;
+
+	/* LDB queues */
+
+	cfg->num_ldb_queues = resources_asked->num_ldb_queues;
+
+	/* LDB ports */
+
+	cfg->cos_strict = 0; /* Best effort */
+	cfg->num_cos_ldb_ports[0] = 0;
+	cfg->num_cos_ldb_ports[1] = 0;
+	cfg->num_cos_ldb_ports[2] = 0;
+	cfg->num_cos_ldb_ports[3] = 0;
+
+	switch (handle->cos_id) {
+	case DLB2_COS_0:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[0] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_1:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[1] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_2:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[2] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_3:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[3] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_DEFAULT:
+		/* all ldb ports are don't care ports from a cos perspective */
+		cfg->num_ldb_ports =
+			resources_asked->num_ldb_ports;
+		break;
+	}
+
+	cfg->num_ldb_credits =
+		resources_asked->num_ldb_credits;
+
+	cfg->num_atomic_inflights =
+		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
+		cfg->num_ldb_queues;
+
+	cfg->num_hist_list_entries = resources_asked->num_ldb_ports *
+		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d\n",
+		     cfg->num_ldb_queues,
+		     resources_asked->num_ldb_ports,
+		     cfg->num_dir_ports,
+		     cfg->num_atomic_inflights,
+		     cfg->num_hist_list_entries,
+		     cfg->num_ldb_credits,
+		     cfg->num_dir_credits);
+
+	/* Configure the QM */
+
+	ret = dlb2_iface_sched_domain_create(handle, cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: domain create failed, ret = %d, extra status: %s\n",
+			     ret,
+			     dlb2_error_strings[cfg->response.status]);
+
+		goto error_exit;
+	}
+
+	handle->domain_id = cfg->response.id;
+	handle->cfg.configured = true;
+
+error_exit:
+
+	return ret;
+}
+
+static void
+dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	enum dlb2_configuration_state config_state;
+	int i, j;
+
+	dlb2_iface_domain_reset(dlb2);
+
+	/* Free all dynamically allocated port memory */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
+
+	/* If reconfiguring, mark the device's queues and ports as "previously
+	 * configured." If the user doesn't reconfigure them, the PMD will
+	 * reapply their previous configuration when the device is started.
+	 */
+	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
+		DLB2_NOT_CONFIGURED;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		dlb2->ev_ports[i].qm_port.config_state = config_state;
+		/* Reset setup_done so ports can be reconfigured */
+		dlb2->ev_ports[i].setup_done = false;
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			dlb2->ev_ports[i].link[j].mapped = false;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++)
+		dlb2->ev_queues[i].qm_queue.config_state = config_state;
+
+	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
+		dlb2->ev_queues[i].setup_done = false;
+
+	dlb2->num_ports = 0;
+	dlb2->num_ldb_ports = 0;
+	dlb2->num_dir_ports = 0;
+	dlb2->num_queues = 0;
+	dlb2->num_ldb_queues = 0;
+	dlb2->num_dir_queues = 0;
+	dlb2->configured = false;
+}
+
+/* Note: 1 QM instance per QM device, QM instance/device == event device */
+static int
+dlb2_eventdev_configure(const struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
+	const struct rte_eventdev_data *data = dev->data;
+	const struct rte_event_dev_config *config = &data->dev_conf;
+	int ret;
+
+	/* If this eventdev is already configured, we must release the current
+	 * scheduling domain before attempting to configure a new one.
+	 */
+	if (dlb2->configured) {
+		dlb2_hw_reset_sched_domain(dev, true);
+
+		ret = dlb2_hw_query_resources(dlb2);
+		if (ret) {
+			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+				     ret, data->dev_id);
+			return ret;
+		}
+	}
+
+	if (config->nb_event_queues > rsrcs->num_queues) {
+		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_queues,
+			     rsrcs->num_queues);
+		return -EINVAL;
+	}
+	if (config->nb_event_ports > (rsrcs->num_ldb_ports
+			+ rsrcs->num_dir_ports)) {
+		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_ports,
+			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
+		return -EINVAL;
+	}
+	if (config->nb_events_limit > rsrcs->nb_events_limit) {
+		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_events_limit,
+			     rsrcs->nb_events_limit);
+		return -EINVAL;
+	}
+
+	if (config->event_dev_cfg & RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT)
+		dlb2->global_dequeue_wait = false;
+	else {
+		uint32_t timeout32;
+
+		dlb2->global_dequeue_wait = true;
+
+		/* note size mismatch of timeout vals in eventdev lib. */
+		timeout32 = config->dequeue_timeout_ns;
+
+		dlb2->global_dequeue_wait_ticks =
+			timeout32 * (rte_get_timer_hz() / 1E9);
+	}
+
+	/* Does this platform support umonitor/umwait? */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
+		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
+			DLB2_LOG_ERR("invalid value (%d) for RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
+				     RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
+			return -EINVAL;
+		}
+		dlb2->umwait_allowed = true;
+	}
+
+	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
+	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs->num_dir_ports;
+	/* 1 dir queue per dir port */
+	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports;
+
+	/* Scale down nb_events_limit by 4 for directed credits, since there
+	 * are 4x as many load-balanced credits.
+	 */
+	rsrcs->num_ldb_credits = 0;
+	rsrcs->num_dir_credits = 0;
+
+	if (rsrcs->num_ldb_queues)
+		rsrcs->num_ldb_credits = config->nb_events_limit;
+	if (rsrcs->num_dir_ports)
+		rsrcs->num_dir_credits = config->nb_events_limit / 4;
+	if (dlb2->num_dir_credits_override != -1)
+		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
+
+	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
+		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
+		return -ENODEV;
+	}
+
+	dlb2->new_event_limit = config->nb_events_limit;
+	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
+
+	/* Save number of ports/queues for this event dev */
+	dlb2->num_ports = config->nb_event_ports;
+	dlb2->num_queues = config->nb_event_queues;
+	dlb2->num_dir_ports = rsrcs->num_dir_ports;
+	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
+	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
+	dlb2->num_dir_queues = dlb2->num_dir_ports;
+	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
+	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
+	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
+	dlb2->max_dir_credits = rsrcs->num_dir_credits;
+
+	dlb2->configured = true;
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dev_infos_get    = dlb2_eventdev_info_get,
+		.dev_configure    = dlb2_eventdev_configure,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 0fc9991..a829b9b 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -26,3 +26,8 @@ int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
+
+int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				struct dlb2_create_sched_domain_args *args);
+
+void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4fb416e..6663dab 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -26,4 +26,8 @@ extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
 
+extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				 struct dlb2_create_sched_domain_args *args);
+
+extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 6de8b95..63a68a6 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -12,6 +12,27 @@
 #include "dlb2_regs.h"
 #include "dlb2_resource.h"
 
+#include "../../dlb2_priv.h"
+#include "../../dlb2_inline_fns.h"
+
+#define DLB2_DOM_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, domain_list, iter)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, func_list, iter)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, domain_list, it, it_tmp)
+
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, func_list, it, it_tmp)
+
 static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
 {
 	int i;
@@ -272,3 +293,3219 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
 
 	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
 }
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+	union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+	r0.field.count = domain->num_ldb_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+	r1.field.count = domain->num_dir_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+		       struct dlb2_function_resources *rsrcs,
+		       u32 domain_id,
+		       u32 cos_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	RTE_SET_USED(iter);
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+				   typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				   struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_ports,
+				   u32 cos_id,
+				   struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_ldb_port *port;
+
+		port = dlb2_get_next_ldb_port(hw, rsrcs,
+					      domain->id.phys_id, cos_id);
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_ports[cos_id],
+			      &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_ports[cos_id],
+			      &port->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_create_sched_domain_args *args,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      num,
+						      i,
+						      resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+					ret = __dlb2_attach_ldb_ports(hw,
+								      rsrcs,
+								      domain,
+								      1,
+								      cos_id,
+								      resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret < 0)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      1,
+						      j,
+						      resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 u32 num_ports,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+					   typeof(*port));
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_dir_pq_pairs, &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_dir_pq_pairs, &port->domain_list);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+					struct dlb2_hw_domain *domain,
+					u32 num_atomic_inflights,
+					struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+				     struct dlb2_hw_domain *domain,
+				     u32 num_hist_list_entries,
+				     struct dlb2_cmd_response *resp)
+{
+	struct dlb2_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb2_bitmap_find_set_bit_range(bitmap,
+						      num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+				  struct dlb2_function_resources *rsrcs,
+				  struct dlb2_hw_domain *domain,
+				  u32 num_queues,
+				  struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb2_ldb_queue *queue;
+
+		queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+					    typeof(*queue));
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_queues, &queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_queues, &queue->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+			     struct dlb2_function_resources *rsrcs,
+			     struct dlb2_hw_domain *domain,
+			     struct dlb2_create_sched_domain_args *args,
+			     struct dlb2_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb2_attach_ldb_queues(hw,
+				     rsrcs,
+				     domain,
+				     args->num_ldb_queues,
+				     resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_ports(hw,
+				    rsrcs,
+				    domain,
+				    args,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_ports(hw,
+				    rsrcs,
+				    domain,
+				    args->num_dir_ports,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_credits(rsrcs,
+				      domain,
+				      args->num_ldb_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_credits(rsrcs,
+				      domain,
+				      args->num_dir_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+						   domain,
+						   args->num_hist_list_entries,
+						   resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_atomic_inflights(rsrcs,
+					   domain,
+					   args->num_atomic_inflights,
+					   resp);
+	if (ret < 0)
+		return ret;
+
+	dlb2_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+				  struct dlb2_create_sched_domain_args *args,
+				  struct dlb2_cmd_response *resp)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb2_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+				  struct dlb2_create_sched_domain_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		    args->num_ldb_queues);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		    args->num_ldb_ports);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		    args->num_cos_ldb_ports[0]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		    args->cos_strict);
+	DLB2_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		    args->num_dir_ports);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		    args->num_atomic_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		    args->num_hist_list_entries);
+	DLB2_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		    args->num_ldb_credits);
+	DLB2_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		    args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ *	domain and its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+	if (ret)
+		return ret;
+
+	domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available domains\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (domain->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_domains contains configured domains.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	dlb2_init_domain_rsrc_lists(domain);
+
+	ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to verify args.\n",
+			    __func__);
+
+		return ret;
+	}
+
+	dlb2_list_del(&rsrcs->avail_domains, &domain->func_list);
+
+	dlb2_list_add(&rsrcs->used_domains, &domain->func_list);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+	DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+			     struct dlb2_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb2_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	int ret;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_disable(hw, port);
+
+		ret = dlb2_drain_dir_cq(hw, port);
+		if (ret < 0)
+			return ret;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_dir_pq_pair *queue)
+{
+	union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *queue)
+{
+	return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+				      struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		dlb2_movdir64b(pp_addr, hcw);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int ret, i;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			if (toggle_port)
+				dlb2_ldb_port_cq_disable(hw, port);
+
+			ret = dlb2_drain_ldb_cq(hw, port);
+			if (ret < 0)
+				return ret;
+
+			if (toggle_port)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+
+	return 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_atm_active r1;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_ldb_queue *queue)
+{
+	return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb2_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+					   struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to unmap domain queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = true;
+
+			dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+			   u32 id,
+			   bool vdev_req,
+			   unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iter1) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter2)
+			if (queue->id.virt_id == id)
+				return queue;
+	}
+
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue, iter1)
+		if (queue->id.virt_id == id)
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+						      u32 id,
+						      bool vdev_req,
+						      unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iteration;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	RTE_SET_USED(iteration);
+
+	if (id >= DLB2_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iteration)
+		if (domain->id.virt_id == id)
+			return domain;
+
+	return NULL;
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot,
+					   enum dlb2_qid_map_state new_state)
+{
+	enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb2_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB2_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB2_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB2_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB2_HW_DBG(hw,
+		    "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB2_HW_ERR(hw,
+		    "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+				enum dlb2_qid_map_state state,
+				int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+				      enum dlb2_qid_map_state state,
+				      struct dlb2_ldb_queue *queue,
+				      int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+					      struct dlb2_hw_domain *domain,
+					      struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain,
+					     struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+						struct dlb2_ldb_port *port,
+						int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 0;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 1;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+					struct dlb2_ldb_port *p,
+					struct dlb2_ldb_queue *q,
+					u8 priority)
+{
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_lsp_cq2qid0 r1;
+	union dlb2_atm_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix_00 r3;
+	union dlb2_lsp_qid2cqidix2_00 r4;
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+	    !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+	r0.field.v |= 1 << i;
+	r0.field.prio |= (priority & 0x7) << i * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+	else
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		r1.field.qid_p0 = q->id.phys_id;
+	if (i == 1 || i == 5)
+		r1.field.qid_p1 = q->id.phys_id;
+	if (i == 2 || i == 6)
+		r1.field.qid_p2 = q->id.phys_id;
+	if (i == 3 || i == 7)
+		r1.field.qid_p3 = q->id.phys_id;
+
+	if (i < 4)
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+	else
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r4.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+						  p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		r2.field.cq_p0 |= 1 << i;
+		r3.field.cq_p0 |= 1 << i;
+		r4.field.cq_p0 |= 1 << i;
+		break;
+
+	case 1:
+		r2.field.cq_p1 |= 1 << i;
+		r3.field.cq_p1 |= 1 << i;
+		r4.field.cq_p1 |= 1 << i;
+		break;
+
+	case 2:
+		r2.field.cq_p2 |= 1 << i;
+		r3.field.cq_p2 |= 1 << i;
+		r4.field.cq_p2 |= 1 << i;
+		break;
+
+	case 3:
+		r2.field.cq_p3 |= 1 << i;
+		r3.field.cq_p3 |= 1 << i;
+		r4.field.cq_p3 |= 1 << i;
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r3.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		    r4.val);
+
+	dlb2_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB2_QUEUE_MAPPED;
+
+	return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	/* Set the atomic scheduling haswork bit */
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.rlist_haswork_v = r0.field.count > 0;
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      u8 slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.rlist_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.nalb_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+					      struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+	r0.field.limit = queue->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+						struct dlb2_ldb_queue *queue)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+						struct dlb2_hw_domain *domain,
+						struct dlb2_ldb_port *port,
+						struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_lsp_qid_ldb_infl_cnt r0;
+	enum dlb2_qid_map_state state;
+	int slot, ret, i;
+	u8 prio;
+	RTE_SET_USED(iter);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: non-zero QID inflight count\n",
+			    __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			state = DLB2_QUEUE_MAPPED;
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+					 struct dlb2_ldb_port *port,
+					 struct dlb2_ldb_queue *queue,
+					 u8 priority)
+{
+	union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	int domain_id, slot, ret;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+		DLB2_HW_ERR(hw,
+			    "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		if (port->enabled)
+			dlb2_ldb_port_cq_enable(hw, port);
+
+		dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain,
+					struct dlb2_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		union dlb2_lsp_qid_ldb_infl_cnt r0;
+		struct dlb2_ldb_queue *queue;
+		int qid;
+
+		if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: unable to find queue %d\n",
+				    __func__, qid);
+			continue;
+		}
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count)
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb2_ldb_port_cq_disable(hw, port);
+
+		dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count) {
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+
+			dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port,
+				   struct dlb2_ldb_queue *queue)
+{
+	enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_atm_qid2cqidix_00 r1;
+	union dlb2_lsp_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix2_00 r3;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB2_QUEUE_MAPPED;
+	in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: QID %d isn't mapped\n",
+			    __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+	r0.field.v &= ~(1 << i);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		r1.field.cq_p0 &= ~(1 << i);
+		r2.field.cq_p0 &= ~(1 << i);
+		r3.field.cq_p0 &= ~(1 << i);
+		break;
+
+	case 1:
+		r1.field.cq_p1 &= ~(1 << i);
+		r2.field.cq_p1 &= ~(1 << i);
+		r3.field.cq_p1 &= ~(1 << i);
+		break;
+
+	case 2:
+		r1.field.cq_p2 &= ~(1 << i);
+		r2.field.cq_p2 &= ~(1 << i);
+		r3.field.cq_p2 &= ~(1 << i);
+		break;
+
+	case 3:
+		r1.field.cq_p3 &= ~(1 << i);
+		r2.field.cq_p3 &= ~(1 << i);
+		r3.field.cq_p3 &= ~(1 << i);
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+		    r1.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+		    r3.val);
+
+	dlb2_flush_csr(hw);
+
+	unmapped = DLB2_QUEUE_UNMAPPED;
+
+	return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_ldb_port *port,
+				 struct dlb2_ldb_queue *queue,
+				 u8 prio)
+{
+	if (domain->started)
+		return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   int slot)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb2_ldb_port_qid_map *map;
+		struct dlb2_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain,
+					  struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (r0.field.count > 0)
+		return false;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = false;
+
+			dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_dir_vpp_v r1;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		unsigned int offs;
+		u32 virt_id;
+
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+	}
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_ldb_vpp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			unsigned int offs;
+			u32 virt_id;
+
+			if (hw->virt_mode == DLB2_VIRT_SRIOV)
+				virt_id = port->id.virt_id;
+			else
+				virt_id = port->id.phys_id;
+
+			offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+			DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+	union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+				    r0.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+				    r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+	union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+			    r0.val);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+			    r1.val);
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+		union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+		union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+				    r1.val);
+
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID_V(idx),
+				    r2.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID2QID(idx),
+				    r3.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID_V(idx),
+				    r1.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID2QID(idx),
+				    r2.val);
+		}
+	}
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+					       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_sn_chk_enbl r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.en = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			int i;
+
+			for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+				if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					    __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		port->enabled = false;
+
+		dlb2_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	union dlb2_sys_dir_pp_v r1;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+			    r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_ldb_pp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_ldb_queue *queue;
+	int i;
+	RTE_SET_USED(iter);
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty ldb queue %d\n",
+				    __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port, iter) {
+			if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb2_ldb_cq_token_count(hw, ldb_port)) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to empty ldb port %d\n",
+					    __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir queue %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb2_dir_cq_token_count(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir port %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						   struct dlb2_ldb_port *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP2PP(offs),
+			    DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP_V(offs),
+			    DLB2_SYS_VF_LDB_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    DLB2_SYS_LDB_PP_V_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_DSBL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_BASE_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID0(port->id.phys_id),
+		    DLB2_LSP_CQ2QID0_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID1(port->id.phys_id),
+		    DLB2_LSP_CQ2QID1_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+		    DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			__dlb2_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+				       struct dlb2_dir_pq_pair *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_DSBL_RST);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_FMT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP2PP(offs),
+			    DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP_V(offs),
+			    DLB2_SYS_VF_DIR_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		__dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+			    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_ITS(queue_id),
+			    DLB2_SYS_LDB_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN(queue_id),
+			    DLB2_CHP_ORD_QID_SN_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+			    DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_V(queue_id),
+			    DLB2_SYS_LDB_QID_V_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_CFG_V(queue_id),
+			    DLB2_SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB2_CSR_WR(hw,
+				    offs[queue->sn_group],
+				    DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX2(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX2_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_ATM_QID2CQIDIX(queue_id, i),
+				    DLB2_ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb2_domain_reset_dir_port_registers(hw, domain);
+
+	dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_dir_pq_pair *tmp_dir_port;
+	struct dlb2_ldb_queue *tmp_ldb_queue;
+	struct dlb2_ldb_port *tmp_ldb_port;
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_list_head *list;
+	int ret, i;
+	RTE_SET_USED(tmp_dir_port);
+	RTE_SET_USED(tmp_ldb_queue);
+	RTE_SET_USED(tmp_ldb_port);
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list = &domain->used_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb2_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		dlb2_list_del(&domain->used_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list = &domain->avail_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		ldb_queue->owned = false;
+
+		dlb2_list_del(&domain->avail_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		list = &domain->used_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB2_QUEUE_UNMAPPED;
+
+			dlb2_list_del(&domain->used_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list = &domain->avail_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			ldb_port->owned = false;
+
+			dlb2_list_del(&domain->avail_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list = &domain->used_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		dlb2_list_del(&domain->used_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list = &domain->avail_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				    domain->hist_list_entry_base,
+				    domain->total_hist_list_entries);
+	if (ret) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			    __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	dlb2_list_del(&rsrcs->used_domains, &domain->func_list);
+	dlb2_list_add(&rsrcs->avail_domains, &domain->func_list);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain,
+					    struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_ldb_port *port;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (!dlb2_list_empty(&domain->used_ldb_ports[i]))
+			break;
+	}
+
+	if (i == DLB2_NUM_COS_DOMAINS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: No configured LDB ports\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb2_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+	 * prior to this in dlb2_domain_drain_mapped_queues().
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings != 0 ||
+		    dlb2_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ *	hardware resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain  == NULL || !domain->configured)
+		return -EINVAL;
+
+	/* Disable VPPs */
+	if (vdev_req) {
+		dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+		dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+	}
+
+	/* Disable CQ interrupts */
+	dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+	dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+	dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb2_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_mapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	dlb2_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb2_domain_disable_dir_cqs(hw, domain);
+
+	/* Disable PPs */
+	dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+	ret = dlb2_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Reset the QID and port state. */
+	dlb2_domain_reset_registers(hw, domain);
+
+	/* Hardware reset complete. Reset the domain's software state */
+	ret = dlb2_domain_reset_software_state(hw, domain);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 210b3186..285ad2c 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -598,3 +598,18 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
 
 	return 0;
 }
+
+int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+			    struct dlb2_create_sched_domain_args *args,
+			    struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_sched_domain(hw, args, resp, NOT_VF_REQ,
+					   PF_ID_ZERO);
+}
+
+int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 7d64309..9310f72 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -108,15 +108,59 @@ dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
 	return 0;
 }
 
+static int
+dlb2_pf_sched_domain_create(struct dlb2_hw_dev *handle,
+			    struct dlb2_create_sched_domain_args *arg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (dlb2_dev->domain_reset_failed) {
+		response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = dlb2_pf_create_sched_domain(&dlb2_dev->hw, arg, &response);
+	if (ret)
+		goto done;
+
+done:
+
+	arg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static void
+dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret;
+
+	dlb2_dev = (struct dlb2_dev *)dlb2->qm_instance.pf_dev;
+	ret = dlb2_pf_reset_domain(&dlb2_dev->hw, dlb2->qm_instance.domain_id);
+	if (ret)
+		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
 	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
 	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_domain_reset = dlb2_pf_domain_reset;
 	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
 	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v9 11/23] event/dlb2: add queue and port default conf

[Timothy McDaniel]

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 6798e66..ef1c000 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -661,12 +661,42 @@ dlb2_eventdev_configure(const struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
+				    uint8_t port_id,
+				    struct rte_event_port_conf *port_conf)
+{
+	RTE_SET_USED(port_id);
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	port_conf->new_event_threshold = dlb2->new_event_limit;
+	port_conf->dequeue_depth = 32;
+	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
+	port_conf->event_port_cfg = 0;
+}
+
+static void
+dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
+				     uint8_t queue_id,
+				     struct rte_event_queue_conf *queue_conf)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(queue_id);
+
+	queue_conf->nb_atomic_flows = 1024;
+	queue_conf->nb_atomic_order_sequences = 64;
+	queue_conf->event_queue_cfg = 0;
+	queue_conf->priority = 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v9 12/23] event/dlb2: add queue setup

[Timothy McDaniel]

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  73 +++--
 drivers/event/dlb2/dlb2.c                  | 312 +++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |  12 +
 drivers/event/dlb2/dlb2_iface.h            |  12 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 464 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  82 +++++
 7 files changed, 926 insertions(+), 39 deletions(-)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 5f6c486..b154cac 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -47,45 +47,40 @@ setup argument and the per-port ``new_event_threshold`` argument apply as
 defined in the eventdev header file. The limit is applied to all enqueues,
 regardless of whether it will consume a directed or load-balanced credit.
 
-Load-balanced and Directed Ports
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
-does not have the same concept, but it has a similar one: ports and queues that
-are singly-linked (i.e. linked to a single queue or port, respectively).
-
-The ``rte_event_dev_info_get()`` function reports the number of available
-event ports and queues (among other things). For the DLB2 PMD, max_event_ports
-and max_event_queues report the number of available load-balanced ports and
-queues, and max_single_link_event_port_queue_pairs reports the number of
-available directed ports and queues.
-
-When a scheduling domain is created in ``rte_event_dev_configure()``, the user
-specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
-control the total number of ports (load-balanced and directed) and the number
-of directed ports. Hence, the number of requested load-balanced ports is
-``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
-specifies the total number of queues (load-balanced and directed). The number
-of directed queues comes from ``nb_single_link_event_port_queues``, since
-directed ports and queues come in pairs.
-
-When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
-whether it should be configured as a directed (the flag is set) or a
-load-balanced (the flag is unset) port. Similarly, the
-``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
-whether it is a directed or load-balanced queue.
-
-Load-balanced ports can only be linked to load-balanced queues, and directed
-ports can only be linked to directed queues. Furthermore, directed ports can
-only be linked to a single directed queue (and vice versa), and that link
-cannot change after the eventdev is started.
-
-The eventdev API does not have a directed scheduling type. To support directed
-traffic, the dlb PMD detects when an event is being sent to a directed queue
-and overrides its scheduling type. Note that the originally selected scheduling
-type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
-will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
-port.
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
 
 Flow ID
 ~~~~~~~
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index ef1c000..8c1e06d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -688,6 +688,317 @@ dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
 	queue_conf->priority = 0;
 }
 
+static int32_t
+dlb2_get_sn_allocation(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_allocation_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_set_sn_allocation(struct dlb2_eventdev *dlb2, int group, int num)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_set_sn_allocation_args cfg;
+	int ret;
+
+	cfg.num = num;
+	cfg.group = group;
+
+	ret = dlb2_iface_set_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: set_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int32_t
+dlb2_get_sn_occupancy(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_occupancy_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_occupancy(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_occupancy ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+/* Query the current sequence number allocations and, if they conflict with the
+ * requested LDB queue configuration, attempt to re-allocate sequence numbers.
+ * This is best-effort; if it fails, the PMD will attempt to configure the
+ * load-balanced queue and return an error.
+ */
+static void
+dlb2_program_sn_allocation(struct dlb2_eventdev *dlb2,
+			   const struct rte_event_queue_conf *queue_conf)
+{
+	int grp_occupancy[DLB2_NUM_SN_GROUPS];
+	int grp_alloc[DLB2_NUM_SN_GROUPS];
+	int i, sequence_numbers;
+
+	sequence_numbers = (int)queue_conf->nb_atomic_order_sequences;
+
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		int total_slots;
+
+		grp_alloc[i] = dlb2_get_sn_allocation(dlb2, i);
+		if (grp_alloc[i] < 0)
+			return;
+
+		total_slots = DLB2_MAX_LDB_SN_ALLOC / grp_alloc[i];
+
+		grp_occupancy[i] = dlb2_get_sn_occupancy(dlb2, i);
+		if (grp_occupancy[i] < 0)
+			return;
+
+		/* DLB has at least one available slot for the requested
+		 * sequence numbers, so no further configuration required.
+		 */
+		if (grp_alloc[i] == sequence_numbers &&
+		    grp_occupancy[i] < total_slots)
+			return;
+	}
+
+	/* None of the sequence number groups are configured for the requested
+	 * sequence numbers, so we have to reconfigure one of them. This is
+	 * only possible if a group is not in use.
+	 */
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		if (grp_occupancy[i] == 0)
+			break;
+	}
+
+	if (i == DLB2_NUM_SN_GROUPS) {
+		printf("[%s()] No groups with %d sequence_numbers are available or have free slots\n",
+		       __func__, sequence_numbers);
+		return;
+	}
+
+	/* Attempt to configure slot i with the requested number of sequence
+	 * numbers. Ignore the return value -- if this fails, the error will be
+	 * caught during subsequent queue configuration.
+	 */
+	dlb2_set_sn_allocation(dlb2, i, sequence_numbers);
+}
+
+static int32_t
+dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 const struct rte_event_queue_conf *evq_conf)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_queue *queue = &ev_queue->qm_queue;
+	struct dlb2_create_ldb_queue_args cfg;
+	int32_t ret;
+	uint32_t qm_qid;
+	int sched_type = -1;
+
+	if (evq_conf == NULL)
+		return -EINVAL;
+
+	if (evq_conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_ALL_TYPES) {
+		if (evq_conf->nb_atomic_order_sequences != 0)
+			sched_type = RTE_SCHED_TYPE_ORDERED;
+		else
+			sched_type = RTE_SCHED_TYPE_PARALLEL;
+	} else
+		sched_type = evq_conf->schedule_type;
+
+	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
+	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
+	cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences;
+
+	if (sched_type != RTE_SCHED_TYPE_ORDERED) {
+		cfg.num_sequence_numbers = 0;
+		cfg.num_qid_inflights = 2048;
+	}
+
+	/* App should set this to the number of hardware flows they want, not
+	 * the overall number of flows they're going to use. E.g. if app is
+	 * using 64 flows and sets compression to 64, best-case they'll get
+	 * 64 unique hashed flows in hardware.
+	 */
+	switch (evq_conf->nb_atomic_flows) {
+	/* Valid DLB2 compression levels */
+	case 64:
+	case 128:
+	case 256:
+	case 512:
+	case (1 * 1024): /* 1K */
+	case (2 * 1024): /* 2K */
+	case (4 * 1024): /* 4K */
+	case (64 * 1024): /* 64K */
+		cfg.lock_id_comp_level = evq_conf->nb_atomic_flows;
+		break;
+	default:
+		/* Invalid compression level */
+		cfg.lock_id_comp_level = 0; /* no compression */
+	}
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create LB event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	qm_qid = cfg.response.id;
+
+	/* Save off queue config for debug, resource lookups, and reconfig */
+	queue->num_qid_inflights = cfg.num_qid_inflights;
+	queue->num_atm_inflights = cfg.num_atomic_inflights;
+
+	queue->sched_type = sched_type;
+	queue->config_state = DLB2_CONFIGURED;
+
+	DLB2_LOG_DBG("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n",
+		     qm_qid,
+		     cfg.num_atomic_inflights,
+		     cfg.num_sequence_numbers,
+		     cfg.num_qid_inflights);
+
+	return qm_qid;
+}
+
+static int
+dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int32_t qm_qid;
+
+	if (queue_conf->nb_atomic_order_sequences)
+		dlb2_program_sn_allocation(dlb2, queue_conf);
+
+	qm_qid = dlb2_hw_create_ldb_queue(dlb2, ev_queue, queue_conf);
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the load-balanced queue\n");
+
+		return qm_qid;
+	}
+
+	dlb2->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int dlb2_num_dir_queues_setup(struct dlb2_eventdev *dlb2)
+{
+	int i, num = 0;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].setup_done &&
+		    dlb2->ev_queues[i].qm_queue.is_directed)
+			num++;
+	}
+
+	return num;
+}
+
+static void
+dlb2_queue_link_teardown(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue)
+{
+	struct dlb2_eventdev_port *ev_port;
+	int i, j;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (!ev_port->link[j].valid ||
+			    ev_port->link[j].queue_id != ev_queue->id)
+				continue;
+
+			ev_port->link[j].valid = false;
+			ev_port->num_links--;
+		}
+	}
+
+	ev_queue->num_links = 0;
+}
+
+static int
+dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
+			  uint8_t ev_qid,
+			  const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_queue *ev_queue;
+	int ret;
+
+	if (queue_conf == NULL)
+		return -EINVAL;
+
+	if (ev_qid >= dlb2->num_queues)
+		return -EINVAL;
+
+	ev_queue = &dlb2->ev_queues[ev_qid];
+
+	ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg &
+		RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+	ev_queue->id = ev_qid;
+	ev_queue->conf = *queue_conf;
+
+	if (!ev_queue->qm_queue.is_directed) {
+		ret = dlb2_eventdev_ldb_queue_setup(dev, ev_queue, queue_conf);
+	} else {
+		/* The directed queue isn't setup until link time, at which
+		 * point we know its directed port ID. Directed queue setup
+		 * will only fail if this queue is already setup or there are
+		 * no directed queues left to configure.
+		 */
+		ret = 0;
+
+		ev_queue->qm_queue.config_state = DLB2_NOT_CONFIGURED;
+
+		if (ev_queue->setup_done ||
+		    dlb2_num_dir_queues_setup(dlb2) == dlb2->num_dir_queues)
+			ret = -EINVAL;
+	}
+
+	/* Tear down pre-existing port->queue links */
+	if (!ret && dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_queue_link_teardown(dlb2, ev_queue);
+
+	if (!ret)
+		ev_queue->setup_done = true;
+
+	return ret;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -696,6 +1007,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a829b9b..4c07574 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -31,3 +31,15 @@ int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				struct dlb2_create_sched_domain_args *args);
 
 void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_ldb_queue_args *cfg);
+
+int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_sn_allocation_args *args);
+
+int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_set_sn_allocation_args *args);
+
+int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_occupancy_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 6663dab..4c88fe0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -30,4 +30,16 @@ extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				 struct dlb2_create_sched_domain_args *args);
 
 extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+extern int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_queue_args *cfg);
+
+extern int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_allocation_args *args);
+
+extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_set_sn_allocation_args *args);
+
+extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				  struct dlb2_get_sn_occupancy_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 63a68a6..e6ea0d7 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3509,3 +3509,467 @@ unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
 
 	return num;
 }
+
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_ldb_queue *queue,
+				     struct dlb2_create_ldb_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+	union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+	union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+	union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+	union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+	union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+	union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+	union dlb2_sys_ldb_qid_its r7 = { {0} };
+	union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+	union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+	union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+	union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+	union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+	union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+	struct dlb2_sn_group *sn_group;
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r3.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	r4.field.limit = args->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+	r5.field.limit = queue->aqed_limit;
+
+	if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+		r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+		    r5.val);
+
+	switch (args->lock_id_comp_level) {
+	case 64:
+		r6.field.compress_code = 1;
+		break;
+	case 128:
+		r6.field.compress_code = 2;
+		break;
+	case 256:
+		r6.field.compress_code = 3;
+		break;
+	case 512:
+		r6.field.compress_code = 4;
+		break;
+	case 1024:
+		r6.field.compress_code = 5;
+		break;
+	case 2048:
+		r6.field.compress_code = 6;
+		break;
+	case 4096:
+		r6.field.compress_code = 7;
+		break;
+	case 0:
+	case 65536:
+		r6.field.compress_code = 0;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+		    r6.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r7.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+		    r7.val);
+
+	r8.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+		    r8.val);
+
+	r9.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+		    r9.val);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	r10.field.qid_fid_limit = 512;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+		    r10.val);
+
+	/* Configure SNs */
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	r11.field.mode = sn_group->mode;
+	r11.field.slot = queue->sn_slot;
+	r11.field.grp  = sn_group->id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+	r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+	r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		r0.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+		r1.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+		r2.field.vqid = queue->id.virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+			    r2.val);
+	}
+
+	r13.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+				  struct dlb2_ldb_queue *queue,
+				  struct dlb2_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb2_sn_group_full(group)) {
+			slot = dlb2_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no sequence number slots available\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+				struct dlb2_hw_domain *domain,
+				struct dlb2_ldb_queue *queue,
+				struct dlb2_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_ldb_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (dlb2_list_empty(&domain->avail_ldb_queues)) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb2_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_ldb_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		    args->num_sequence_numbers);
+	DLB2_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		    args->num_qid_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		    args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+
+	dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			    __func__, __LINE__);
+		return ret;
+	}
+
+	dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_queues, &queue->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_queues, &queue->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+						unsigned int group_id,
+						unsigned long val)
+{
+	DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+	DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+	DLB2_HW_DBG(hw, "\tValue:    %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val)
+{
+	u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+	union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+	struct dlb2_sn_group *group;
+	int mode;
+
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	group = &hw->rsrcs.sn_groups[group_id];
+
+	/*
+	 * Once the first load-balanced queue using an SN group is configured,
+	 * the group cannot be changed.
+	 */
+	if (group->slot_use_bitmap != 0)
+		return -EPERM;
+
+	for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+		if (val == valid_allocations[mode])
+			break;
+
+	if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+		return -EINVAL;
+
+	group->mode = mode;
+	group->sequence_numbers_per_queue = val;
+
+	r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+	r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+	DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+	dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 285ad2c..14bb04b 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -613,3 +613,13 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
 {
 	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_ldb_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 9310f72..a6cd178 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -150,6 +150,84 @@ dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
 		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
 }
 
+static int
+dlb2_pf_ldb_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_ldb_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_ldb_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw_dev *handle,
+			 struct dlb2_get_sn_occupancy_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_number_occupancy(&dlb2_dev->hw,
+						       args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_numbers(&dlb2_dev->hw, args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_set_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_set_group_sequence_numbers(&dlb2_dev->hw, args->group,
+					      args->num);
+
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -161,6 +239,10 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
+	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
+	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
+	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v9 13/23] event/dlb2: add port setup

[Timothy McDaniel]

Configure the load balanced (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  75 +++
 drivers/event/dlb2/dlb2.c                  | 498 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   8 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 922 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  28 +
 drivers/event/dlb2/pf/dlb2_pf.c            | 180 ++++++
 7 files changed, 1719 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index b154cac..bbd6ac8 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -82,6 +82,81 @@ The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
 the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
 load-balanced queues can use the full 16-bit flow ID range.
 
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 8c1e06d..7898e42 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -999,6 +999,503 @@ dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
 	return ret;
 }
 
+static int
+dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_cq_pop_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_cq_pop_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL)	{
+		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->consume_qe = qe;
+
+	qe->qe_valid = 0;
+	qe->qe_frag = 0;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
+	 * and so on.
+	 */
+	qe->tokens = 0;	/* set at run time */
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_enqueue_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_enqueue_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL) {
+		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->int_arm_qe = qe;
+
+	/* V2 - INT ARM is CQ_TOKEN + FRAG */
+	qe->qe_valid = 0;
+	qe->qe_frag = 1;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
+{
+	int ret, sz;
+
+	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct dlb2_enqueue_qe);
+
+	qm_port->qe4 = rte_zmalloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
+
+	if (qm_port->qe4 == NULL) {
+		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	ret = dlb2_init_consume_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	return 0;
+
+error_exit:
+
+	dlb2_free_qe_mem(qm_port);
+
+	return ret;
+}
+
+static int
+dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_ldb_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_CQ_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	cfg.cq_history_list_size = DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	if (handle->cos_id == DLB2_COS_DEFAULT)
+		cfg.cos_id = 0;
+	else
+		cfg.cos_id = handle->cos_id;
+
+	cfg.cos_strict = 0;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* If there are no directed ports, the kernel driver will ignore this
+	 * port's directed credit settings. Don't use enqueue_depth if it would
+	 * require more directed credits than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2; /* back ptr */
+	/*
+	 * Allocate and init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_ldb_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(qm_port->cfg.ldb));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     qm_port->ldb_credits,
+		     qm_port->dir_credits);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
+
+	return ret;
+}
+
+static void
+dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port)
+{
+	struct dlb2_eventdev_queue *ev_queue;
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (!ev_port->link[i].valid)
+			continue;
+
+		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
+
+		ev_port->link[i].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+}
+
+static int
+dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (dlb2 == NULL || handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* Directed queues are configured at link time. */
+	cfg.queue_id = -1;
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* Don't use enqueue_depth if it would require more directed credits
+	 * than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2;  /* back ptr */
+
+	/*
+	 * Init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE to be aligned
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_dir_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = cfg.cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(qm_port->cfg.dir));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     dir_credit_high_watermark,
+		     ldb_credit_high_watermark);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_setup(struct rte_eventdev *dev,
+			 uint8_t ev_port_id,
+			 const struct rte_event_port_conf *port_conf)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_eventdev_port *ev_port;
+	int ret;
+
+	if (dev == NULL || port_conf == NULL) {
+		DLB2_LOG_ERR("Null parameter\n");
+		return -EINVAL;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
+		return -EINVAL;
+
+	if (port_conf->dequeue_depth >
+		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
+	    port_conf->enqueue_depth >
+		evdev_dlb2_default_info.max_event_port_enqueue_depth)
+		return -EINVAL;
+
+	ev_port = &dlb2->ev_ports[ev_port_id];
+	/* configured? */
+	if (ev_port->setup_done) {
+		DLB2_LOG_ERR("evport %d is already configured\n", ev_port_id);
+		return -EINVAL;
+	}
+
+	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
+		RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (!ev_port->qm_port.is_directed) {
+		ret = dlb2_hw_create_ldb_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the lB port ve portId=%d\n",
+				     ev_port_id);
+
+			return ret;
+		}
+	} else {
+		ret = dlb2_hw_create_dir_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the DIR port\n");
+			return ret;
+		}
+	}
+
+	/* Save off port config for reconfig */
+	ev_port->conf = *port_conf;
+
+	ev_port->id = ev_port_id;
+	ev_port->enq_configured = true;
+	ev_port->setup_done = true;
+	ev_port->inflight_max = port_conf->new_event_threshold;
+	ev_port->implicit_release = !(port_conf->event_port_cfg &
+		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
+	ev_port->outstanding_releases = 0;
+	ev_port->inflight_credits = 0;
+	ev_port->credit_update_quanta = RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
+	ev_port->dlb2 = dlb2; /* reverse link */
+
+	/* Tear down pre-existing port->queue links */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
+
+	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1009,6 +1506,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
+		.port_setup       = dlb2_eventdev_port_setup,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 4c07574..01818c7 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -43,3 +43,11 @@ int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				   struct dlb2_get_sn_occupancy_args *args);
+
+int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_dir_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4c88fe0..a6b923a 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -42,4 +42,12 @@ extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				  struct dlb2_get_sn_occupancy_args *args);
+
+extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_ldb_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_dir_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index e6ea0d7..a633d3e 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3973,3 +3973,925 @@ int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_ldb_port *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_pp2vas r0 = { {0} };
+	union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+		union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_ldb_port *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_ldb_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+	union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+	union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+	union dlb2_chp_hist_list_lim r5 = { {0} };
+	union dlb2_chp_hist_list_base r6 = { {0} };
+	union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+	union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+	union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+	union dlb2_sys_ldb_cq_at r10 = { {0} };
+	union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+	union dlb2_chp_ldb_cq2vas r12 = { {0} };
+	union dlb2_lsp_cq2priov r13 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r14.field.token_count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    r14.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	r5.field.limit = port->hist_list_entry_limit - 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+	r6.field.base = port->hist_list_entry_base;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	r7.field.limit = args->cq_history_list_size;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+	r8.field.push_ptr = r6.field.base;
+	r8.field.generation = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    r8.val);
+
+	r9.field.pop_ptr = r6.field.base;
+	r9.field.generation = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+	/* Disable the port's QID mappings */
+	r13.field.v = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+	return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_ldb_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb2_ldb_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+	if (ret < 0)
+		return ret;
+
+	dlb2_ldb_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_ldb_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		    args->cq_history_list_size);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+	DLB2_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		    args->cos_strict);
+}
+
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_ldb_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+		resp->status = DLB2_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		if (dlb2_list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	} else {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			if (!dlb2_list_empty(&domain->avail_ldb_ports[i]))
+				break;
+		}
+
+		if (i == DLB2_NUM_COS_DOMAINS) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ *	its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	int ret, cos_id, i;
+
+	dlb2_log_create_ldb_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->cos_strict) {
+		cos_id = args->cos_id;
+
+		port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+					  typeof(*port));
+	} else {
+		int idx;
+
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+			port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+						  typeof(*port));
+			if (port)
+				break;
+		}
+
+		cos_id = idx;
+	}
+
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (port->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_ldb_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_ports[cos_id], &port->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_ports[cos_id], &port->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_dir_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+}
+
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+			    bool vdev_req,
+			    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+
+	return NULL;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_dir_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the queue is already configured, validate
+	 * the queue ID, its domain, and whether the queue is configured.
+	 */
+	if (args->queue_id != -1) {
+		struct dlb2_dir_pq_pair *queue;
+
+		queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+						    vdev_req,
+						    domain);
+
+		if (queue == NULL || queue->domain_id.phys_id !=
+				domain->id.phys_id ||
+				!queue->queue_configured) {
+			resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the port's queue is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->queue_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_dir_pq_pair *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_dir_pp2vas r0 = { {0} };
+	union dlb2_sys_dir_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+		union dlb2_sys_dir_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_dir_pq_pair *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_dir_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+	union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+	union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+	union dlb2_sys_dir_cq_fmt r9 = { {0} };
+	union dlb2_sys_dir_cq_at r10 = { {0} };
+	union dlb2_sys_dir_cq_pasid r11 = { {0} };
+	union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r13.field.count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    r13.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.disable_wb_opt = 0;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	r9.field.keep_pf_ppid = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+	return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_dir_pq_pair *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_dir_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb2_dir_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+
+	if (ret < 0)
+		return ret;
+
+	dlb2_dir_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ *	and queue. The port/queue pair have the same ID and name.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *port;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (args->queue_id != -1)
+		port = dlb2_get_domain_used_dir_pq(args->queue_id,
+						   vdev_req,
+						   domain);
+	else
+		port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					  typeof(*port));
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_dir_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs, &port->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs, &port->domain_list);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 14bb04b..2a089f6 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -623,3 +623,31 @@ dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_ldb_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
+
+int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_dir_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a6cd178..2a82b8f 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -228,6 +228,184 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static void *
+dlb2_alloc_coherent_aligned(const struct rte_memzone **mz, uintptr_t *phys,
+			    size_t size, int align)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t core_id = rte_lcore_id();
+	unsigned int socket_id;
+
+	snprintf(mz_name, sizeof(mz_name) - 1, "event_dlb2_pf_%lx",
+		 (unsigned long)rte_get_timer_cycles());
+	if (core_id == (unsigned int)LCORE_ID_ANY)
+		core_id = rte_get_main_lcore();
+	socket_id = rte_lcore_to_socket_id(core_id);
+	*mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
+					 RTE_MEMZONE_IOVA_CONTIG, align);
+	if (*mz == NULL) {
+		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu bytes - %s\n",
+			     size, rte_strerror(rte_errno));
+		*phys = 0;
+		return NULL;
+	}
+	*phys = (*mz)->iova;
+	return (*mz)->addr;
+}
+
+static int
+dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_ldb_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret, cq_alloc_depth;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = false;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	cq_alloc_depth = RTE_MAX(cfg->cq_depth, DLB2_MIN_HARDWARE_CQ_DEPTH);
+	alloc_sz = cq_alloc_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].cq_base = (void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+	return ret;
+}
+
+static int
+dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_dir_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = true;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	alloc_sz = cfg->cq_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
+		(void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -240,6 +418,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v9 14/23] event/dlb2: add port link

[Timothy McDaniel]

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 302 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 633 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
 6 files changed, 1007 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7898e42..5068d6d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1496,6 +1496,307 @@ dlb2_eventdev_port_setup(struct rte_eventdev *dev,
 	return 0;
 }
 
+static int16_t
+dlb2_hw_map_ldb_qid_to_port(struct dlb2_hw_dev *handle,
+			    uint32_t qm_port_id,
+			    uint16_t qm_qid,
+			    uint8_t priority)
+{
+	struct dlb2_map_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	/* Build message */
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+	cfg.priority = EV_TO_DLB2_PRIO(priority);
+
+	ret = dlb2_iface_map_qid(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: map qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		DLB2_LOG_ERR("dlb2: grp=%d, qm_port=%d, qm_qid=%d prio=%d\n",
+			     handle->domain_id, cfg.port_id,
+			     cfg.qid,
+			     cfg.priority);
+	} else {
+		DLB2_LOG_DBG("dlb2: mapped queue %d to qm_port %d\n",
+			     qm_qid, qm_port_id);
+	}
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port,
+			  struct dlb2_eventdev_queue *ev_queue,
+			  uint8_t priority)
+{
+	int first_avail = -1;
+	int ret, i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid) {
+			if (ev_port->link[i].queue_id == ev_queue->id &&
+			    ev_port->link[i].priority == priority) {
+				if (ev_port->link[i].mapped)
+					return 0; /* already mapped */
+				first_avail = i;
+			}
+		} else if (first_avail == -1)
+			first_avail = i;
+	}
+	if (first_avail == -1) {
+		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
+			     ev_port->qm_port.id);
+		return -EINVAL;
+	}
+
+	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
+					  ev_port->qm_port.id,
+					  ev_queue->qm_queue.id,
+					  priority);
+
+	if (!ret)
+		ev_port->link[first_avail].mapped = true;
+
+	return ret;
+}
+
+static int32_t
+dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 int32_t qm_port_id)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_queue_args cfg;
+	int32_t ret;
+
+	/* The directed port is always configured before its queue */
+	cfg.port_id = qm_port_id;
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_dir_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	int32_t qm_qid;
+
+	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port->qm_port.id);
+
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the DIR queue\n");
+		return qm_qid;
+	}
+
+	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int
+dlb2_do_port_link(struct rte_eventdev *dev,
+		  struct dlb2_eventdev_queue *ev_queue,
+		  struct dlb2_eventdev_port *ev_port,
+		  uint8_t prio)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int err;
+
+	/* Don't link until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	if (ev_queue->qm_queue.is_directed)
+		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue, ev_port);
+	else
+		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue, prio);
+
+	if (err) {
+		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
+			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
+			     ev_queue->id, ev_port->id);
+
+		rte_errno = err;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
+			uint8_t queue_id,
+			bool link_exists,
+			int index)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	bool port_is_dir, queue_is_dir;
+
+	if (queue_id > dlb2->num_queues) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	ev_queue = &dlb2->ev_queues[queue_id];
+
+	if (!ev_queue->setup_done &&
+	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	port_is_dir = ev_port->qm_port.is_directed;
+	queue_is_dir = ev_queue->qm_queue.is_directed;
+
+	if (port_is_dir != queue_is_dir) {
+		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
+			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
+			     port_is_dir ? "DIR" : "LDB", ev_port->id);
+
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if there is space for the requested link */
+	if (!link_exists && index == -1) {
+		DLB2_LOG_ERR("no space for new link\n");
+		rte_errno = -ENOSPC;
+		return -1;
+	}
+
+	/* Check if the directed port is already linked */
+	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if the directed queue is already linked */
+	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
+			     ev_queue->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
+			const uint8_t queues[], const uint8_t priorities[],
+			uint16_t nb_links)
+
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i, j;
+
+	RTE_SET_USED(dev);
+
+	if (ev_port == NULL) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!ev_port->setup_done &&
+	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
+	 * queues pointer.
+	 */
+	if (nb_links == 0) {
+		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
+		     nb_links,
+		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
+		     ev_port->id);
+
+	for (i = 0; i < nb_links; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		uint8_t queue_id, prio;
+		bool found = false;
+		int index = -1;
+
+		queue_id = queues[i];
+		prio = priorities[i];
+
+		/* Check if the link already exists. */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].valid) {
+				if (ev_port->link[j].queue_id == queue_id) {
+					found = true;
+					index = j;
+					break;
+				}
+			} else if (index == -1) {
+				index = j;
+			}
+
+		/* could not link */
+		if (index == -1)
+			break;
+
+		/* Check if already linked at the requested priority */
+		if (found && ev_port->link[j].priority == prio)
+			continue;
+
+		if (dlb2_validate_port_link(ev_port, queue_id, found, index))
+			break; /* return index of offending queue */
+
+		ev_queue = &dlb2->ev_queues[queue_id];
+
+		if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+			break; /* return index of offending queue */
+
+		ev_queue->num_links++;
+
+		ev_port->link[index].queue_id = queue_id;
+		ev_port->link[index].priority = prio;
+		ev_port->link[index].valid = true;
+		/* Entry already exists?  If so, then must be prio change */
+		if (!found)
+			ev_port->num_links++;
+	}
+	return i;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1507,6 +1808,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_link        = dlb2_eventdev_port_link,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 01818c7..7d1ba73 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -51,3 +51,9 @@ int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 				  struct dlb2_create_dir_port_args *cfg,
 				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_dir_queue_args *cfg);
+
+int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+			  struct dlb2_map_qid_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index a6b923a..4eddd14 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -50,4 +50,10 @@ extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 struct dlb2_create_dir_port_args *cfg,
 					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+					struct dlb2_create_dir_queue_args *cfg);
+
+extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+				 struct dlb2_map_qid_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index a633d3e..14ad262 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -4895,3 +4895,636 @@ int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_dir_pq_pair *queue,
+				     struct dlb2_create_dir_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_dir_vasqid_v r0 = { {0} };
+	union dlb2_sys_dir_qid_its r1 = { {0} };
+	union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+	union dlb2_sys_dir_qid_v r5 = { {0} };
+
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r0.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r1.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+		    r1.val);
+
+	r2.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+		    r2.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		r3.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+		r4.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+	}
+
+	r5.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+	queue->queue_configured = true;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_dir_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_dir_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = dlb2_get_domain_used_dir_pq(args->port_id,
+						   vdev_req,
+						   domain);
+
+		if (port == NULL || port->domain_id.phys_id !=
+				domain->id.phys_id || !port->port_configured) {
+			resp->status = DLB2_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the queue's port is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->port_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->port_id != -1)
+		queue = dlb2_get_domain_used_dir_pq(args->port_id,
+						    vdev_req,
+						    domain);
+	else
+		queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					   typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &queue->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs,
+			      &queue->domain_list);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot,
+					      struct dlb2_map_qid_args *args)
+{
+	union dlb2_lsp_cq2priov r0;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+	r0.field.v |= 1 << slot;
+	r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+	dlb2_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+					      struct dlb2_ldb_queue *queue,
+					      struct dlb2_cmd_response *resp)
+{
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB2_QUEUE_UNMAPPED;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+			  bool vdev_req,
+			  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter)
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+			      bool vdev_req,
+			      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+
+		DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+	}
+
+	return NULL;
+}
+
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+				    u32 domain_id,
+				    struct dlb2_map_qid_args *args,
+				    struct dlb2_cmd_response *resp,
+				    bool vdev_req,
+				    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	struct dlb2_ldb_queue *queue;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB2_QID_PRIORITIES) {
+		resp->status = DLB2_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_map_qid_args *args,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	DLB2_HW_DBG(hw, "\tPriority:  %d\n",
+		    args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	int ret, i, id;
+	u8 prio;
+
+	dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_map_qid_args(hw,
+				       domain_id,
+				       args,
+				       resp,
+				       vdev_req,
+				       vdev_id);
+	if (ret)
+		return ret;
+
+	prio = args->priority;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	if (port->num_pending_removals)
+		dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].pending_priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+		if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb2_qid_map_state st;
+
+			if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+				DLB2_HW_ERR(hw,
+					    "[%s():%d] Internal error: port slot tracking failed\n",
+					    __func__, __LINE__);
+				return -EFAULT;
+			}
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb2_port_slot_state_transition(hw, port, queue,
+							      i, st);
+			if (ret)
+				return ret;
+
+			DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 2a089f6..a010e25 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -651,3 +651,13 @@ dlb2_pf_create_dir_port(struct dlb2_hw *hw,
 				       NOT_VF_REQ,
 				       PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_dir_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 2a82b8f..37be4b1 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -406,6 +406,54 @@ dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_dir_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_dir_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_dir_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
+		struct dlb2_map_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_map_qid(&dlb2_dev->hw,
+			      handle->domain_id,
+			      cfg,
+			      &response,
+			      false,
+			      0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -419,7 +467,9 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
 	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
+	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v9 15/23] event/dlb2: add port unlink and port unlinks in progress

[Timothy McDaniel]

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 163 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 283 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  51 ++++++
 5 files changed, 509 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 5068d6d..c221a61 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1797,6 +1797,166 @@ dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
 	return i;
 }
 
+static int16_t
+dlb2_hw_unmap_ldb_qid_from_port(struct dlb2_hw_dev *handle,
+				uint32_t qm_port_id,
+				uint16_t qm_qid)
+{
+	struct dlb2_unmap_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+
+	ret = dlb2_iface_unmap_qid(handle, &cfg);
+	if (ret < 0)
+		DLB2_LOG_ERR("dlb2: unmap qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_detach_ldb(struct dlb2_eventdev *dlb2,
+			    struct dlb2_eventdev_port *ev_port,
+			    struct dlb2_eventdev_queue *ev_queue)
+{
+	int ret, i;
+
+	/* Don't unlink until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid &&
+		    ev_port->link[i].queue_id == ev_queue->id)
+			break; /* found */
+	}
+
+	/* This is expected with eventdev API!
+	 * It blindly attemmpts to unmap all queues.
+	 */
+	if (i == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_LOG_DBG("dlb2: ignoring LB QID %d not mapped for qm_port %d.\n",
+			     ev_queue->qm_queue.id,
+			     ev_port->qm_port.id);
+		return 0;
+	}
+
+	ret = dlb2_hw_unmap_ldb_qid_from_port(&dlb2->qm_instance,
+					      ev_port->qm_port.id,
+					      ev_queue->qm_queue.id);
+	if (!ret)
+		ev_port->link[i].mapped = false;
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
+			  uint8_t queues[], uint16_t nb_unlinks)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (queues == NULL || nb_unlinks == 0) {
+		DLB2_LOG_DBG("dlb2: queues is NULL or nb_unlinks is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	/* FIXME: How to handle unlink on a directed port? */
+	if (ev_port->qm_port.is_directed) {
+		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
+			     ev_port->id);
+		rte_errno = 0;
+		return nb_unlinks; /* as if success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	for (i = 0; i < nb_unlinks; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		int ret, j;
+
+		if (queues[i] >= dlb2->num_queues) {
+			DLB2_LOG_ERR("dlb2: invalid queue id %d\n", queues[i]);
+			rte_errno = -EINVAL;
+			return i; /* return index of offending queue */
+		}
+
+		ev_queue = &dlb2->ev_queues[queues[i]];
+
+		/* Does a link exist? */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].queue_id == queues[i] &&
+			    ev_port->link[j].valid)
+				break;
+
+		if (j == DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+			continue;
+
+		ret = dlb2_event_queue_detach_ldb(dlb2, ev_port, ev_queue);
+		if (ret) {
+			DLB2_LOG_ERR("unlink err=%d for port %d queue %d\n",
+				     ret, ev_port->id, queues[i]);
+			rte_errno = -ENOENT;
+			return i; /* return index of offending queue */
+		}
+
+		ev_port->link[j].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+
+	return nb_unlinks;
+}
+
+static int
+dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
+				       void *event_port)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	struct dlb2_pending_port_unmaps_args cfg;
+	int ret;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	cfg.port_id = ev_port->qm_port.id;
+	dlb2 = ev_port->dlb2;
+	handle = &dlb2->qm_instance;
+	ret = dlb2_iface_pending_port_unmaps(handle, &cfg);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: num_unlinks_in_progress ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1809,6 +1969,9 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
 		.port_link        = dlb2_eventdev_port_link,
+		.port_unlink      = dlb2_eventdev_port_unlink,
+		.port_unlinks_in_progress =
+				    dlb2_eventdev_port_unlinks_in_progress,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 7d1ba73..bd241f3 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -57,3 +57,9 @@ int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 			  struct dlb2_map_qid_args *cfg);
+
+int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+			    struct dlb2_unmap_qid_args *cfg);
+
+int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4eddd14..e892036 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -56,4 +56,10 @@ extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 				 struct dlb2_map_qid_args *cfg);
+
+extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+				   struct dlb2_unmap_qid_args *cfg);
+
+extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 14ad262..4a3d96d 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5528,3 +5528,286 @@ int dlb2_hw_map_qid(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_unmap_qid_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+		DLB2_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			    hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+				      u32 domain_id,
+				      struct dlb2_unmap_qid_args *args,
+				      struct dlb2_cmd_response *resp,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	struct dlb2_ldb_port *port;
+	int slot;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			    __func__, args->qid);
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		return 0;
+
+	resp->status = DLB2_ST_INVALID_QID;
+	return -EINVAL;
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	bool unmap_complete;
+	int i, ret, id;
+
+	dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_unmap_qid_args(hw,
+					 domain_id,
+					 args,
+					 resp,
+					 vdev_req,
+					 vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+		st = DLB2_QUEUE_UNMAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		st = DLB2_QUEUE_UNMAP_IN_PROG;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB2_QUEUE_MAPPED;
+	if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: no available CQ slots\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asynchronous procedure. It requires
+	 * stopping the DLB2 from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb2_ldb_port_cq_disable(hw, port);
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete && !os_worker_active(hw))
+		os_schedule_work(hw);
+
+unmap_qid_done:
+	resp->status = 0;
+
+	return 0;
+}
+
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+				  struct dlb2_pending_port_unmaps_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+
+	dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 37be4b1..33ea73e 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -454,6 +454,55 @@ dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw_dev *handle,
+		  struct dlb2_unmap_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_unmap_qid(&dlb2_dev->hw,
+				handle->domain_id,
+				cfg,
+				&response,
+				false,
+				0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
+			    struct dlb2_pending_port_unmaps_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_pending_port_unmaps(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -470,6 +519,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
+	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v9 16/23] event/dlb2: add eventdev start

[Timothy McDaniel]

Add support for the eventdev start entry point.
We delay initializing some resources until
eventdev start, since the number of linked queues can be
used to determine if we are dealing with a ldb or dir resource.
If this is a device restart, then the previous configuration
will be reapplied.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 133 +++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   3 +
 drivers/event/dlb2/dlb2_iface.h            |   3 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 123 ++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +++
 drivers/event/dlb2/pf/dlb2_pf.c            |  25 ++++++
 6 files changed, 297 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index c221a61..051836d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1957,6 +1957,138 @@ dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
 	return cfg.response.id;
 }
 
+static int
+dlb2_eventdev_reapply_configuration(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret, i;
+
+	/* If an event queue or port was previously configured, but hasn't been
+	 * reconfigured, reapply its original configuration.
+	 */
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+
+		ev_queue = &dlb2->ev_queues[i];
+
+		if (ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_queue_setup(dev, i, &ev_queue->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure queue %d", i);
+			return ret;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+
+		if (ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_port_setup(dev, i, &ev_port->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure ev_port %d",
+				     i);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_apply_port_links(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int i;
+
+	/* Perform requested port->queue links */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+		int j;
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			struct dlb2_eventdev_queue *ev_queue;
+			uint8_t prio, queue_id;
+
+			if (!ev_port->link[j].valid)
+				continue;
+
+			prio = ev_port->link[j].priority;
+			queue_id = ev_port->link[j].queue_id;
+
+			if (dlb2_validate_port_link(ev_port, queue_id, true, j))
+				return -EINVAL;
+
+			ev_queue = &dlb2->ev_queues[queue_id];
+
+			if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_start(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_start_domain_args cfg;
+	int ret, i;
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_ERR("bad state %d for dev_start\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return -EINVAL;
+	}
+	dlb2->run_state = DLB2_RUN_STATE_STARTING;
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	/* If the device was configured more than once, some event ports and/or
+	 * queues may need to be reconfigured.
+	 */
+	ret = dlb2_eventdev_reapply_configuration(dev);
+	if (ret)
+		return ret;
+
+	/* The DLB PMD delays port links until the device is started. */
+	ret = dlb2_eventdev_apply_port_links(dev);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		if (!dlb2->ev_ports[i].setup_done) {
+			DLB2_LOG_ERR("dlb2: port %d not setup", i);
+			return -ESTALE;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0) {
+			DLB2_LOG_ERR("dlb2: queue %d is not linked", i);
+			return -ENOLINK;
+		}
+	}
+
+	ret = dlb2_iface_sched_domain_start(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: sched_domain_start ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STARTED;
+	DLB2_LOG_DBG("dlb2: sched_domain_start completed OK\n");
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1964,6 +2096,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.dev_start        = dlb2_eventdev_start,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index bd241f3..a86191d 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -63,3 +63,6 @@ int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				     struct dlb2_start_domain_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index e892036..7ead374 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -62,4 +62,7 @@ extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				struct dlb2_start_domain_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 4a3d96d..c14a2f3 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5811,3 +5811,126 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 struct dlb2_cmd_response *resp,
+					 bool vdev_req,
+					 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_start_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 start domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_hw_start_domain() - Lock the domain configuration
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @arg: User-provided arguments (unused, here for ioctl callback template).
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int
+dlb2_hw_start_domain(struct dlb2_hw *hw,
+		     u32 domain_id,
+		     __attribute((unused)) struct dlb2_start_domain_args *arg,
+		     struct dlb2_cmd_response *resp,
+		     bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_queue;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+	RTE_SET_USED(arg);
+	RTE_SET_USED(iter);
+
+	dlb2_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb2_verify_start_domain_args(hw,
+					    domain_id,
+					    resp,
+					    vdev_req,
+					    vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB2 will drop all
+	 * incoming traffic to those queues.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, ldb_queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r0.val);
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+	}
+
+	dlb2_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index a010e25..aee8336 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -661,3 +661,13 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_start_domain(struct dlb2_hw *hw,
+		     u32 id,
+		     struct dlb2_start_domain_args *args,
+		     struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_start_domain(hw, id, args, resp, NOT_VF_REQ,
+				    PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 33ea73e..fff2e57 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -503,6 +503,30 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
 
 	return ret;
 }
+
+static int
+dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
+			   struct dlb2_start_domain_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_start_domain(&dlb2_dev->hw,
+				   handle->domain_id,
+				   cfg,
+				   &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -520,6 +544,7 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v9 17/23] event/dlb2: add enqueue and its burst variants

[Timothy McDaniel]

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst | 118 +++++++++
 drivers/event/dlb2/dlb2.c     | 540 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 658 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index bbd6ac8..aa8bf01 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -163,6 +163,124 @@ Flow ID
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Hardware Credits
+~~~~~~~~~~~~~~~~
+
+DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
+event storage, with each unit of storage represented by a credit. A port spends
+a credit to enqueue an event, and hardware refills the ports with credits as the
+events are scheduled to ports. Refills come from credit pools, and each port is
+a member of a load-balanced credit pool and a directed credit pool. The
+load-balanced credits are used to enqueue to load-balanced queues, and directed
+credits are used for directed queues.
+
+A DLB2 eventdev contains one load-balanced and one directed credit pool. These
+pools' sizes are controlled by the nb_events_limit field in struct
+rte_event_dev_config. The load-balanced pool is sized to contain
+nb_events_limit credits, and the directed pool is sized to contain
+nb_events_limit/4 credits. The directed pool size can be overridden with the
+num_dir_credits vdev argument, like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,num_dir_credits=<value>
+
+This can be used if the default allocation is too low or too high for the
+specific application needs. The PMD also supports a vdev arg that limits the
+max_num_events reported by rte_event_dev_info_get():
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,max_num_events=<value>
+
+By default, max_num_events is reported as the total available load-balanced
+credits. If multiple DLB2-based applications are being used, it may be desirable
+to control how many load-balanced credits each application uses, particularly
+when application(s) are written to configure nb_events_limit equal to the
+reported max_num_events.
+
+Each port is a member of both credit pools. A port's credit allocation is
+defined by its low watermark, high watermark, and refill quanta. These three
+parameters are calculated by the dlb PMD like so:
+
+- The load-balanced high watermark is set to the port's enqueue_depth.
+  The directed high watermark is set to the minimum of the enqueue_depth and
+  the directed pool size divided by the total number of ports.
+- The refill quanta is set to half the high watermark.
+- The low watermark is set to the minimum of 16 and the refill quanta.
+
+When the eventdev is started, each port is pre-allocated a high watermark's
+worth of credits. For example, if an eventdev contains four ports with enqueue
+depths of 32 and a load-balanced credit pool size of 4096, each port will start
+with 32 load-balanced credits, and there will be 3968 credits available to
+replenish the ports. Thus, a single port is not capable of enqueueing up to the
+nb_events_limit (without any events being dequeued), since the other ports are
+retaining their initial credit allocation; in short, all ports must enqueue in
+order to reach the limit.
+
+If a port attempts to enqueue and has no credits available, the enqueue
+operation will fail and the application must retry the enqueue. Credits are
+replenished asynchronously by the DLB2 hardware.
+
+Software Credits
+~~~~~~~~~~~~~~~~
+
+The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
+credit scheme on top of the hardware credit scheme in order to comply with
+the per-port backpressure described in the eventdev API.
+
+The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
+credit when it enqueues to a queue, and credits are later replenished after the
+events are dequeued and released.
+
+In the software credit scheme, a credit is consumed when a new (.op =
+RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
+replenished when the event is released from the system (either explicitly with
+RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
+
+In this model, an event is "in the system" from its first enqueue into eventdev
+until it is last dequeued. If the event goes through multiple event queues, it
+is still considered "in the system" while a worker thread is processing it.
+
+A port will fail to enqueue if the number of events in the system exceeds its
+``new_event_threshold`` (specified at port setup time). A port will also fail
+to enqueue if it lacks enough hardware credits to enqueue; load-balanced
+credits are used to enqueue to a load-balanced queue, and directed credits are
+used to enqueue to a directed queue.
+
+The out-of-credit situations are typically transient, and an eventdev
+application using the DLB2 ought to retry its enqueues if they fail.
+If enqueue fails, DLB2 PMD sets rte_errno as follows:
+
+- -ENOSPC: Credit exhaustion (either hardware or software)
+- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
+
+Depending on the pipeline the application has constructed, it's possible to
+enter a credit deadlock scenario wherein the worker thread lacks the credit
+to enqueue an event, and it must dequeue an event before it can recover the
+credit. If the worker thread retries its enqueue indefinitely, it will not
+make forward progress. Such deadlock is possible if the application has event
+"loops", in which an event in dequeued from queue A and later enqueued back to
+queue A.
+
+Due to this, workers should stop retrying after a time, release the events it
+is attempting to enqueue, and dequeue more events. It is important that the
+worker release the events and don't simply set them aside to retry the enqueue
+again later, because the port has limited history list size (by default, twice
+the port's dequeue_depth).
+
+Priority
+~~~~~~~~
+
+The DLB2 supports event priority and per-port queue service priority, as
+described in the eventdev header file. The DLB2 does not support 'global' event
+queue priority established at queue creation time.
+
+DLB2 supports 8 event and queue service priority levels. For both priority
+types, the PMD uses the upper three bits of the priority field to determine the
+DLB2 priority, discarding the 5 least significant bits. The 5 least significant
+event priority bits are not preserved when an event is enqueued.
+
 Reconfiguration
 ~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 051836d..4f5cf25 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2089,6 +2089,540 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
 	return 0;
 }
 
+static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
+	{
+		/* Load-balanced cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
+	},
+	{
+		/* Directed cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
+	},
+};
+
+static inline uint32_t
+dlb2_port_credits_get(struct dlb2_port *qm_port,
+		      enum dlb2_hw_queue_types type)
+{
+	uint32_t credits = *qm_port->credit_pool[type];
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	if (unlikely(credits < batch_size))
+		batch_size = credits;
+
+	if (likely(credits &&
+		   __atomic_compare_exchange_n(
+			qm_port->credit_pool[type],
+			&credits, credits - batch_size, false,
+			__ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
+		return batch_size;
+	else
+		return 0;
+}
+
+static inline void
+dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port)
+{
+	uint16_t quanta = ev_port->credit_update_quanta;
+
+	if (ev_port->inflight_credits >= quanta * 2) {
+		/* Replenish credits, saving one quanta for enqueues */
+		uint16_t val = ev_port->inflight_credits - quanta;
+
+		__atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits -= val;
+	}
+}
+
+static inline int
+dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
+						__ATOMIC_SEQ_CST);
+	const int num = 1;
+
+	if (unlikely(ev_port->inflight_max < sw_inflights)) {
+		DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
+		rte_errno = -ENOSPC;
+		return 1;
+	}
+
+	if (ev_port->inflight_credits < num) {
+		/* check if event enqueue brings ev_port over max threshold */
+		uint32_t credit_update_quanta = ev_port->credit_update_quanta;
+
+		if (sw_inflights + credit_update_quanta >
+				dlb2->new_event_limit) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_new_event_limit,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+
+		__atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
+				   __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits += (credit_update_quanta);
+
+		if (ev_port->inflight_credits < num) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_inflight_credits,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_ldb_credits == 0)) {
+		qm_port->cached_ldb_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_LDB_QUEUE);
+		if (unlikely(qm_port->cached_ldb_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
+			1);
+			DLB2_LOG_DBG("ldb credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_dir_credits == 0)) {
+		qm_port->cached_dir_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_DIR_QUEUE);
+		if (unlikely(qm_port->cached_dir_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
+			1);
+			DLB2_LOG_DBG("dir credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static __rte_always_inline void
+dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
+	      struct process_local_port_data *port_data)
+{
+	dlb2_movdir64b(port_data->pp_addr, qe4);
+}
+
+static inline void
+dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
+		   bool do_sfence,
+		   struct process_local_port_data *port_data)
+{
+	/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
+	 * application writes complete before enqueueing the QE.
+	 */
+	if (do_sfence)
+		rte_wmb();
+
+	dlb2_pp_write(qm_port->qe4, port_data);
+}
+
+static inline void
+dlb2_event_build_hcws(struct dlb2_port *qm_port,
+		      const struct rte_event ev[],
+		      int num,
+		      uint8_t *sched_type,
+		      uint8_t *queue_id)
+{
+	struct dlb2_enqueue_qe *qe;
+	uint16_t sched_word[4];
+	__m128i sse_qe[2];
+	int i;
+
+	qe = qm_port->qe4;
+
+	sse_qe[0] = _mm_setzero_si128();
+	sse_qe[1] = _mm_setzero_si128();
+
+	switch (num) {
+	case 4:
+		/* Construct the metadata portion of two HCWs in one 128b SSE
+		 * register. HCW metadata is constructed in the SSE registers
+		 * like so:
+		 * sse_qe[0][63:0]:   qe[0]'s metadata
+		 * sse_qe[0][127:64]: qe[1]'s metadata
+		 * sse_qe[1][63:0]:   qe[2]'s metadata
+		 * sse_qe[1][127:64]: qe[3]'s metadata
+		 */
+
+		/* Convert the event operation into a command byte and store it
+		 * in the metadata:
+		 * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
+		 * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
+		 * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
+		 * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
+		 */
+#define DLB2_QE_CMD_BYTE 7
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[0].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[1].op],
+				DLB2_QE_CMD_BYTE + 8);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[2].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[3].op],
+				DLB2_QE_CMD_BYTE + 8);
+
+		/* Store priority, scheduling type, and queue ID in the sched
+		 * word array because these values are re-used when the
+		 * destination is a directed queue.
+		 */
+		sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
+				sched_type[0] << 8 |
+				queue_id[0];
+		sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
+				sched_type[1] << 8 |
+				queue_id[1];
+		sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
+				sched_type[2] << 8 |
+				queue_id[2];
+		sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
+				sched_type[3] << 8 |
+				queue_id[3];
+
+		/* Store the event priority, scheduling type, and queue ID in
+		 * the metadata:
+		 * sse_qe[0][31:16] = sched_word[0]
+		 * sse_qe[0][95:80] = sched_word[1]
+		 * sse_qe[1][31:16] = sched_word[2]
+		 * sse_qe[1][95:80] = sched_word[3]
+		 */
+#define DLB2_QE_QID_SCHED_WORD 1
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[0],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[1],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[2],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[3],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+
+		/* If the destination is a load-balanced queue, store the lock
+		 * ID. If it is a directed queue, DLB places this field in
+		 * bytes 10-11 of the received QE, so we format it accordingly:
+		 * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
+		 * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
+		 * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
+		 * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
+		 */
+#define DLB2_QE_LOCK_ID_WORD 2
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[0] == DLB2_SCHED_DIRECTED) ?
+					sched_word[0] : ev[0].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[1] == DLB2_SCHED_DIRECTED) ?
+					sched_word[1] : ev[1].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[2] == DLB2_SCHED_DIRECTED) ?
+					sched_word[2] : ev[2].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[3] == DLB2_SCHED_DIRECTED) ?
+					sched_word[3] : ev[3].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+
+		/* Store the event type and sub event type in the metadata:
+		 * sse_qe[0][15:0]  = flow_id[0]
+		 * sse_qe[0][79:64] = flow_id[1]
+		 * sse_qe[1][15:0]  = flow_id[2]
+		 * sse_qe[1][79:64] = flow_id[3]
+		 */
+#define DLB2_QE_EV_TYPE_WORD 0
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[0].sub_event_type << 8 |
+						ev[0].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[1].sub_event_type << 8 |
+						ev[1].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[2].sub_event_type << 8 |
+						ev[2].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[3].sub_event_type << 8 |
+						ev[3].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+
+		/* Store the metadata to memory (use the double-precision
+		 * _mm_storeh_pd because there is no integer function for
+		 * storing the upper 64b):
+		 * qe[0] metadata = sse_qe[0][63:0]
+		 * qe[1] metadata = sse_qe[0][127:64]
+		 * qe[2] metadata = sse_qe[1][63:0]
+		 * qe[3] metadata = sse_qe[1][127:64]
+		 */
+		_mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
+		_mm_storeh_pd((double *)&qe[1].u.opaque_data,
+			      (__m128d)sse_qe[0]);
+		_mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
+		_mm_storeh_pd((double *)&qe[3].u.opaque_data,
+			      (__m128d)sse_qe[1]);
+
+		qe[0].data = ev[0].u64;
+		qe[1].data = ev[1].u64;
+		qe[2].data = ev[2].u64;
+		qe[3].data = ev[3].u64;
+
+		break;
+	case 3:
+	case 2:
+	case 1:
+		/* At least one QE will be valid, so only zero out three */
+		qe[1].cmd_byte = 0;
+		qe[2].cmd_byte = 0;
+		qe[3].cmd_byte = 0;
+
+		for (i = 0; i < num; i++) {
+			qe[i].cmd_byte =
+				cmd_byte_map[qm_port->is_directed][ev[i].op];
+			qe[i].sched_type = sched_type[i];
+			qe[i].data = ev[i].u64;
+			qe[i].qid = queue_id[i];
+			qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
+			qe[i].lock_id = ev[i].flow_id;
+			if (sched_type[i] == DLB2_SCHED_DIRECTED) {
+				struct dlb2_msg_info *info =
+					(struct dlb2_msg_info *)&qe[i].lock_id;
+
+				info->qid = queue_id[i];
+				info->sched_type = DLB2_SCHED_DIRECTED;
+				info->priority = qe[i].priority;
+			}
+			qe[i].u.event_type.major = ev[i].event_type;
+			qe[i].u.event_type.sub = ev[i].sub_event_type;
+		}
+		break;
+	}
+}
+
+static inline int
+dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
+			struct dlb2_port *qm_port,
+			const struct rte_event ev[],
+			uint8_t *sched_type,
+			uint8_t *queue_id)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	uint16_t *cached_credits = NULL;
+	struct dlb2_queue *qm_queue;
+
+	ev_queue = &dlb2->ev_queues[ev->queue_id];
+	qm_queue = &ev_queue->qm_queue;
+	*queue_id = qm_queue->id;
+
+	/* Ignore sched_type and hardware credits on release events */
+	if (ev->op == RTE_EVENT_OP_RELEASE)
+		goto op_check;
+
+	if (!qm_queue->is_directed) {
+		/* Load balanced destination queue */
+
+		if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_ldb_credits;
+
+		switch (ev->sched_type) {
+		case RTE_SCHED_TYPE_ORDERED:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
+			if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
+				DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
+					     *queue_id);
+				rte_errno = -EINVAL;
+				return 1;
+			}
+			*sched_type = DLB2_SCHED_ORDERED;
+			break;
+		case RTE_SCHED_TYPE_ATOMIC:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
+			*sched_type = DLB2_SCHED_ATOMIC;
+			break;
+		case RTE_SCHED_TYPE_PARALLEL:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
+			if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
+				*sched_type = DLB2_SCHED_ORDERED;
+			else
+				*sched_type = DLB2_SCHED_UNORDERED;
+			break;
+		default:
+			DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
+			DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
+			rte_errno = -EINVAL;
+			return 1;
+		}
+	} else {
+		/* Directed destination queue */
+
+		if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_dir_credits;
+
+		DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
+
+		*sched_type = DLB2_SCHED_DIRECTED;
+	}
+
+op_check:
+	switch (ev->op) {
+	case RTE_EVENT_OP_NEW:
+		/* Check that a sw credit is available */
+		if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		ev_port->inflight_credits--;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_FORWARD:
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_RELEASE:
+		ev_port->inflight_credits++;
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+
+		/* Replenish s/w credits if enough are cached */
+		dlb2_replenish_sw_credits(dlb2, ev_port);
+		break;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+	if (ev->op != RTE_EVENT_OP_RELEASE) {
+		DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
+		DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
+	}
+#endif
+
+	return 0;
+}
+
+static inline uint16_t
+dlb2_event_enqueue_burst(void *event_port,
+			 const struct rte_event events[],
+			 uint16_t num)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct process_local_port_data *port_data;
+	int i, cnt;
+
+	RTE_ASSERT(ev_port->enq_configured);
+	RTE_ASSERT(events != NULL);
+
+	cnt = 0;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
+		int j = 0;
+
+		for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
+			const struct rte_event *ev = &events[i + j];
+
+			if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
+						    &sched_types[j],
+						    &queue_ids[j]))
+				break;
+		}
+
+		if (j == 0)
+			break;
+
+		dlb2_event_build_hcws(qm_port, &events[i], j,
+				      sched_types, queue_ids);
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+
+		if (j < DLB2_NUM_QES_PER_CACHE_LINE)
+			break;
+	}
+
+	return cnt;
+}
+
+static inline uint16_t
+dlb2_event_enqueue(void *event_port,
+		   const struct rte_event events[])
+{
+	return dlb2_event_enqueue_burst(event_port, events, 1);
+}
+
+static uint16_t
+dlb2_event_enqueue_new_burst(void *event_port,
+			     const struct rte_event events[],
+			     uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
+static uint16_t
+dlb2_event_enqueue_forward_burst(void *event_port,
+				 const struct rte_event events[],
+				 uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2112,7 +2646,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
 	};
 
+	/* Expose PMD's eventdev interface */
+
 	dev->dev_ops = &dlb2_eventdev_entry_ops;
+	dev->enqueue = dlb2_event_enqueue;
+	dev->enqueue_burst = dlb2_event_enqueue_burst;
+	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
+	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v9 18/23] event/dlb2: add dequeue and its burst variants

[Timothy McDaniel]

Add support for dequeue, dequeue_burst, ...

DLB2 does not currently support interrupts, but instead use
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB2 supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst |  21 ++
 drivers/event/dlb2/dlb2.c     | 783 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 804 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index aa8bf01..b9f57fd 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -314,6 +314,27 @@ The PMD does not support the following configuration sequences:
 This sequence is not supported because the event device must be reconfigured
 before its ports or queues can be.
 
+Deferred Scheduling
+~~~~~~~~~~~~~~~~~~~
+
+The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
+dequeued event before returning from rte_event_dequeue_burst(). This frees the
+corresponding entries in the CQ, which enables the DLB2 to schedule more events
+to it.
+
+To support applications seeking finer-grained scheduling control -- for example
+deferring scheduling to get the best possible priority scheduling and
+load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
+the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
+call. This mode only applies to load-balanced event ports with dequeue depth of
+1.
+
+To enable deferred scheduling, use the defer_sched vdev argument like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,defer_sched=on
+
 Atomic Inflights Allocation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 4f5cf25..7cd227f 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -26,6 +26,7 @@
 #include <rte_log.h>
 #include <rte_malloc.h>
 #include <rte_mbuf.h>
+#include <rte_power_intrinsics.h>
 #include <rte_prefetch.h>
 #include <rte_ring.h>
 #include <rte_string_fns.h>
@@ -2227,6 +2228,32 @@ dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
 	dlb2_movdir64b(port_data->pp_addr, qe4);
 }
 
+static inline int
+dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_cq_pop_qe *qe;
+
+	RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
+
+	qe = qm_port->consume_qe;
+
+	qe->tokens = num - 1;
+
+	/* No store fence needed since no pointer is being sent, and CQ token
+	 * pops can be safely reordered with other HCWs.
+	 */
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	dlb2_movntdq_single(port_data->pp_addr, qe);
+
+	DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
+
+	qm_port->owed_tokens = 0;
+
+	return 0;
+}
+
 static inline void
 dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
 		   bool do_sfence,
@@ -2623,9 +2650,756 @@ dlb2_event_enqueue_forward_burst(void *event_port,
 	return dlb2_event_enqueue_burst(event_port, events, num);
 }
 
+static inline void
+dlb2_port_credits_inc(struct dlb2_port *qm_port, int num)
+{
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	/* increment port credits, and return to pool if exceeds threshold */
+	if (!qm_port->is_directed) {
+		qm_port->cached_ldb_credits += num;
+		if (qm_port->cached_ldb_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_LDB_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_ldb_credits -= batch_size;
+		}
+	} else {
+		qm_port->cached_dir_credits += num;
+		if (qm_port->cached_dir_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_DIR_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_dir_credits -= batch_size;
+		}
+	}
+}
+
+static inline int
+dlb2_dequeue_wait(struct dlb2_eventdev *dlb2,
+		  struct dlb2_eventdev_port *ev_port,
+		  struct dlb2_port *qm_port,
+		  uint64_t timeout,
+		  uint64_t start_ticks)
+{
+	struct process_local_port_data *port_data;
+	uint64_t elapsed_ticks;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	elapsed_ticks = rte_get_timer_cycles() - start_ticks;
+
+	/* Wait/poll time expired */
+	if (elapsed_ticks >= timeout) {
+		return 1;
+	} else if (dlb2->umwait_allowed) {
+		volatile struct dlb2_dequeue_qe *cq_base;
+		union {
+			uint64_t raw_qe[2];
+			struct dlb2_dequeue_qe qe;
+		} qe_mask;
+		uint64_t expected_value;
+		volatile uint64_t *monitor_addr;
+
+		qe_mask.qe.cq_gen = 1; /* set mask */
+
+		cq_base = port_data->cq_base;
+		monitor_addr = (volatile uint64_t *)(volatile void *)
+			&cq_base[qm_port->cq_idx];
+		monitor_addr++; /* cq_gen bit is in second 64bit location */
+
+		if (qm_port->gen_bit)
+			expected_value = qe_mask.raw_qe[1];
+		else
+			expected_value = 0;
+
+		rte_power_monitor(monitor_addr, expected_value,
+				  qe_mask.raw_qe[1], timeout + start_ticks,
+				  sizeof(uint64_t));
+
+		DLB2_INC_STAT(ev_port->stats.traffic.rx_umonitor_umwait, 1);
+	} else {
+		uint64_t poll_interval = RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL;
+		uint64_t curr_ticks = rte_get_timer_cycles();
+		uint64_t init_ticks = curr_ticks;
+
+		while ((curr_ticks - start_ticks < timeout) &&
+		       (curr_ticks - init_ticks < poll_interval))
+			curr_ticks = rte_get_timer_cycles();
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
+			 struct dlb2_port *qm_port,
+			 struct rte_event *events,
+			 struct dlb2_dequeue_qe *qes,
+			 int cnt)
+{
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	int i, num, evq_id;
+
+	for (i = 0, num = 0; i < cnt; i++) {
+		struct dlb2_dequeue_qe *qe = &qes[i];
+		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
+			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+			[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+		};
+
+		/* Fill in event information.
+		 * Note that flow_id must be embedded in the data by
+		 * the app, such as the mbuf RSS hash field if the data
+		 * buffer is a mbuf.
+		 */
+		if (unlikely(qe->error)) {
+			DLB2_LOG_ERR("QE error bit ON\n");
+			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
+			dlb2_consume_qe_immediate(qm_port, 1);
+			continue; /* Ignore */
+		}
+
+		events[num].u64 = qe->data;
+		events[num].flow_id = qe->flow_id;
+		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
+		events[num].event_type = qe->u.event_type.major;
+		events[num].sub_event_type = qe->u.event_type.sub;
+		events[num].sched_type = sched_type_map[qe->sched_type];
+		events[num].impl_opaque = qe->qid_depth;
+
+		/* qid not preserved for directed queues */
+		if (qm_port->is_directed)
+			evq_id = ev_port->link[0].queue_id;
+		else
+			evq_id = qid_mappings[qe->qid];
+
+		events[num].queue_id = evq_id;
+		DLB2_INC_STAT(
+			ev_port->stats.queue[evq_id].qid_depth[qe->qid_depth],
+			1);
+		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type], 1);
+		num++;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num);
+
+	return num;
+}
+
+static inline int
+dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
+			      struct dlb2_port *qm_port,
+			      struct rte_event *events,
+			      struct dlb2_dequeue_qe *qes)
+{
+	int sched_type_map[] = {
+		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+	};
+	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	__m128i sse_evt[2];
+
+	/* In the unlikely case that any of the QE error bits are set, process
+	 * them one at a time.
+	 */
+	if (unlikely(qes[0].error || qes[1].error ||
+		     qes[2].error || qes[3].error))
+		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
+						 qes, num_events);
+
+	events[0].u64 = qes[0].data;
+	events[1].u64 = qes[1].data;
+	events[2].u64 = qes[2].data;
+	events[3].u64 = qes[3].data;
+
+	/* Construct the metadata portion of two struct rte_events
+	 * in one 128b SSE register. Event metadata is constructed in the SSE
+	 * registers like so:
+	 * sse_evt[0][63:0]:   event[0]'s metadata
+	 * sse_evt[0][127:64]: event[1]'s metadata
+	 * sse_evt[1][63:0]:   event[2]'s metadata
+	 * sse_evt[1][127:64]: event[3]'s metadata
+	 */
+	sse_evt[0] = _mm_setzero_si128();
+	sse_evt[1] = _mm_setzero_si128();
+
+	/* Convert the hardware queue ID to an event queue ID and store it in
+	 * the metadata:
+	 * sse_evt[0][47:40]   = qid_mappings[qes[0].qid]
+	 * sse_evt[0][111:104] = qid_mappings[qes[1].qid]
+	 * sse_evt[1][47:40]   = qid_mappings[qes[2].qid]
+	 * sse_evt[1][111:104] = qid_mappings[qes[3].qid]
+	 */
+#define DLB_EVENT_QUEUE_ID_BYTE 5
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[0].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[1].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[2].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[3].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+
+	/* Convert the hardware priority to an event priority and store it in
+	 * the metadata, while also returning the queue depth status
+	 * value captured by the hardware, storing it in impl_opaque, which can
+	 * be read by the application but not modified
+	 * sse_evt[0][55:48]   = DLB2_TO_EV_PRIO(qes[0].priority)
+	 * sse_evt[0][63:56]   = qes[0].qid_depth
+	 * sse_evt[0][119:112] = DLB2_TO_EV_PRIO(qes[1].priority)
+	 * sse_evt[0][127:120] = qes[1].qid_depth
+	 * sse_evt[1][55:48]   = DLB2_TO_EV_PRIO(qes[2].priority)
+	 * sse_evt[1][63:56]   = qes[2].qid_depth
+	 * sse_evt[1][119:112] = DLB2_TO_EV_PRIO(qes[3].priority)
+	 * sse_evt[1][127:120] = qes[3].qid_depth
+	 */
+#define DLB_EVENT_PRIO_IMPL_OPAQUE_WORD 3
+#define DLB_BYTE_SHIFT 8
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[0].priority) |
+			(qes[0].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[1].priority) |
+			(qes[1].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[2].priority) |
+			(qes[2].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[3].priority) |
+			(qes[3].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+
+	/* Write the event type, sub event type, and flow_id to the event
+	 * metadata.
+	 * sse_evt[0][31:0]   = qes[0].flow_id |
+	 *			qes[0].u.event_type.major << 28 |
+	 *			qes[0].u.event_type.sub << 20;
+	 * sse_evt[0][95:64]  = qes[1].flow_id |
+	 *			qes[1].u.event_type.major << 28 |
+	 *			qes[1].u.event_type.sub << 20;
+	 * sse_evt[1][31:0]   = qes[2].flow_id |
+	 *			qes[2].u.event_type.major << 28 |
+	 *			qes[2].u.event_type.sub << 20;
+	 * sse_evt[1][95:64]  = qes[3].flow_id |
+	 *			qes[3].u.event_type.major << 28 |
+	 *			qes[3].u.event_type.sub << 20;
+	 */
+#define DLB_EVENT_EV_TYPE_DW 0
+#define DLB_EVENT_EV_TYPE_SHIFT 28
+#define DLB_EVENT_SUB_EV_TYPE_SHIFT 20
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[0].flow_id |
+			qes[0].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[0].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[1].flow_id |
+			qes[1].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[1].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[2].flow_id |
+			qes[2].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[2].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[3].flow_id |
+			qes[3].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT  |
+			qes[3].u.event_type.sub << DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+
+	/* Write the sched type to the event metadata. 'op' and 'rsvd' are not
+	 * set:
+	 * sse_evt[0][39:32]  = sched_type_map[qes[0].sched_type] << 6
+	 * sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6
+	 * sse_evt[1][39:32]  = sched_type_map[qes[2].sched_type] << 6
+	 * sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6
+	 */
+#define DLB_EVENT_SCHED_TYPE_BYTE 4
+#define DLB_EVENT_SCHED_TYPE_SHIFT 6
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[0].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[1].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[2].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[3].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+
+	/* Store the metadata to the event (use the double-precision
+	 * _mm_storeh_pd because there is no integer function for storing the
+	 * upper 64b):
+	 * events[0].event = sse_evt[0][63:0]
+	 * events[1].event = sse_evt[0][127:64]
+	 * events[2].event = sse_evt[1][63:0]
+	 * events[3].event = sse_evt[1][127:64]
+	 */
+	_mm_storel_epi64((__m128i *)&events[0].event, sse_evt[0]);
+	_mm_storeh_pd((double *)&events[1].event, (__m128d) sse_evt[0]);
+	_mm_storel_epi64((__m128i *)&events[2].event, sse_evt[1]);
+	_mm_storeh_pd((double *)&events[3].event, (__m128d) sse_evt[1]);
+
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[0].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[1].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[2].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[3].sched_type], 1);
+
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[0].queue_id].
+			qid_depth[qes[0].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[1].queue_id].
+			qid_depth[qes[1].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[2].queue_id].
+			qid_depth[qes[2].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[3].queue_id].
+			qid_depth[qes[3].qid_depth],
+		1);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num_events);
+
+	return num_events;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe_sparse(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe)
+{
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	uint8_t xor_mask[2] = {0x0F, 0x00};
+	const uint8_t and_mask = 0x0F;
+	__m128i *qes = (__m128i *)qe;
+	uint8_t gen_bits, gen_bit;
+	uintptr_t addr[4];
+	uint16_t idx;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+
+	idx = qm_port->cq_idx;
+
+	/* Load the next 4 QEs */
+	addr[0] = (uintptr_t)&cq_addr[idx];
+	addr[1] = (uintptr_t)&cq_addr[(idx +  4) & qm_port->cq_depth_mask];
+	addr[2] = (uintptr_t)&cq_addr[(idx +  8) & qm_port->cq_depth_mask];
+	addr[3] = (uintptr_t)&cq_addr[(idx + 12) & qm_port->cq_depth_mask];
+
+	/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */
+	rte_prefetch0(&cq_addr[(idx + 16) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 20) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 24) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 28) & qm_port->cq_depth_mask]);
+
+	/* Correct the xor_mask for wrap-around QEs */
+	gen_bit = qm_port->gen_bit;
+	xor_mask[gen_bit] ^= !!((idx +  4) > qm_port->cq_depth_mask) << 1;
+	xor_mask[gen_bit] ^= !!((idx +  8) > qm_port->cq_depth_mask) << 2;
+	xor_mask[gen_bit] ^= !!((idx + 12) > qm_port->cq_depth_mask) << 3;
+
+	/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is
+	 * valid, then QEs[0:N-1] are too.
+	 */
+	qes[3] = _mm_load_si128((__m128i *)(void *)addr[3]);
+	rte_compiler_barrier();
+	qes[2] = _mm_load_si128((__m128i *)(void *)addr[2]);
+	rte_compiler_barrier();
+	qes[1] = _mm_load_si128((__m128i *)(void *)addr[1]);
+	rte_compiler_barrier();
+	qes[0] = _mm_load_si128((__m128i *)(void *)addr[0]);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[gen_bit];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask;
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline void
+dlb2_inc_cq_idx(struct dlb2_port *qm_port, int cnt)
+{
+	uint16_t idx = qm_port->cq_idx_unmasked + cnt;
+
+	qm_port->cq_idx_unmasked = idx;
+	qm_port->cq_idx = idx & qm_port->cq_depth_mask;
+	qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) & 0x1;
+}
+
+static int
+dlb2_event_release(struct dlb2_eventdev *dlb2,
+		   uint8_t port_id,
+		   int n)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_eventdev_port *ev_port;
+	struct dlb2_port *qm_port;
+	int i, cnt;
+
+	if (port_id > dlb2->num_ports) {
+		DLB2_LOG_ERR("Invalid port id %d in dlb2-event_release\n",
+			     port_id);
+		rte_errno = -EINVAL;
+		return rte_errno;
+	}
+
+	ev_port = &dlb2->ev_ports[port_id];
+	qm_port = &ev_port->qm_port;
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	cnt = 0;
+
+	if (qm_port->is_directed) {
+		cnt = n;
+		goto sw_credit_update;
+	}
+
+	for (i = 0; i < n; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		int j;
+
+		/* Zero-out QEs */
+		qm_port->qe4[0].cmd_byte = 0;
+		qm_port->qe4[1].cmd_byte = 0;
+		qm_port->qe4[2].cmd_byte = 0;
+		qm_port->qe4[3].cmd_byte = 0;
+
+		for (j = 0; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++)
+			qm_port->qe4[j].cmd_byte = DLB2_COMP_CMD_BYTE;
+
+		qm_port->issued_releases += j;
+
+		if (j == 0)
+			break;
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+	}
+
+sw_credit_update:
+	/* each release returns one credit */
+	if (!ev_port->outstanding_releases) {
+		DLB2_LOG_ERR("Unrecoverable application error. Outstanding releases underflowed.\n");
+		rte_errno = -ENOTRECOVERABLE;
+		return rte_errno;
+	}
+
+	ev_port->outstanding_releases -= cnt;
+	ev_port->inflight_credits += cnt;
+
+	/* Replenish s/w credits if enough releases are performed */
+	dlb2_replenish_sw_credits(dlb2, ev_port);
+	return 0;
+}
+
+static inline int16_t
+dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
+		       struct dlb2_eventdev_port *ev_port,
+		       struct rte_event *events,
+		       uint16_t max_num,
+		       uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe_sparse(qm_port, qes);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail << 2);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							      qm_port,
+							      &events[num],
+							      &qes[0]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							 qm_port,
+							 &events[num],
+							 &qes[0],
+							 num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe,
+	     uint8_t *offset)
+{
+	uint8_t xor_mask[2][4] = { {0x0F, 0x0E, 0x0C, 0x08},
+				   {0x00, 0x01, 0x03, 0x07} };
+	uint8_t and_mask[4] = {0x0F, 0x0E, 0x0C, 0x08};
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	__m128i *qes = (__m128i *)qe;
+	uint64_t *cache_line_base;
+	uint8_t gen_bits;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	cq_addr = &cq_addr[qm_port->cq_idx];
+
+	cache_line_base = (void *)(((uintptr_t)cq_addr) & ~0x3F);
+	*offset = ((uintptr_t)cq_addr & 0x30) >> 4;
+
+	/* Load the next CQ cache line from memory. Pack these reads as tight
+	 * as possible to reduce the chance that DLB invalidates the line while
+	 * the CPU is reading it. Read the cache line backwards to ensure that
+	 * if QE[N] (N > 0) is valid, then QEs[0:N-1] are too.
+	 *
+	 * (Valid QEs start at &qe[offset])
+	 */
+	qes[3] = _mm_load_si128((__m128i *)&cache_line_base[6]);
+	qes[2] = _mm_load_si128((__m128i *)&cache_line_base[4]);
+	qes[1] = _mm_load_si128((__m128i *)&cache_line_base[2]);
+	qes[0] = _mm_load_si128((__m128i *)&cache_line_base[0]);
+
+	/* Evict the cache line ASAP */
+	dlb2_cldemote(cache_line_base);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[qm_port->gen_bit][*offset];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask[*offset];
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline int16_t
+dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
+		struct dlb2_eventdev_port *ev_port,
+		struct rte_event *events,
+		uint16_t max_num,
+		uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t offset;
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe(qm_port, qes, &offset);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							     qm_port,
+							     &events[num],
+							     &qes[offset]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							qm_port,
+							&events[num],
+							&qes[offset],
+							num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static uint16_t
+dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
+			 uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue(void *event_port, struct rte_event *ev, uint64_t wait)
+{
+	return dlb2_event_dequeue_burst(event_port, ev, 1, wait);
+}
+
+static uint16_t
+dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
+				uint16_t num, uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
+			  uint64_t wait)
+{
+	return dlb2_event_dequeue_burst_sparse(event_port, ev, 1, wait);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
+	struct dlb2_eventdev *dlb2;
+
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
@@ -2653,6 +3427,15 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	dev->enqueue_burst = dlb2_event_enqueue_burst;
 	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
 	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
+
+	dlb2 = dev->data->dev_private;
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE) {
+		dev->dequeue = dlb2_event_dequeue_sparse;
+		dev->dequeue_burst = dlb2_event_dequeue_burst_sparse;
+	} else {
+		dev->dequeue = dlb2_event_dequeue;
+		dev->dequeue_burst = dlb2_event_dequeue_burst;
+	}
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v9 19/23] event/dlb2: add eventdev stop and close

[Timothy McDaniel]

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 257 +++++++++++++++++++++++++++--
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c |  91 ++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  52 ++++++
 5 files changed, 396 insertions(+), 16 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7cd227f..85b7d34 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -71,21 +71,6 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
-/*
- * DUMMY - added so that xstats path will compile/link.
- * Will be replaced by real version in a subsequent
- * patch.
- */
-uint32_t
-dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
-		     struct dlb2_eventdev_queue *queue)
-{
-	RTE_SET_USED(dlb2);
-	RTE_SET_USED(queue);
-
-	return 0;
-}
-
 static void
 dlb2_free_qe_mem(struct dlb2_port *qm_port)
 {
@@ -1878,7 +1863,6 @@ dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
 		return 0; /* Ignore and return success */
 	}
 
-	/* FIXME: How to handle unlink on a directed port? */
 	if (ev_port->qm_port.is_directed) {
 		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
 			     ev_port->id);
@@ -3396,6 +3380,245 @@ dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
 }
 
 static void
+dlb2_flush_port(struct rte_eventdev *dev, int port_id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	eventdev_stop_flush_t flush;
+	struct rte_event ev;
+	uint8_t dev_id;
+	void *arg;
+	int i;
+
+	flush = dev->dev_ops->dev_stop_flush;
+	dev_id = dev->data->dev_id;
+	arg = dev->data->dev_stop_flush_arg;
+
+	while (rte_event_dequeue_burst(dev_id, port_id, &ev, 1, 0)) {
+		if (flush)
+			flush(dev_id, ev, arg);
+
+		if (dlb2->ev_ports[port_id].qm_port.is_directed)
+			continue;
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+	}
+
+	/* Enqueue any additional outstanding releases */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = dlb2->ev_ports[port_id].outstanding_releases; i > 0; i--)
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+}
+
+static uint32_t
+dlb2_get_ldb_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_ldb_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_ldb_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_ldb_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static uint32_t
+dlb2_get_dir_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_dir_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_dir_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_dir_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	if (queue->qm_queue.is_directed)
+		return dlb2_get_dir_queue_depth(dlb2, queue);
+	else
+		return dlb2_get_ldb_queue_depth(dlb2, queue);
+}
+
+static bool
+dlb2_queue_is_empty(struct dlb2_eventdev *dlb2,
+		    struct dlb2_eventdev_queue *queue)
+{
+	return dlb2_get_queue_depth(dlb2, queue) == 0;
+}
+
+static bool
+dlb2_linked_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0)
+			continue;
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static bool
+dlb2_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static void
+dlb2_drain(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_port *ev_port = NULL;
+	uint8_t dev_id;
+	int i;
+
+	dev_id = dev->data->dev_id;
+
+	while (!dlb2_linked_queues_empty(dlb2)) {
+		/* Flush all the ev_ports, which will drain all their connected
+		 * queues.
+		 */
+		for (i = 0; i < dlb2->num_ports; i++)
+			dlb2_flush_port(dev, i);
+	}
+
+	/* The queues are empty, but there may be events left in the ports. */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_flush_port(dev, i);
+
+	/* If the domain's queues are empty, we're done. */
+	if (dlb2_queues_empty(dlb2))
+		return;
+
+	/* Else, there must be at least one unlinked load-balanced queue.
+	 * Select a load-balanced port with which to drain the unlinked
+	 * queue(s).
+	 */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		if (!ev_port->qm_port.is_directed)
+			break;
+	}
+
+	if (i == dlb2->num_ports) {
+		DLB2_LOG_ERR("internal error: no LDB ev_ports\n");
+		return;
+	}
+
+	rte_errno = 0;
+	rte_event_port_unlink(dev_id, ev_port->id, NULL, 0);
+
+	if (rte_errno) {
+		DLB2_LOG_ERR("internal error: failed to unlink ev_port %d\n",
+			     ev_port->id);
+		return;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		uint8_t qid, prio;
+		int ret;
+
+		if (dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			continue;
+
+		qid = i;
+		prio = 0;
+
+		/* Link the ev_port to the queue */
+		ret = rte_event_port_link(dev_id, ev_port->id, &qid, &prio, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to link ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+
+		/* Flush the queue */
+		while (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			dlb2_flush_port(dev, ev_port->id);
+
+		/* Drain any extant events in the ev_port. */
+		dlb2_flush_port(dev, ev_port->id);
+
+		/* Unlink the ev_port from the queue */
+		ret = rte_event_port_unlink(dev_id, ev_port->id, &qid, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to unlink ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+	}
+}
+
+static void
+dlb2_eventdev_stop(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_DBG("Internal error: already stopped\n");
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	} else if (dlb2->run_state != DLB2_RUN_STATE_STARTED) {
+		DLB2_LOG_ERR("Internal error: bad state %d for dev_stop\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPING;
+
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	dlb2_drain(dev);
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPED;
+}
+
+static int
+dlb2_eventdev_close(struct rte_eventdev *dev)
+{
+	dlb2_hw_reset_sched_domain(dev, false);
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3405,6 +3628,8 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.dev_start        = dlb2_eventdev_start,
+		.dev_stop         = dlb2_eventdev_stop,
+		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a86191d..5471dd8 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -66,3 +66,9 @@ int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				     struct dlb2_start_domain_args *cfg);
+
+int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 7ead374..b508eb0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -65,4 +65,10 @@ extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				struct dlb2_start_domain_args *cfg);
+
+extern int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+extern int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index c14a2f3..ae5ef2f 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5934,3 +5934,94 @@ dlb2_hw_start_domain(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_dir_queue_depth(hw, queue);
+
+	return 0;
+}
+
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+
+	dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index fff2e57..632c4e0 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -527,6 +527,56 @@ dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_ldb_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_ldb_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_dir_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret = 0;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_dir_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -544,6 +594,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth;
+	dlb2_iface_get_dir_queue_depth = dlb2_pf_get_dir_queue_depth;
 	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v9 20/23] event/dlb2: add PMD's token pop public interface

[Timothy McDaniel]

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/api/doxy-api-index.md         |  3 +-
 drivers/event/dlb2/dlb2.c         | 65 +++++++++++++++++++++++++++++++----
 drivers/event/dlb2/dlb2_priv.h    |  3 ++
 drivers/event/dlb2/meson.build    |  5 ++-
 drivers/event/dlb2/rte_pmd_dlb2.c | 39 +++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2.h | 72 +++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/version.map    |  6 ++++
 7 files changed, 185 insertions(+), 8 deletions(-)
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index a9c12d1..96e84db 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -51,7 +51,8 @@ The public API headers are grouped by topics:
   [dpaa2_mempool]      (@ref rte_dpaa2_mempool.h),
   [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
-  [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
+  [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h),
+  [dlb2]               (@ref rte_pmd_dlb2.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 85b7d34..7a22231 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1197,7 +1197,7 @@ dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
 	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
 
 	qm_port->dequeue_depth = dequeue_depth;
-
+	qm_port->token_pop_thresh = dequeue_depth;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -1365,6 +1365,8 @@ dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
 
 	qm_port->dequeue_depth = dequeue_depth;
 
+	/* Directed ports are auto-pop, by default. */
+	qm_port->token_pop_mode = AUTO_POP;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -2253,6 +2255,18 @@ dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
 }
 
 static inline void
+dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
+{
+	struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
+	int num = qm_port->owed_tokens;
+
+	qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
+	qe[idx].tokens = num - 1;
+
+	qm_port->owed_tokens = 0;
+}
+
+static inline void
 dlb2_event_build_hcws(struct dlb2_port *qm_port,
 		      const struct rte_event ev[],
 		      int num,
@@ -2600,6 +2614,14 @@ dlb2_event_enqueue_burst(void *event_port,
 		dlb2_event_build_hcws(qm_port, &events[i], j,
 				      sched_types, queue_ids);
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
@@ -2608,6 +2630,11 @@ dlb2_event_enqueue_burst(void *event_port,
 			break;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
 	return cnt;
 }
 
@@ -3085,11 +3112,25 @@ dlb2_event_release(struct dlb2_eventdev *dlb2,
 		if (j == 0)
 			break;
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
 sw_credit_update:
 	/* each release returns one credit */
 	if (!ev_port->outstanding_releases) {
@@ -3173,8 +3214,8 @@ dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3300,8 +3341,8 @@ dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3316,6 +3357,7 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 			 uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3331,6 +3373,9 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3350,6 +3395,7 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 				uint16_t num, uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3365,6 +3411,9 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3669,7 +3718,7 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    struct dlb2_devargs *dlb2_args)
 {
 	struct dlb2_eventdev *dlb2;
-	int err;
+	int err, i;
 
 	dlb2 = dev->data->dev_private;
 
@@ -3719,6 +3768,10 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Initialize each port's token pop mode */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
+		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 61567a6..b73cf3f 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -12,6 +12,7 @@
 #include <rte_config.h>
 #include "dlb2_user.h"
 #include "dlb2_log.h"
+#include "rte_pmd_dlb2.h"
 
 #ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
 #define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
@@ -290,6 +291,7 @@ struct dlb2_port {
 	bool gen_bit;
 	uint16_t dir_credits;
 	uint32_t dequeue_depth;
+	enum dlb2_token_pop_mode token_pop_mode;
 	union dlb2_port_config cfg;
 	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
 	uint16_t cached_ldb_credits;
@@ -298,6 +300,7 @@ struct dlb2_port {
 	bool int_armed;
 	uint16_t owed_tokens;
 	int16_t issued_releases;
+	int16_t token_pop_thresh;
 	int cq_depth;
 	uint16_t cq_idx;
 	uint16_t cq_idx_unmasked;
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index dc16c96..b734ff6 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,3 +1,4 @@
+
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
@@ -12,7 +13,9 @@ sources = files('dlb2.c',
 		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
-		'pf/base/dlb2_resource.c'
+		'pf/base/dlb2_resource.c',
+		'rte_pmd_dlb2.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
+install_headers('rte_pmd_dlb2.h')
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
new file mode 100644
index 0000000..b09b585
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "rte_pmd_dlb2.h"
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode)
+{
+	struct dlb2_eventdev *dlb2;
+	struct rte_eventdev *dev;
+
+	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+	dev = &rte_eventdevs[dev_id];
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (mode >= NUM_TOKEN_POP_MODES)
+		return -EINVAL;
+
+	/* The event device must be configured, but not yet started */
+	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
+		return -EINVAL;
+
+	/* The token pop mode must be set before configuring the port */
+	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
+		return -EINVAL;
+
+	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
new file mode 100644
index 0000000..74399db
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+/*!
+ *  @file      rte_pmd_dlb2.h
+ *
+ *  @brief     DLB PMD-specific functions
+ */
+
+#ifndef _RTE_PMD_DLB2_H_
+#define _RTE_PMD_DLB2_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Selects the token pop mode for a DLB2 port.
+ */
+enum dlb2_token_pop_mode {
+	/* Pop the CQ tokens immediately after dequeueing. */
+	AUTO_POP,
+	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
+	 * Supported on load-balanced ports only.
+	 */
+	DELAYED_POP,
+	/* Pop the CQ tokens during next dequeue operation. */
+	DEFERRED_POP,
+
+	/* NUM_TOKEN_POP_MODES must be last */
+	NUM_TOKEN_POP_MODES
+};
+
+/*!
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Configure the token pop mode for a DLB2 port. By default, all ports use
+ * AUTO_POP. This function must be called before calling rte_event_port_setup()
+ * for the port, but after calling rte_event_dev_configure().
+ *
+ * @param dev_id
+ *    The identifier of the event device.
+ * @param port_id
+ *    The identifier of the event port.
+ * @param mode
+ *    The token pop mode.
+ *
+ * @return
+ * - 0: Success
+ * - EINVAL: Invalid dev_id, port_id, or mode
+ * - EINVAL: The DLB2 is not configured, is already running, or the port is
+ *   already setup
+ */
+
+__rte_experimental
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_DLB2_H_ */
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
index 4a76d1d..b1e4dff 100644
--- a/drivers/event/dlb2/version.map
+++ b/drivers/event/dlb2/version.map
@@ -1,3 +1,9 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmd_dlb2_set_token_pop_mode;
+};
-- 
2.6.4

[dpdk-dev] [PATCH v9 21/23] event/dlb2: add PMD self-tests

[Timothy McDaniel]

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 app/test/test_eventdev.c           |    7 +
 drivers/event/dlb2/dlb2.c          |    1 +
 drivers/event/dlb2/dlb2_selftest.c | 1524 ++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build     |    3 +-
 4 files changed, 1534 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c

diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
index 62019c1..bcfaa53 100644
--- a/app/test/test_eventdev.c
+++ b/app/test/test_eventdev.c
@@ -1030,6 +1030,12 @@ test_eventdev_selftest_dpaa2(void)
 	return test_eventdev_selftest_impl("event_dpaa2", "");
 }
 
+static int
+test_eventdev_selftest_dlb2(void)
+{
+	return test_eventdev_selftest_impl("dlb2_event", "");
+}
+
 REGISTER_TEST_COMMAND(eventdev_common_autotest, test_eventdev_common);
 REGISTER_TEST_COMMAND(eventdev_selftest_sw, test_eventdev_selftest_sw);
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
@@ -1037,3 +1043,4 @@ REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
 		test_eventdev_selftest_octeontx2);
 REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2, test_eventdev_selftest_dpaa2);
+REGISTER_TEST_COMMAND(eventdev_selftest_dlb2, test_eventdev_selftest_dlb2);
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7a22231..99a22e6 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3692,6 +3692,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
 		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+		.dev_selftest     = test_dlb2_eventdev,
 	};
 
 	/* Expose PMD's eventdev interface */
diff --git a/drivers/event/dlb2/dlb2_selftest.c b/drivers/event/dlb2/dlb2_selftest.c
new file mode 100644
index 0000000..e300067
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_selftest.c
@@ -0,0 +1,1524 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <unistd.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_eventdev.h>
+#include <rte_pause.h>
+
+#include "dlb2_priv.h"
+#include "rte_pmd_dlb2.h"
+
+#define MAX_PORTS 32
+#define MAX_QIDS 32
+#define DEFAULT_NUM_SEQ_NUMS 64
+
+static struct rte_mempool *eventdev_func_mempool;
+static int evdev;
+
+struct test {
+	struct rte_mempool *mbuf_pool;
+	int nb_qids;
+};
+
+/* initialization and config */
+static inline int
+init(struct test *t, int nb_queues, int nb_ports)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	memset(t, 0, sizeof(*t));
+
+	t->mbuf_pool = eventdev_func_mempool;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = nb_queues;
+	config.nb_event_ports = nb_ports;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0)
+		printf("%d: Error configuring device\n", __LINE__);
+
+	return ret;
+}
+
+static inline int
+create_ports(int num_ports)
+{
+	int i;
+
+	if (num_ports > MAX_PORTS)
+		return -1;
+
+	for (i = 0; i < num_ports; i++) {
+		struct rte_event_port_conf conf;
+
+		if (rte_event_port_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default port conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		if (rte_event_port_setup(evdev, i, &conf) < 0) {
+			printf("%d: Error setting up port %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+create_lb_qids(struct test *t, int num_qids, uint32_t flags)
+{
+	int i;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		struct rte_event_queue_conf conf;
+
+		if (rte_event_queue_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default queue conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		conf.schedule_type = flags;
+
+		if (conf.schedule_type == RTE_SCHED_TYPE_PARALLEL)
+			conf.nb_atomic_order_sequences = 0;
+		else
+			conf.nb_atomic_order_sequences = DEFAULT_NUM_SEQ_NUMS;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+static inline int
+create_atomic_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ATOMIC);
+}
+
+/* destruction */
+static inline void
+cleanup(void)
+{
+	int ret = 0;
+
+	rte_event_dev_stop(evdev);
+	ret = rte_event_dev_close(evdev);
+
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+};
+
+static inline int
+enqueue_timeout(uint8_t port_id, struct rte_event *ev, uint64_t tmo_us)
+{
+	const uint64_t start = rte_get_timer_cycles();
+	const uint64_t ticks = (tmo_us * rte_get_timer_hz()) / 1E6;
+
+	while ((rte_get_timer_cycles() - start) < ticks) {
+		if (rte_event_enqueue_burst(evdev, port_id, ev, 1) == 1)
+			return 0;
+
+		if (rte_errno != -ENOSPC) {
+			printf("enqueue_burst returned rte_errno %d\n",
+			       rte_errno);
+			return -1;
+		}
+	}
+	printf("%s time out\n", __func__);
+	return -1;
+}
+
+static void
+flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused)
+{
+	rte_pktmbuf_free(event.mbuf);
+}
+
+static int
+test_stop_flush(struct test *t) /* test to check we can properly flush events */
+{
+	struct rte_event ev;
+	uint32_t dequeue_depth;
+	unsigned int i, count;
+	uint8_t queue_id;
+
+	ev.op = RTE_EVENT_OP_NEW;
+
+	if (init(t, 2, 1) < 0 ||
+	    create_ports(1) < 0 ||
+	    create_atomic_qids(t, 2) < 0) {
+		printf("%d: Error initializing device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_port_link(evdev, 0, NULL, NULL, 0) != 2) {
+		printf("%d: Error linking queues to the port\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: Error with start call\n", __LINE__);
+		goto err;
+	}
+
+	/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked
+	 * queue.
+	 */
+	queue_id = 1;
+
+	if (rte_event_port_unlink(evdev, 0, &queue_id, 1) != 1) {
+		printf("%d: Error unlinking queue 1 from port\n", __LINE__);
+		goto err;
+	}
+
+	count = rte_mempool_avail_count(t->mbuf_pool);
+
+	if (rte_event_port_attr_get(evdev,
+				    0,
+				    RTE_EVENT_PORT_ATTR_DEQ_DEPTH,
+				    &dequeue_depth)) {
+		printf("%d: Error retrieveing dequeue depth\n", __LINE__);
+		goto err;
+	}
+
+	/* Send QEs to queue 0 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 0;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Send QEs to queue 1 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 1;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Now the DLB is scheduling events from the port to the IQ, and at
+	 * least one event should be remaining in each queue.
+	 */
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, flush, NULL)) {
+		printf("%d: Error installing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+
+	if (count != rte_mempool_avail_count(t->mbuf_pool)) {
+		printf("%d: Error executing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, NULL, NULL)) {
+		printf("%d: Error uninstalling the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	return 0;
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_single_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Create a directed port */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) == 0) {
+		printf("%d: port 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Create a load-balanced port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a directed queue */
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed queue */
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) == 0) {
+		printf("%d: queue 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a load-balanced queue */
+	queue_conf.event_queue_cfg = 0;
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to link directed and load-balanced resources */
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 0 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 1 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Link ports to queues */
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+#define NUM_LDB_PORTS 64
+#define NUM_LDB_QUEUES 32
+
+static int
+test_info_get(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	config.nb_event_ports = info.max_event_ports;
+	config.nb_event_queues = NUM_LDB_QUEUES + info.max_event_ports / 2;
+	config.nb_single_link_event_port_queues = info.max_event_ports / 2;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		goto err;
+	}
+
+	/* The DLB2 PMD only reports load-balanced ports and queues in its
+	 * info_get function. Confirm that these values don't include the
+	 * directed port or queue counts.
+	 */
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret) {
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+		return -1;
+	}
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+static int
+test_reconfiguration_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 2 LDB queues */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+			printf("%d: P0->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P1->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+			printf("%d: P1->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Configure device with 2 DIR ports and 2 DIR queues */
+	config.nb_single_link_event_port_queues = 2;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q1 */
+	queue_id = 1;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: P0->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Link P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: P1->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_queues = 5;
+	config.nb_event_ports = 5;
+	config.nb_single_link_event_port_queues = 1;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < config.nb_event_queues - 1; i++) {
+		port_conf.event_port_cfg = 0;
+		queue_conf.event_queue_cfg = 0;
+
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: P%d->Q%d link expected to succeed\n",
+			       __LINE__, i, i);
+			goto err;
+		}
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+		printf("%d: port %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+		printf("%d: queue %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	queue_id = i;
+
+	if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+		printf("%d: P%d->Q%d link expected to succeed\n",
+		       __LINE__, i, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_ports += 1;
+
+	/* Reconfigure device with 1 more load-balanced port */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Configure the new port */
+	if (rte_event_port_setup(evdev, config.nb_event_ports - 1,
+				 &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_load_balanced_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_directed_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 DIR port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (ev.queue_id != 0) {
+		printf("%d: invalid dequeued event queue ID (%d)\n",
+		       __LINE__, ev.queue_id);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_deferred_sched(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	const int num_events = 128;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 1 queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 1, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.dequeue_depth = 1;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.schedule_type = RTE_SCHED_TYPE_PARALLEL;
+	queue_conf.nb_atomic_order_sequences = 0;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 128 NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_PARALLEL;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue one event from port 0 */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue (and release) all other events from port 1. Deferred
+	 * scheduling ensures no other events are scheduled to port 0 without a
+	 * subsequent rte_event_dequeue_burst() call.
+	 */
+	for (i = 0; i < num_events - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 1, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		if (rte_event_enqueue_burst(evdev, 1, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_delayed_pop(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	int ret, i, num_events;
+	struct rte_event ev;
+	uint8_t queue_id;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DELAYED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	num_events = 2 * port_conf.dequeue_depth;
+
+	/* Enqueue 2 * dequeue_depth NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue dequeue_depth events but only release dequeue_depth - 2.
+	 * Delayed pop won't perform the pop and no more events will be
+	 * scheduled.
+	 */
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = 0; i < port_conf.dequeue_depth - 2; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	ret = rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout);
+	if (ret != 0) {
+		printf("%d: event dequeue expected to fail (ret = %d)\n",
+		       __LINE__, ret);
+		goto err;
+	}
+
+	/* Release one more event. This will trigger the token pop, and
+	 * dequeue_depth - 1 more events will be scheduled to the device.
+	 */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: RELEASE enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 0) {
+		printf("%d: event dequeue expected to fail\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+do_selftest(void)
+{
+	struct test t;
+	int ret;
+
+	/* Only create mbuf pool once, reuse for each test run */
+	if (!eventdev_func_mempool) {
+		eventdev_func_mempool =
+			rte_pktmbuf_pool_create("EVENTDEV_DLB2_ST_POOL",
+						(1 << 12), /* 4k buffers */
+						32 /*MBUF_CACHE_SIZE*/,
+						0,
+						512, /* use very small mbufs */
+						rte_socket_id());
+		if (!eventdev_func_mempool) {
+			printf("ERROR creating mempool\n");
+			goto test_fail;
+		}
+	}
+	t.mbuf_pool = eventdev_func_mempool;
+
+	printf("*** Running Stop Flush test...\n");
+	ret = test_stop_flush(&t);
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Single Link test...\n");
+	ret = test_single_link();
+	if (ret != 0) {
+		printf("ERROR - Single Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Info Get test...\n");
+	ret = test_info_get();
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Reconfiguration Link test...\n");
+	ret = test_reconfiguration_link();
+	if (ret != 0) {
+		printf("ERROR - Reconfiguration Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Load-Balanced Traffic test...\n");
+	ret = test_load_balanced_traffic();
+	if (ret != 0) {
+		printf("ERROR - Load-Balanced Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Directed Traffic test...\n");
+	ret = test_directed_traffic();
+	if (ret != 0) {
+		printf("ERROR - Directed Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Deferred Scheduling test...\n");
+	ret = test_deferred_sched();
+	if (ret != 0) {
+		printf("ERROR - Deferred Scheduling test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Delayed Pop test...\n");
+	ret = test_delayed_pop();
+	if (ret != 0) {
+		printf("ERROR - Delayed Pop test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	return 0;
+
+test_fail:
+	return -1;
+}
+
+int
+test_dlb2_eventdev(void)
+{
+	const char *dlb2_eventdev_name = "dlb2_event";
+	uint8_t num_evdevs = rte_event_dev_count();
+	int i, ret = 0;
+	int found = 0, skipped = 0, passed = 0, failed = 0;
+	struct rte_event_dev_info info;
+
+	for (i = 0; found + skipped < num_evdevs && i < RTE_EVENT_MAX_DEVS;
+	     i++) {
+		ret = rte_event_dev_info_get(i, &info);
+		if (ret < 0)
+			continue;
+
+		/* skip non-dlb2 event devices */
+		if (strncmp(info.driver_name, dlb2_eventdev_name,
+			    sizeof(*info.driver_name)) != 0) {
+			skipped++;
+			continue;
+		}
+
+		evdev = rte_event_dev_get_dev_id(info.driver_name);
+		if (evdev < 0) {
+			printf("Could not get dev_id for eventdev with name %s, i=%d\n",
+			       info.driver_name, i);
+			skipped++;
+			continue;
+		}
+		found++;
+		printf("Running selftest on eventdev %s\n", info.driver_name);
+		ret = do_selftest();
+		if (ret == 0) {
+			passed++;
+			printf("Selftest passed for eventdev %s\n",
+			       info.driver_name);
+		} else {
+			failed++;
+			printf("Selftest failed for eventdev %s, err=%d\n",
+			       info.driver_name, ret);
+		}
+	}
+
+	printf("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n",
+	       found, skipped, passed, failed);
+	return ret;
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index b734ff6..340bc57 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -14,7 +14,8 @@ sources = files('dlb2.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c',
-		'rte_pmd_dlb2.c'
+		'rte_pmd_dlb2.c',
+		'dlb2_selftest.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v9 22/23] event/dlb2: add queue and port release

[Timothy McDaniel]

DLB does not support reconfiguring individual queues
or ports on the fly. The entire device must be reconfigured.
Previously allocated port QE and memzone memory
is freed in this patch.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 99a22e6..d9b13a1 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3668,6 +3668,28 @@ dlb2_eventdev_close(struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(id);
+
+	/* This function intentionally left blank. */
+}
+
+static void
+dlb2_eventdev_port_release(void *port)
+{
+	struct dlb2_eventdev_port *ev_port = port;
+	struct dlb2_port *qm_port;
+
+	if (ev_port) {
+		qm_port = &ev_port->qm_port;
+		if (qm_port->config_state == DLB2_CONFIGURED)
+			dlb2_free_qe_mem(qm_port);
+	}
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3681,8 +3703,10 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
+		.queue_release    = dlb2_eventdev_queue_release,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_release     = dlb2_eventdev_port_release,
 		.port_link        = dlb2_eventdev_port_link,
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
-- 
2.6.4

[dpdk-dev] [PATCH v9 23/23] event/dlb2: add timeout ticks entry point

[Timothy McDaniel]

Adds the timeout ticks conversion function.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index d9b13a1..d12cbf7 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3689,6 +3689,18 @@ dlb2_eventdev_port_release(void *port)
 	}
 }
 
+static int
+dlb2_eventdev_timeout_ticks(struct rte_eventdev *dev, uint64_t ns,
+			    uint64_t *timeout_ticks)
+{
+	RTE_SET_USED(dev);
+	uint64_t cycles_per_ns = rte_get_timer_hz() / 1E9;
+
+	*timeout_ticks = ns * cycles_per_ns;
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -3711,6 +3723,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
 				    dlb2_eventdev_port_unlinks_in_progress,
+		.timeout_ticks    = dlb2_eventdev_timeout_ticks,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v10 00/23] Add DLB2 PMD

[Timothy McDaniel]

The following patch series adds support for a new eventdev PMD. The DLB2
PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
hardware.
The DLB2 is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB2 hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB2 ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a
directed queue. Additionally, even though LDB ports may link multiple
queues,
the number of queues that may be linked is limited by hardware.

While reviewing the code, please be aware that this PMD has full
control over the DLB2 hardware. Intel will be extending the DLB2 PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.
Note that the DLB2 hardware is a successor of the DLB hardware, and
as such is structured similarly, both in terms of code layout and
implementation.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Changes since V9
================
Address comments from David Marchand:
- this patch-set is based on Nov 1, 2020 dpdk-next-eventdev
- fix docs build (doxy-api.conf.in and doxy-api-index.md)
- restore blank line in MAINTAINERS file
- move dlb announcement in release_20_11.rst after ethdev
- use headers = files() for exported meson public headers
- fix a typo in 'add documentation ..." commit message
- use eal version of cldemote
- convert a couple of printfs to LOG messages
- all patches build incrementally (gcc), and checkpatches reports
  success
- I am not able to run clang locally. If clang errors are still
  present I will ask IT to install clang on a build server tomorrow.

Changes since V8:
=================
- Fixed format errors in doc/api/doxy-api-index.md
- Delayed introduction of dlb2_consume_qe_immediate until
  add-dequeue-and-its-burst-variants.patch
- Delayed introduction of dlb2_construct_token_pop_qe until 
  add-PMD-s-token-pop-public-interface.patch

Changes since V7:
================
- fix CENTOS build error: use __m128i instead of __v2di with
  _mm_stream_si128
- deleted unused dlb2_umwait and dlb2_umonitor functions

Changes since V6:
=================
- removed unused function, fixing build error
- fixed typo in port_setup commit message
- this patch series is based on dpdk-next-eventdev

Changes since V5:
================
- convert to use rte_power_monitor patches
- replace __builtin_ia32_movntdq() with _mm_stream_si128()
- remove unused functions in dlb_selftest.c
- workaround for RHEL 7 gcc brace bug

Changes since V4:
================
- moved introduction of dlb in relnotes_20_11 to first patch in series
- fixed underlines in dlb.rst that were too short
- note that the code still uses its private byte-encoded versions of
  umonitor/umwait, rather than the new functions in the power
  patch that are built on top of those intrinsics. This is intentional.

Changes since V3:
================
- updated MAINTAINERS file to alphabetically insert DLB
- don't create RTE_ symbols in pmd
- converted to use version.map scheme
- converted to use .._master_lcore instead of .._main_lcore
- this patch set is based on dpdk-next-eventdev

Changes since V2:
================
- fixed meson conditional build. Moved test into driver’s meson.build
  file instead of event/meson.build
- documentation is populated as associated code is introduced
- add log_register in add dynamic logging patch
- rename RTE_xxx symbol(s) as DLB2_xxx
- replaced function ptr enqueue_four with direct call to movdir64b
- remove unused port_pages
- broke up probe patch into 3 smaller patches for easier review
- changed param order of movdir64b/movntdq to match intrinsics
- added self to MAINTAINERS files
- squashed announcement of availability into last patch in series
- correct spelling errors and delete repeated words
- DPDK_21.0 -> DPDK 21 in map file
- add experimental banner to public structs and APIs

Changes since V1:
=================
- implement changes requested in code reviews by Gage Eads and Mike
  Chen
- fix a memzone leak
- convert to use eal rte-cpuflags patch from Liang Ma

Known Issues:
- the documentation contained in dlb2.rst is not complete, and
  should be updated to include command line parameters (class of service,
  etc ...).

Depends-on: patch-82202 ("eventdev: increase MAX QUEUES PER DEV to 255")

Timothy McDaniel (23):
  event/dlb2: add documentation and meson build infrastructure
  event/dlb2: add dynamic logging
  event/dlb2: add private data structures and constants
  event/dlb2: add definitions shared with LKM or shared code
  event/dlb2: add inline functions
  event/dlb2: add eventdev probe
  event/dlb2: add flexible interface
  event/dlb2: add probe-time hardware init
  event/dlb2: add xstats
  event/dlb2: add infos get and configure
  event/dlb2: add queue and port default conf
  event/dlb2: add queue setup
  event/dlb2: add port setup
  event/dlb2: add port link
  event/dlb2: add port unlink and port unlinks in progress
  event/dlb2: add eventdev start
  event/dlb2: add enqueue and its burst variants
  event/dlb2: add dequeue and its burst variants
  event/dlb2: add eventdev stop and close
  event/dlb2: add PMD's token pop public interface
  event/dlb2: add PMD self-tests
  event/dlb2: add queue and port release
  event/dlb2: add timeout ticks entry point

 MAINTAINERS                                    |    5 +
 app/test/test_eventdev.c                       |    7 +
 config/rte_config.h                            |    7 +
 doc/api/doxy-api-index.md                      |    3 +-
 doc/api/doxy-api.conf.in                       |    1 +
 doc/guides/eventdevs/dlb2.rst                  |  365 ++
 doc/guides/eventdevs/index.rst                 |    1 +
 doc/guides/rel_notes/release_20_11.rst         |    5 +
 drivers/event/dlb2/dlb2.c                      | 3947 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c                |   74 +
 drivers/event/dlb2/dlb2_iface.h                |   74 +
 drivers/event/dlb2/dlb2_inline_fns.h           |   33 +
 drivers/event/dlb2/dlb2_log.h                  |   25 +
 drivers/event/dlb2/dlb2_priv.h                 |  578 +++
 drivers/event/dlb2/dlb2_selftest.c             | 1524 ++++++
 drivers/event/dlb2/dlb2_user.h                 |  679 +++
 drivers/event/dlb2/dlb2_xstats.c               | 1235 +++++
 drivers/event/dlb2/meson.build                 |   23 +
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 +++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 +
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c     | 6027 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  673 +++
 drivers/event/dlb2/pf/dlb2_main.h              |   97 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  728 +++
 drivers/event/dlb2/rte_pmd_dlb2.c              |   39 +
 drivers/event/dlb2/rte_pmd_dlb2.h              |   72 +
 drivers/event/dlb2/version.map                 |    9 +
 drivers/event/meson.build                      |    3 +-
 34 files changed, 22467 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
 create mode 100644 drivers/event/dlb2/dlb2_log.h
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c
 create mode 100644 drivers/event/dlb2/dlb2_user.h
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
 create mode 100644 drivers/event/dlb2/version.map

-- 
2.6.4

[dpdk-dev] [PATCH v10 01/23] event/dlb2: add documentation and meson build infrastructure

[Timothy McDaniel]

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux 64 bit X86 platforms at this time.

Adds announcement of availability for the new driver
for Intel Dynamic Load Balancer 2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 MAINTAINERS                            |  5 +++++
 config/rte_config.h                    |  7 ++++++
 doc/guides/eventdevs/dlb2.rst          | 40 ++++++++++++++++++++++++++++++++++
 doc/guides/eventdevs/index.rst         |  1 +
 doc/guides/rel_notes/release_20_11.rst |  5 +++++
 drivers/event/dlb2/meson.build         | 15 +++++++++++++
 drivers/event/dlb2/version.map         |  3 +++
 drivers/event/meson.build              |  3 ++-
 8 files changed, 78 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 5b390d1..c67fa74 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1176,6 +1176,11 @@ Cavium OCTEON TX timvf
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 F: drivers/event/octeontx/timvf_*
 
+Intel DLB2
+M: Timothy McDaniel <timothy.mcdaniel@intel.com>
+F: drivers/event/dlb2/
+F: doc/guides/eventdevs/dlb2.rst
+
 Marvell OCTEON TX2
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 M: Jerin Jacob <jerinj@marvell.com>
diff --git a/config/rte_config.h b/config/rte_config.h
index b78c6aa..64aa333 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -135,4 +135,11 @@
 /* QEDE PMD defines */
 #define RTE_LIBRTE_QEDE_FW ""
 
+/* DLB2 defines */
+#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
+#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
+#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
+#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
+
 #endif /* _RTE_CONFIG_H_ */
diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
new file mode 100644
index 0000000..e3091c9
--- /dev/null
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -0,0 +1,40 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Driver for the Intel® Dynamic Load Balancer (DLB2)
+==================================================
+
+The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
+
+Prerequisites
+-------------
+
+Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
+the basic DPDK environment.
+
+Configuration
+-------------
+
+The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
+device access. To use this operation mode, the PCIe PF device must be bound
+to a DPDK-compatible VFIO driver, such as vfio-pci.
+
+Eventdev API Notes
+------------------
+
+The DLB2 provides the functions of a DPDK event device; specifically, it
+supports atomic, ordered, and parallel scheduling events from queues to ports.
+However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
+features are abstracted by the PMD such as directed ports.
+
+In general the dlb PMD is designed for ease-of-use and does not require a
+detailed understanding of the hardware, but these details are important when
+writing high-performance code. This section describes the places where the
+eventdev API and DLB2 misalign.
+
+Flow ID
+~~~~~~~
+
+The flow ID field is preserved in the event when it is scheduled in the
+DLB2.
+
diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
index bb66a5e..738788d 100644
--- a/doc/guides/eventdevs/index.rst
+++ b/doc/guides/eventdevs/index.rst
@@ -11,6 +11,7 @@ application through the eventdev API.
     :maxdepth: 2
     :numbered:
 
+    dlb2
     dpaa
     dpaa2
     dsw
diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
index 88b9086..e1b9cc1 100644
--- a/doc/guides/rel_notes/release_20_11.rst
+++ b/doc/guides/rel_notes/release_20_11.rst
@@ -295,6 +295,11 @@ New Features
   Added performance tuning arguments to allow tuning the scheduler for
   better throughtput in high core count use cases.
 
+* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
+
+  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
+  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
+
 * **Updated ioat rawdev driver**
 
   The ioat rawdev driver has been updated and enhanced. Changes include:
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
new file mode 100644
index 0000000..68cb0ae
--- /dev/null
+++ b/drivers/event/dlb2/meson.build
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2020 Intel Corporation
+
+if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
+        build = false
+        reason = 'only supported on ARCH_X86_64 Linux'
+        subdir_done()
+endif
+
+sources = files(
+)
+
+headers = files()
+
+deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
new file mode 100644
index 0000000..4a76d1d
--- /dev/null
+++ b/drivers/event/dlb2/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index a7dac99..c433c10 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -5,7 +5,8 @@ if is_windows
 	subdir_done()
 endif
 
-drivers = ['dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw', 'dsw']
+drivers = ['dlb2', 'dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw',
+	   'dsw']
 if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
-- 
2.6.4

[dpdk-dev] [PATCH v10 02/23] event/dlb2: add dynamic logging

[Timothy McDaniel]

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c      |  7 +++++++
 drivers/event/dlb2/dlb2_log.h  | 25 +++++++++++++++++++++++++
 drivers/event/dlb2/meson.build |  3 +--
 3 files changed, 33 insertions(+), 2 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_log.h

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
new file mode 100644
index 0000000..2082dd6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2.c
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <rte_log.h>
+
+RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
new file mode 100644
index 0000000..dc1481e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_log.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_EVDEV_LOG_H_
+#define _DLB2_EVDEV_LOG_H_
+
+extern int eventdev_dlb2_log_level;
+
+/* Dynamic logging */
+#define DLB2_LOG_IMPL(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
+		__func__, ##args)
+
+#define DLB2_LOG_INFO(fmt, args...) \
+	DLB2_LOG_IMPL(INFO, fmt, ## args)
+
+#define DLB2_LOG_ERR(fmt, args...) \
+	DLB2_LOG_IMPL(ERR, fmt, ## args)
+
+/* remove debug logs at compile time unless actually debugging */
+#define DLB2_LOG_DBG(fmt, args...) \
+	RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
+
+#endif /* _DLB2_EVDEV_LOG_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 68cb0ae..6a569c9 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,8 +7,7 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files(
-)
+sources = files('dlb2.c')
 
 headers = files()
 
-- 
2.6.4

[dpdk-dev] [PATCH v10 03/23] event/dlb2: add private data structures and constants

[Timothy McDaniel]

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_priv.h | 575 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 575 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_priv.h

diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
new file mode 100644
index 0000000..61567a6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -0,0 +1,575 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_PRIV_H_
+#define _DLB2_PRIV_H_
+
+#include <emmintrin.h>
+#include <stdbool.h>
+
+#include <rte_eventdev.h>
+#include <rte_config.h>
+#include "dlb2_user.h"
+#include "dlb2_log.h"
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
+#else
+#define DLB2_INC_STAT(_stat, _incr_val)
+#endif
+
+#define EVDEV_DLB2_NAME_PMD dlb2_event
+
+/*  command line arg strings */
+#define NUMA_NODE_ARG "numa_node"
+#define DLB2_MAX_NUM_EVENTS "max_num_events"
+#define DLB2_NUM_DIR_CREDITS "num_dir_credits"
+#define DEV_ID_ARG "dev_id"
+#define DLB2_DEFER_SCHED_ARG "defer_sched"
+#define DLB2_QID_DEPTH_THRESH_ARG "qid_depth_thresh"
+#define DLB2_COS_ARG "cos"
+
+/* Begin HW related defines and structs */
+
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_VFS 16
+#define DLB2_MAX_NUM_LDB_QUEUES 32
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS 64
+#define DLB2_MAX_NUM_DIR_QUEUES 64
+#define DLB2_MAX_NUM_FLOWS (64 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDITS (8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS (2 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_DIR_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQOS_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_MAX_DEVICE_PATH 32
+#define DLB2_MIN_DEQUEUE_TIMEOUT_NS 1
+/* Note: "- 1" here to support the timeout range check in eventdev_autotest */
+#define DLB2_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1)
+#define DLB2_SW_CREDIT_BATCH_SZ 32
+#define DLB2_NUM_SN_GROUPS 2
+#define DLB2_MAX_LDB_SN_ALLOC 1024
+#define DLB2_MAX_QUEUE_DEPTH_THRESHOLD 8191
+
+/* 2048 total hist list entries and 64 total ldb ports, which
+ * makes for 2048/64 == 32 hist list entries per port. However, CQ
+ * depth must be a power of 2 and must also be >= HIST LIST entries.
+ * As a result we just limit the maximum dequeue depth to 32.
+ */
+#define DLB2_MIN_CQ_DEPTH 1
+#define DLB2_MAX_CQ_DEPTH 32
+#define DLB2_MIN_HARDWARE_CQ_DEPTH 8
+#define DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \
+	DLB2_MAX_CQ_DEPTH
+
+/*
+ * Static per queue/port provisioning values
+ */
+#define DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 64
+
+#define CQ_BASE(is_dir) ((is_dir) ? DLB2_DIR_CQ_BASE : DLB2_LDB_CQ_BASE)
+#define CQ_SIZE(is_dir) ((is_dir) ? DLB2_DIR_CQ_MAX_SIZE : \
+				    DLB2_LDB_CQ_MAX_SIZE)
+#define PP_BASE(is_dir) ((is_dir) ? DLB2_DIR_PP_BASE : DLB2_LDB_PP_BASE)
+
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+
+#define DLB2_NUM_QES_PER_CACHE_LINE 4
+
+#define DLB2_MAX_ENQUEUE_DEPTH 64
+#define DLB2_MIN_ENQUEUE_DEPTH 4
+
+#define DLB2_NAME_SIZE 64
+
+#define DLB2_1K 1024
+#define DLB2_2K (2 * DLB2_1K)
+#define DLB2_4K (4 * DLB2_1K)
+#define DLB2_16K (16 * DLB2_1K)
+#define DLB2_32K (32 * DLB2_1K)
+#define DLB2_1MB (DLB2_1K * DLB2_1K)
+#define DLB2_16MB (16 * DLB2_1MB)
+
+/* Use the upper 3 bits of the event priority to select the DLB2 priority */
+#define EV_TO_DLB2_PRIO(x) ((x) >> 5)
+#define DLB2_TO_EV_PRIO(x) ((x) << 5)
+
+enum dlb2_hw_port_types {
+	DLB2_LDB_PORT,
+	DLB2_DIR_PORT,
+	DLB2_NUM_PORT_TYPES /* Must be last */
+};
+
+enum dlb2_hw_queue_types {
+	DLB2_LDB_QUEUE,
+	DLB2_DIR_QUEUE,
+	DLB2_NUM_QUEUE_TYPES /* Must be last */
+};
+
+#define PORT_TYPE(p) ((p)->is_directed ? DLB2_DIR_PORT : DLB2_LDB_PORT)
+
+/* Do not change - must match hardware! */
+enum dlb2_hw_sched_type {
+	DLB2_SCHED_ATOMIC = 0,
+	DLB2_SCHED_UNORDERED,
+	DLB2_SCHED_ORDERED,
+	DLB2_SCHED_DIRECTED,
+	/* DLB2_NUM_HW_SCHED_TYPES must be last */
+	DLB2_NUM_HW_SCHED_TYPES
+};
+
+struct dlb2_hw_rsrcs {
+	int32_t nb_events_limit;
+	uint32_t num_queues;		/* Total queues (lb + dir) */
+	uint32_t num_ldb_queues;	/* Number of available ldb queues */
+	uint32_t num_ldb_ports;         /* Number of load balanced ports */
+	uint32_t num_dir_ports;         /* Number of directed ports */
+	uint32_t num_ldb_credits;       /* Number of load balanced credits */
+	uint32_t num_dir_credits;       /* Number of directed credits */
+	uint32_t reorder_window_size;   /* Size of reorder window */
+};
+
+struct dlb2_hw_resource_info {
+	/**> Max resources that can be provided */
+	struct dlb2_hw_rsrcs hw_rsrc_max;
+	int num_sched_domains;
+	uint32_t socket_id;
+};
+
+enum dlb2_enqueue_type {
+	/**>
+	 * New : Used to inject a new packet into the QM.
+	 */
+	DLB2_ENQ_NEW,
+	/**>
+	 * Forward : Enqueues a packet, and
+	 *  - if atomic: release any lock it holds in the QM
+	 *  - if ordered: release the packet for egress re-ordering
+	 */
+	DLB2_ENQ_FWD,
+	/**>
+	 * Enqueue Drop : Release an inflight packet. Must be called with
+	 * event == NULL. Used to drop a packet.
+	 *
+	 * Note that all packets dequeued from a load-balanced port must be
+	 * released, either with DLB2_ENQ_DROP or DLB2_ENQ_FWD.
+	 */
+	DLB2_ENQ_DROP,
+
+	/* marker for array sizing etc. */
+	_DLB2_NB_ENQ_TYPES
+};
+
+/* hw-specific format - do not change */
+
+struct dlb2_event_type {
+	uint8_t major:4;
+	uint8_t unused:4;
+	uint8_t sub;
+};
+
+union dlb2_opaque_data {
+	uint16_t opaque_data;
+	struct dlb2_event_type event_type;
+};
+
+struct dlb2_msg_info {
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+};
+
+#define DLB2_NEW_CMD_BYTE 0x08
+#define DLB2_FWD_CMD_BYTE 0x0A
+#define DLB2_COMP_CMD_BYTE 0x02
+#define DLB2_POP_CMD_BYTE 0x01
+#define DLB2_NOOP_CMD_BYTE 0x00
+
+/* hw-specific format - do not change */
+struct dlb2_enqueue_qe {
+	uint64_t data;
+	/* Word 3 */
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	/* Word 4 */
+	uint16_t lock_id;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_cq_pop_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t tokens:10;
+	uint16_t rsvd2:6;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_dequeue_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t flow_id:16; /* was pp_id in v1 */
+	uint8_t debug;
+	uint8_t cq_gen:1;
+	uint8_t qid_depth:2; /* 2 bits in v2 */
+	uint8_t rsvd1:2;
+	uint8_t error:1;
+	uint8_t rsvd2:2;
+};
+
+union dlb2_port_config {
+	struct dlb2_create_ldb_port_args ldb;
+	struct dlb2_create_dir_port_args dir;
+};
+
+enum dlb2_port_state {
+	PORT_CLOSED,
+	PORT_STARTED,
+	PORT_STOPPED
+};
+
+enum dlb2_configuration_state {
+	/* The resource has not been configured */
+	DLB2_NOT_CONFIGURED,
+	/* The resource was configured, but the device was stopped */
+	DLB2_PREV_CONFIGURED,
+	/* The resource is currently configured */
+	DLB2_CONFIGURED
+};
+
+struct dlb2_port {
+	uint32_t id;
+	bool is_directed;
+	bool gen_bit;
+	uint16_t dir_credits;
+	uint32_t dequeue_depth;
+	union dlb2_port_config cfg;
+	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
+	uint16_t cached_ldb_credits;
+	uint16_t ldb_credits;
+	uint16_t cached_dir_credits;
+	bool int_armed;
+	uint16_t owed_tokens;
+	int16_t issued_releases;
+	int cq_depth;
+	uint16_t cq_idx;
+	uint16_t cq_idx_unmasked;
+	uint16_t cq_depth_mask;
+	uint16_t gen_bit_shift;
+	enum dlb2_port_state state;
+	enum dlb2_configuration_state config_state;
+	int num_mapped_qids;
+	uint8_t *qid_mappings;
+	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
+	struct dlb2_enqueue_qe *int_arm_qe;
+	struct dlb2_cq_pop_qe *consume_qe;
+	struct dlb2_eventdev *dlb2; /* back ptr */
+	struct dlb2_eventdev_port *ev_port; /* back ptr */
+};
+
+/* Per-process per-port mmio and memory pointers */
+struct process_local_port_data {
+	uint64_t *pp_addr;
+	struct dlb2_dequeue_qe *cq_base;
+	const struct rte_memzone *mz;
+	bool mmaped;
+};
+
+struct dlb2_eventdev;
+
+struct dlb2_config {
+	int configured;
+	int reserved;
+	uint32_t num_ldb_credits;
+	uint32_t num_dir_credits;
+	struct dlb2_create_sched_domain_args resources;
+};
+
+enum dlb2_cos {
+	DLB2_COS_DEFAULT = -1,
+	DLB2_COS_0 = 0,
+	DLB2_COS_1,
+	DLB2_COS_2,
+	DLB2_COS_3
+};
+
+struct dlb2_hw_dev {
+	struct dlb2_config cfg;
+	struct dlb2_hw_resource_info info;
+	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
+	uint32_t domain_id;
+	enum dlb2_cos cos_id;
+	rte_spinlock_t resource_lock; /* for MP support */
+} __rte_cache_aligned;
+
+/* End HW related defines and structs */
+
+/* Begin DLB2 PMD Eventdev related defines and structs */
+
+#define DLB2_MAX_NUM_QUEUES \
+	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
+
+#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS + DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
+
+/** Structure to hold the queue to port link establishment attributes */
+
+struct dlb2_event_queue_link {
+	uint8_t queue_id;
+	uint8_t priority;
+	bool mapped;
+	bool valid;
+};
+
+struct dlb2_traffic_stats {
+	uint64_t rx_ok;
+	uint64_t rx_drop;
+	uint64_t rx_interrupt_wait;
+	uint64_t rx_umonitor_umwait;
+	uint64_t tx_ok;
+	uint64_t total_polls;
+	uint64_t zero_polls;
+	uint64_t tx_nospc_ldb_hw_credits;
+	uint64_t tx_nospc_dir_hw_credits;
+	uint64_t tx_nospc_inflight_max;
+	uint64_t tx_nospc_new_event_limit;
+	uint64_t tx_nospc_inflight_credits;
+};
+
+/* DLB2 HW sets the 2bit qid_depth in rx QEs based on the programmable depth
+ * threshold. The global default value in config/common_base (or rte_config.h)
+ * can be overridden on a per-qid basis using a vdev command line parameter.
+ * 3: depth > threshold
+ * 2: threshold >= depth > 3/4 threshold
+ * 1: 3/4 threshold >= depth > 1/2 threshold
+ * 0: depth <= 1/2 threshold.
+ */
+#define DLB2_QID_DEPTH_LE50 0
+#define DLB2_QID_DEPTH_GT50_LE75 1
+#define DLB2_QID_DEPTH_GT75_LE100 2
+#define DLB2_QID_DEPTH_GT100 3
+#define DLB2_NUM_QID_DEPTH_STAT_VALS 4 /* 2 bits */
+
+struct dlb2_queue_stats {
+	uint64_t enq_ok;
+	uint64_t qid_depth[DLB2_NUM_QID_DEPTH_STAT_VALS];
+};
+
+struct dlb2_port_stats {
+	struct dlb2_traffic_stats traffic;
+	uint64_t tx_op_cnt[4]; /* indexed by rte_event.op */
+	uint64_t tx_implicit_rel;
+	uint64_t tx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t tx_invalid;
+	uint64_t rx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t rx_sched_invalid;
+	struct dlb2_queue_stats queue[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_eventdev_port {
+	struct dlb2_port qm_port; /* hw specific data structure */
+	struct rte_event_port_conf conf; /* user-supplied configuration */
+	uint16_t inflight_credits; /* num credits this port has right now */
+	uint16_t credit_update_quanta;
+	struct dlb2_eventdev *dlb2; /* backlink optimization */
+	struct dlb2_port_stats stats __rte_cache_aligned;
+	struct dlb2_event_queue_link link[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	int num_links;
+	uint32_t id; /* port id */
+	/* num releases yet to be completed on this port.
+	 * Only applies to load-balanced ports.
+	 */
+	uint16_t outstanding_releases;
+	uint16_t inflight_max; /* app requested max inflights for this port */
+	/* setup_done is set when the event port is setup */
+	bool setup_done;
+	/* enq_configured is set when the qm port is created */
+	bool enq_configured;
+	uint8_t implicit_release; /* release events before dequeueing */
+}  __rte_cache_aligned;
+
+struct dlb2_queue {
+	uint32_t num_qid_inflights; /* User config */
+	uint32_t num_atm_inflights; /* User config */
+	enum dlb2_configuration_state config_state;
+	int sched_type; /* LB queue only */
+	uint32_t id;
+	bool is_directed;
+};
+
+struct dlb2_eventdev_queue {
+	struct dlb2_queue qm_queue;
+	struct rte_event_queue_conf conf; /* User config */
+	int depth_threshold; /* use default if 0 */
+	uint32_t id;
+	bool setup_done;
+	uint8_t num_links;
+};
+
+enum dlb2_run_state {
+	DLB2_RUN_STATE_STOPPED = 0,
+	DLB2_RUN_STATE_STOPPING,
+	DLB2_RUN_STATE_STARTING,
+	DLB2_RUN_STATE_STARTED
+};
+
+struct dlb2_eventdev {
+	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
+	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each queue */
+	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
+	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each port */
+	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
+	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
+	struct dlb2_get_num_resources_args hw_rsrc_query_results;
+	uint32_t xstats_count_mode_queue;
+	struct dlb2_hw_dev qm_instance; /* strictly hw related */
+	uint64_t global_dequeue_wait_ticks;
+	struct dlb2_xstats_entry *xstats;
+	struct rte_eventdev *event_dev; /* backlink to dev */
+	uint32_t xstats_count_mode_dev;
+	uint32_t xstats_count_mode_port;
+	uint32_t xstats_count;
+	uint32_t inflights; /* use __atomic builtins */
+	uint32_t new_event_limit;
+	int max_num_events_override;
+	int num_dir_credits_override;
+	volatile enum dlb2_run_state run_state;
+	uint16_t num_dir_queues; /* total num of evdev dir queues requested */
+	uint16_t num_dir_credits;
+	uint16_t num_ldb_credits;
+	uint16_t num_queues; /* total queues */
+	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested */
+	uint16_t num_ports; /* total num of evdev ports requested */
+	uint16_t num_ldb_ports; /* total num of ldb ports requested */
+	uint16_t num_dir_ports; /* total num of dir ports requested */
+	bool umwait_allowed;
+	bool global_dequeue_wait; /* Not using per dequeue wait if true */
+	bool defer_sched;
+	enum dlb2_cq_poll_modes poll_mode;
+	uint8_t revision;
+	bool configured;
+	uint16_t max_ldb_credits;
+	uint16_t max_dir_credits;
+
+	/* force hw credit pool counters into exclusive cache lines */
+
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t ldb_credit_pool __rte_cache_aligned;
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t dir_credit_pool __rte_cache_aligned;
+};
+
+/* used for collecting and passing around the dev args */
+struct dlb2_qid_depth_thresholds {
+	int val[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_devargs {
+	int socket_id;
+	int max_num_events;
+	int num_dir_credits_override;
+	int dev_id;
+	int defer_sched;
+	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
+	enum dlb2_cos cos_id;
+};
+
+/* End Eventdev related defines and structs */
+
+/* Forwards for non-inlined functions */
+
+void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
+
+int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
+
+void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
+
+int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n);
+
+int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstat_names,
+		unsigned int *ids, unsigned int size);
+
+uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+					  const char *name, unsigned int *id);
+
+int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode,
+		int16_t queue_port_id,
+		const uint32_t ids[],
+		uint32_t nb_ids);
+
+int test_dlb2_eventdev(void);
+
+int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+				const char *name,
+				struct dlb2_devargs *dlb2_args);
+
+int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+				  const char *name);
+
+uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *queue);
+
+int dlb2_parse_params(const char *params,
+		      const char *name,
+		      struct dlb2_devargs *dlb2_args);
+
+/* Extern globals */
+extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
+
+#endif	/* _DLB2_PRIV_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v10 04/23] event/dlb2: add definitions shared with LKM or shared code

[Timothy McDaniel]

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_user.h | 679 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 679 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_user.h

diff --git a/drivers/event/dlb2/dlb2_user.h b/drivers/event/dlb2/dlb2_user.h
new file mode 100644
index 0000000..f4bda78
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_user.h
@@ -0,0 +1,679 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#define DLB2_MAX_NAME_LEN 64
+
+#include <linux/types.h>
+
+enum dlb2_error {
+	DLB2_ST_SUCCESS = 0,
+	DLB2_ST_NAME_EXISTS,
+	DLB2_ST_DOMAIN_UNAVAILABLE,
+	DLB2_ST_LDB_PORTS_UNAVAILABLE,
+	DLB2_ST_DIR_PORTS_UNAVAILABLE,
+	DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB2_ST_INVALID_DOMAIN_ID,
+	DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB2_ST_INVALID_LDB_QUEUE_ID,
+	DLB2_ST_INVALID_CQ_DEPTH,
+	DLB2_ST_INVALID_CQ_VIRT_ADDR,
+	DLB2_ST_INVALID_PORT_ID,
+	DLB2_ST_INVALID_QID,
+	DLB2_ST_INVALID_PRIORITY,
+	DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+	DLB2_ST_INVALID_DIR_QUEUE_ID,
+	DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB2_ST_DOMAIN_NOT_CONFIGURED,
+	DLB2_ST_INTERNAL_ERROR,
+	DLB2_ST_DOMAIN_IN_USE,
+	DLB2_ST_DOMAIN_NOT_FOUND,
+	DLB2_ST_QUEUE_NOT_FOUND,
+	DLB2_ST_DOMAIN_STARTED,
+	DLB2_ST_DOMAIN_NOT_STARTED,
+	DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB2_ST_DOMAIN_RESET_FAILED,
+	DLB2_ST_MBOX_ERROR,
+	DLB2_ST_INVALID_HIST_LIST_DEPTH,
+	DLB2_ST_NO_MEMORY,
+	DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB2_ST_INVALID_COS_ID,
+};
+
+static const char dlb2_error_strings[][128] = {
+	"DLB2_ST_SUCCESS",
+	"DLB2_ST_NAME_EXISTS",
+	"DLB2_ST_DOMAIN_UNAVAILABLE",
+	"DLB2_ST_LDB_PORTS_UNAVAILABLE",
+	"DLB2_ST_DIR_PORTS_UNAVAILABLE",
+	"DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+	"DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+	"DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+	"DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+	"DLB2_ST_INVALID_DOMAIN_ID",
+	"DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+	"DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+	"DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+	"DLB2_ST_INVALID_LDB_QUEUE_ID",
+	"DLB2_ST_INVALID_CQ_DEPTH",
+	"DLB2_ST_INVALID_CQ_VIRT_ADDR",
+	"DLB2_ST_INVALID_PORT_ID",
+	"DLB2_ST_INVALID_QID",
+	"DLB2_ST_INVALID_PRIORITY",
+	"DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+	"DLB2_ST_INVALID_DIR_QUEUE_ID",
+	"DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+	"DLB2_ST_DOMAIN_NOT_CONFIGURED",
+	"DLB2_ST_INTERNAL_ERROR",
+	"DLB2_ST_DOMAIN_IN_USE",
+	"DLB2_ST_DOMAIN_NOT_FOUND",
+	"DLB2_ST_QUEUE_NOT_FOUND",
+	"DLB2_ST_DOMAIN_STARTED",
+	"DLB2_ST_DOMAIN_NOT_STARTED",
+	"DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+	"DLB2_ST_DOMAIN_RESET_FAILED",
+	"DLB2_ST_MBOX_ERROR",
+	"DLB2_ST_INVALID_HIST_LIST_DEPTH",
+	"DLB2_ST_NO_MEMORY",
+	"DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+	"DLB2_ST_INVALID_COS_ID",
+};
+
+struct dlb2_cmd_response {
+	__u32 status; /* Interpret using enum dlb2_error */
+	__u32 id;
+};
+
+/*******************/
+/* 'dlb2' commands */
+/*******************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+	DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	This ioctl interface is the same in all driver versions and is always
+ *	the first ioctl.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+	/* Output parameters */
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ *	load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ *		percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ *	load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ *   ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+enum dlb2_cq_poll_modes {
+	DLB2_CQ_POLL_MODE_STD,
+	DLB2_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB2_CQ_POLL_MODE must be last */
+	NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/********************************/
+/* 'scheduling domain' commands */
+/********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_start_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
+#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
+
+
+#endif /* __DLB2_USER_H */
-- 
2.6.4

[dpdk-dev] [PATCH v10 05/23] event/dlb2: add inline functions

[Timothy McDaniel]

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_inline_fns.h | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h

diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
new file mode 100644
index 0000000..ac8d01a
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_inline_fns.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_INLINE_FNS_H_
+#define _DLB2_INLINE_FNS_H_
+
+/* Inline functions required in more than one source file. */
+
+static inline struct dlb2_eventdev *
+dlb2_pmd_priv(const struct rte_eventdev *eventdev)
+{
+	return eventdev->data->dev_private;
+}
+
+static inline void
+dlb2_movntdq_single(void *pp_addr, void *qe4)
+{
+	long long *_qe  = (long long *)qe4;
+	__m128i src_data0 = (__m128i){_qe[0], _qe[1]};
+
+	_mm_stream_si128(pp_addr, src_data0);
+}
+
+static inline void
+dlb2_movdir64b(void *dest, void *src)
+{
+	asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
+		:
+	: "a" (dest), "d" (src));
+}
+
+#endif /* _DLB2_INLINE_FNS_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v10 06/23] event/dlb2: add eventdev probe

[Timothy McDaniel]

Add the eventdev portion of probe, and parse command line
options, but do not initialize hardware.

Changes since v2 patch-set probe:

Primary and secondary probe-time init has been removed, and
will be introduced in subsequent patches contained in
the v3 patch-set.

Hardware init has been moved to a subsequent patch in order to
minimize the patch size. The files dlb2/pf/dlb2_pf.c and
dlb2/pf/dlb_main.c include stubs of hardware init functions,
and these will be replaced with the real versions when the
hardware layer is committed.

Initialization of the flexible interface layer has been moved to
a subsequent patch in order to minimize patch size.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                      |  343 ++++
 drivers/event/dlb2/meson.build                 |    5 +-
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +++
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  621 ++++++
 drivers/event/dlb2/pf/dlb2_main.h              |   98 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  196 ++
 13 files changed, 7497 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2082dd6..1b11979 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2,6 +2,349 @@
  * Copyright(c) 2016-2020 Intel Corporation
  */
 
+#include <assert.h>
+#include <errno.h>
+#include <nmmintrin.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_cycles.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_io.h>
+#include <rte_kvargs.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_ring.h>
+#include <rte_string_fns.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+/*
+ * Resources exposed to eventdev. Some values overridden at runtime using
+ * values returned by the DLB kernel driver.
+ */
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
+#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
+#endif
+
+struct process_local_port_data
+dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
+
+#define DLB2_BASE_10 10
+
+static int
+dlb2_string_to_int(int *result, const char *str)
+{
+	long ret;
+	char *endptr;
+
+	if (str == NULL || result == NULL)
+		return -EINVAL;
+
+	errno = 0;
+	ret = strtol(str, &endptr, DLB2_BASE_10);
+	if (errno)
+		return -errno;
+
+	/* long int and int may be different width for some architectures */
+	if (ret < INT_MIN || ret > INT_MAX || endptr == str)
+		return -EINVAL;
+
+	*result = ret;
+	return 0;
+}
+
+static int
+set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
+{
+	int *socket_id = opaque;
+	int ret;
+
+	ret = dlb2_string_to_int(socket_id, value);
+	if (ret < 0)
+		return ret;
+
+	if (*socket_id > RTE_MAX_NUMA_NODES)
+		return -EINVAL;
+	return 0;
+}
+
+static int
+set_max_num_events(const char *key __rte_unused,
+		   const char *value,
+		   void *opaque)
+{
+	int *max_num_events = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(max_num_events, value);
+	if (ret < 0)
+		return ret;
+
+	if (*max_num_events < 0 || *max_num_events >
+			DLB2_MAX_NUM_LDB_CREDITS) {
+		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
+			     DLB2_MAX_NUM_LDB_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_num_dir_credits(const char *key __rte_unused,
+		    const char *value,
+		    void *opaque)
+{
+	int *num_dir_credits = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(num_dir_credits, value);
+	if (ret < 0)
+		return ret;
+
+	if (*num_dir_credits < 0 ||
+	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
+		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
+			     DLB2_MAX_NUM_DIR_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_dev_id(const char *key __rte_unused,
+	   const char *value,
+	   void *opaque)
+{
+	int *dev_id = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(dev_id, value);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+set_cos(const char *key __rte_unused,
+	const char *value,
+	void *opaque)
+{
+	enum dlb2_cos *cos_id = opaque;
+	int x = 0;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(&x, value);
+	if (ret < 0)
+		return ret;
+
+	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
+		DLB2_LOG_ERR(
+			"COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
+			x);
+		return -EINVAL;
+	}
+
+	*cos_id = x;
+
+	return 0;
+}
+
+
+static int
+set_qid_depth_thresh(const char *key __rte_unused,
+		     const char *value,
+		     void *opaque)
+{
+	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
+	int first, last, thresh, i;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	/* command line override may take one of the following 3 forms:
+	 * qid_depth_thresh=all:<threshold_value> ... all queues
+	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
+	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
+	 */
+	if (sscanf(value, "all:%d", &thresh) == 1) {
+		first = 0;
+		last = DLB2_MAX_NUM_QUEUES - 1;
+	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
+		/* we have everything we need */
+	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
+		last = first;
+	} else {
+		DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val, qid-qid:val, or qid:val\n");
+		return -EINVAL;
+	}
+
+	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
+		return -EINVAL;
+	}
+
+	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
+			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
+		return -EINVAL;
+	}
+
+	for (i = first; i <= last; i++)
+		qid_thresh->val[i] = thresh; /* indexed by qid */
+
+	return 0;
+}
+
+int
+dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+			    const char *name,
+			    struct dlb2_devargs *dlb2_args)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+	RTE_SET_USED(dlb2_args);
+
+	return 0;
+}
+
+int
+dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+			      const char *name)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+int
+dlb2_parse_params(const char *params,
+		  const char *name,
+		  struct dlb2_devargs *dlb2_args)
+{
+	int ret = 0;
+	static const char * const args[] = { NUMA_NODE_ARG,
+					     DLB2_MAX_NUM_EVENTS,
+					     DLB2_NUM_DIR_CREDITS,
+					     DEV_ID_ARG,
+					     DLB2_QID_DEPTH_THRESH_ARG,
+					     DLB2_COS_ARG,
+					     NULL };
+
+	if (params != NULL && params[0] != '\0') {
+		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
+
+		if (kvlist == NULL) {
+			RTE_LOG(INFO, PMD,
+				"Ignoring unsupported parameters when creating device '%s'\n",
+				name);
+		} else {
+			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
+						     set_numa_node,
+						     &dlb2_args->socket_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing numa node parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
+						 set_max_num_events,
+						 &dlb2_args->max_num_events);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist,
+					DLB2_NUM_DIR_CREDITS,
+					set_num_dir_credits,
+					&dlb2_args->num_dir_credits_override);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
+						 set_dev_id,
+						 &dlb2_args->dev_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(
+					kvlist,
+					DLB2_QID_DEPTH_THRESH_ARG,
+					set_qid_depth_thresh,
+					&dlb2_args->qid_depth_thresholds);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
+						 set_cos,
+						 &dlb2_args->cos_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing cos parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
 
+			rte_kvargs_free(kvlist);
+		}
+	}
+	return ret;
+}
 RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 6a569c9..2855dd5 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,7 +7,10 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files('dlb2.c')
+sources = files('dlb2.c',
+		'pf/dlb2_main.c',
+		'pf/dlb2_pf.c'
+)
 
 headers = files()
 
diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
new file mode 100644
index 0000000..1d99f1e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_osdep_list.h"
+#include "dlb2_osdep_types.h"
+
+#define DLB2_MAX_NUM_VDEVS			16
+#define DLB2_MAX_NUM_DOMAINS			32
+#define DLB2_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS			64
+#define DLB2_MAX_NUM_DIR_PORTS			64
+#define DLB2_MAX_NUM_LDB_CREDITS		(8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS		(2 * 1024)
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB2_MAX_NUM_AQED_ENTRIES		2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB2_QID_PRIORITIES			8
+#define DLB2_NUM_ARB_WEIGHTS			8
+#define DLB2_MAX_WEIGHT				255
+#define DLB2_NUM_COS_DOMAINS			4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#ifdef FPGA
+#define DLB2_HZ					2000000
+#else
+#define DLB2_HZ					800000000
+#endif
+
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
+
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+	DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+	DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID		3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID		4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW		5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ	1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
+#define DLB2_VF_BASE_CQ_VECTOR_ID	     0
+#define DLB2_VF_LAST_CQ_VECTOR_ID	     30
+#define DLB2_VF_MBOX_VECTOR_ID		     31
+#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
+
+#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
+					     DLB2_MAX_NUM_DIR_PORTS + 1)
+
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE   0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
+				DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE   0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
+				DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE       0x2100000
+#define DLB2_LDB_PP_STRIDE     0x1000
+#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
+				DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE       0x2000000
+#define DLB2_DIR_PP_STRIDE     0x1000
+#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
+				DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
+struct dlb2_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb2_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb2_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
+struct dlb2_ldb_queue {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB2_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB2_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB2_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB2_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+	enum dlb2_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb2_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+	u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+	u32 bound[6] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & (1 << i))) {
+			group->slot_use_bitmap |= 1 << i;
+			return i;
+		}
+	}
+
+	return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+	int i, cnt = 0;
+
+	for (i = 0; i < 32; i++)
+		cnt += !!(group->slot_use_bitmap & (1 << i));
+
+	return cnt;
+}
+
+struct dlb2_hw_domain {
+	struct dlb2_function_resources *parent_func;
+	struct dlb2_list_entry func_list;
+	struct dlb2_list_head used_ldb_queues;
+	struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head used_dir_pq_pairs;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb2_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb2_bitmap;
+
+struct dlb2_function_resources {
+	struct dlb2_list_head avail_domains;
+	struct dlb2_list_head used_domains;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	struct dlb2_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+	struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+	struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+	struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
+struct dlb2_mbox {
+	u32 *mbox;
+	u32 *isr_in_progress;
+};
+
+struct dlb2_sw_mbox {
+	struct dlb2_mbox vdev_to_pf;
+	struct dlb2_mbox pf_to_vdev;
+	void (*pf_to_vdev_inject)(void *arg);
+	void *pf_to_vdev_inject_arg;
+};
+
+struct dlb2_hw {
+	/* BAR 0 address */
+	void  *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void  *func_kva;
+	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb2_hw_resources rsrcs;
+	struct dlb2_function_resources pf;
+	struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+	struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+	/* Virtualization */
+	int virt_mode;
+	struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
+	unsigned int pasid[DLB2_MAX_NUM_VDEVS];
+};
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
new file mode 100644
index 0000000..ce462c0
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
@@ -0,0 +1,596 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_BASE_DLB2_MBOX_H
+#define __DLB2_BASE_DLB2_MBOX_H
+
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+
+#define DLB2_MBOX_INTERFACE_VERSION 1
+
+/*
+ * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
+ * to send requests of its own (e.g. notifying a VF of an impending FLR).
+ * To avoid communication race conditions, e.g. the PF sends a response and then
+ * sends a request before the VF reads the response, the PF->VF mailbox is
+ * divided into two sections:
+ * - Bytes 0-47: PF responses
+ * - Bytes 48-63: PF requests
+ *
+ * Partitioning the PF->VF mailbox allows responses and requests to occupy the
+ * mailbox simultaneously.
+ */
+#define DLB2_PF2VF_RESP_BYTES	  48
+#define DLB2_PF2VF_RESP_BASE	  0
+#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
+
+#define DLB2_PF2VF_REQ_BYTES	  16
+#define DLB2_PF2VF_REQ_BASE	  (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
+#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
+
+/*
+ * Similarly, the VF->PF mailbox is divided into two sections:
+ * - Bytes 0-239: VF requests
+ * -- (Bytes 0-3 are unused due to a hardware errata)
+ * - Bytes 240-255: VF responses
+ */
+#define DLB2_VF2PF_REQ_BYTES	 236
+#define DLB2_VF2PF_REQ_BASE	 4
+#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
+
+#define DLB2_VF2PF_RESP_BYTES	  16
+#define DLB2_VF2PF_RESP_BASE	  (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
+#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
+
+/* VF-initiated commands */
+enum dlb2_mbox_cmd_type {
+	DLB2_MBOX_CMD_REGISTER,
+	DLB2_MBOX_CMD_UNREGISTER,
+	DLB2_MBOX_CMD_GET_NUM_RESOURCES,
+	DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
+	DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
+	DLB2_MBOX_CMD_CREATE_LDB_PORT,
+	DLB2_MBOX_CMD_CREATE_DIR_PORT,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT,
+	DLB2_MBOX_CMD_DISABLE_LDB_PORT,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT,
+	DLB2_MBOX_CMD_DISABLE_DIR_PORT,
+	DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_MAP_QID,
+	DLB2_MBOX_CMD_UNMAP_QID,
+	DLB2_MBOX_CMD_START_DOMAIN,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
+	DLB2_MBOX_CMD_ARM_CQ_INTR,
+	DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
+	DLB2_MBOX_CMD_GET_SN_ALLOCATION,
+	DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
+	DLB2_MBOX_CMD_GET_COS_BW,
+	DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
+	DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_QE_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_CMD_TYPES,
+};
+
+static const char dlb2_mbox_cmd_type_strings[][128] = {
+	"DLB2_MBOX_CMD_REGISTER",
+	"DLB2_MBOX_CMD_UNREGISTER",
+	"DLB2_MBOX_CMD_GET_NUM_RESOURCES",
+	"DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_LDB_PORT",
+	"DLB2_MBOX_CMD_CREATE_DIR_PORT",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_DISABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_DISABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_MAP_QID",
+	"DLB2_MBOX_CMD_UNMAP_QID",
+	"DLB2_MBOX_CMD_START_DOMAIN",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
+	"DLB2_MBOX_CMD_ARM_CQ_INTR",
+	"DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
+	"DLB2_MBOX_CMD_GET_SN_ALLOCATION",
+	"DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
+	"DLB2_MBOX_CMD_GET_COS_BW",
+	"DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
+	"DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
+};
+
+/* PF-initiated commands */
+enum dlb2_mbox_vf_cmd_type {
+	DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
+	DLB2_MBOX_VF_CMD_NOTIFICATION,
+	DLB2_MBOX_VF_CMD_IN_USE,
+
+	/* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_VF_CMD_TYPES,
+};
+
+static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
+	"DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
+	"DLB2_MBOX_VF_CMD_NOTIFICATION",
+	"DLB2_MBOX_VF_CMD_IN_USE",
+};
+
+#define DLB2_MBOX_CMD_TYPE(hdr) \
+	(((struct dlb2_mbox_req_hdr *)hdr)->type)
+#define DLB2_MBOX_CMD_STRING(hdr) \
+	dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
+
+enum dlb2_mbox_status_type {
+	DLB2_MBOX_ST_SUCCESS,
+	DLB2_MBOX_ST_INVALID_CMD_TYPE,
+	DLB2_MBOX_ST_VERSION_MISMATCH,
+	DLB2_MBOX_ST_INVALID_OWNER_VF,
+};
+
+static const char dlb2_mbox_status_type_strings[][128] = {
+	"DLB2_MBOX_ST_SUCCESS",
+	"DLB2_MBOX_ST_INVALID_CMD_TYPE",
+	"DLB2_MBOX_ST_VERSION_MISMATCH",
+	"DLB2_MBOX_ST_INVALID_OWNER_VF",
+};
+
+#define DLB2_MBOX_ST_TYPE(hdr) \
+	(((struct dlb2_mbox_resp_hdr *)hdr)->status)
+#define DLB2_MBOX_ST_STRING(hdr) \
+	dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
+
+/* This structure is always the first field in a request structure */
+struct dlb2_mbox_req_hdr {
+	u32 type;
+};
+
+/* This structure is always the first field in a response structure */
+struct dlb2_mbox_resp_hdr {
+	u32 status;
+};
+
+struct dlb2_mbox_register_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 min_interface_version;
+	u16 max_interface_version;
+};
+
+struct dlb2_mbox_register_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 interface_version;
+	u8 pf_id;
+	u8 vf_id;
+	u8 is_auxiliary_vf;
+	u8 primary_vf_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u16 num_sched_domains;
+	u16 num_ldb_queues;
+	u16 num_ldb_ports;
+	u16 num_cos_ldb_ports[4];
+	u16 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 max_contiguous_hist_list_entries;
+	u16 num_ldb_credits;
+	u16 num_dir_credits;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 num_ldb_queues;
+	u32 num_ldb_ports;
+	u32 num_cos_ldb_ports[4];
+	u32 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u8 cos_strict;
+	u8 padding0[3];
+	u32 padding1;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 num_sequence_numbers;
+	u32 num_qid_inflights;
+	u32 num_atomic_inflights;
+	u32 lock_id_comp_level;
+	u32 depth_threshold;
+	u32 padding;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 depth_threshold;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u16 cq_depth;
+	u16 cq_history_list_size;
+	u8 cos_id;
+	u8 cos_strict;
+	u16 padding1;
+	u64 cq_base_address;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u64 cq_base_address;
+	u16 cq_depth;
+	u16 padding0;
+	s32 queue_id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_map_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+	u32 priority;
+	u32 padding0;
+};
+
+struct dlb2_mbox_map_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_start_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+};
+
+struct dlb2_mbox_start_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 is_ldb;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding0;
+};
+
+/*
+ * The alert_id and aux_alert_data follows the format of the alerts defined in
+ * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
+ * the aux_alert_data value varies depending on the alert.
+ */
+struct dlb2_mbox_vf_alert_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 alert_id;
+	u32 aux_alert_data;
+};
+
+enum dlb2_mbox_vf_notification_type {
+	DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
+	DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
+
+	/* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
+	NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
+};
+
+struct dlb2_mbox_vf_notification_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 notification;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 in_use;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 num;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 cos_id;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 mode;
+};
+
+#endif /* __DLB2_BASE_DLB2_MBOX_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
new file mode 100644
index 0000000..c4c34eb
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
@@ -0,0 +1,230 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_H
+#define __DLB2_OSDEP_H
+
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <rte_string_fns.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include "../dlb2_main.h"
+#include "dlb2_resource.h"
+#include "../../dlb2_log.h"
+#include "../../dlb2_user.h"
+
+
+#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
+#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
+#define DLB2_CSR_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
+#define DLB2_FUNC_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
+/* Map to PMDs logging interface */
+#define DLB2_ERR(dev, fmt, args...) \
+	DLB2_LOG_ERR(fmt, ## args)
+
+#define DLB2_INFO(dev, fmt, args...) \
+	DLB2_LOG_INFO(fmt, ## args)
+
+#define DLB2_DEBUG(dev, fmt, args...) \
+	DLB2_LOG_DBG(fmt, ## args)
+
+/**
+ * os_udelay() - busy-wait for a number of microseconds
+ * @usecs: delay duration.
+ */
+static inline void os_udelay(int usecs)
+{
+	rte_delay_us(usecs);
+}
+
+/**
+ * os_msleep() - sleep for a number of milliseconds
+ * @usecs: delay duration.
+ */
+static inline void os_msleep(int msecs)
+{
+	rte_delay_ms(msecs);
+}
+
+#define DLB2_PP_BASE(__is_ldb) \
+	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
+
+/**
+ * os_map_producer_port() - map a producer port into the caller's address space
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ *
+ * This function maps the requested producer port memory into the caller's
+ * address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void *os_map_producer_port(struct dlb2_hw *hw,
+					 u8 port_id,
+					 bool is_ldb)
+{
+	uint64_t addr;
+	uint64_t pp_dma_base;
+
+	pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
+	addr = (pp_dma_base + (PAGE_SIZE * port_id));
+
+	return (void *)(uintptr_t)addr;
+}
+
+/**
+ * os_unmap_producer_port() - unmap a producer port
+ * @addr: mapped producer port address
+ *
+ * This function undoes os_map_producer_port() by unmapping the producer port
+ * memory from the caller's address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(addr);
+}
+
+/**
+ * os_fence_hcw() - fence an HCW to ensure it arrives at the device
+ * @hw: dlb2_hw handle for a particular device.
+ * @pp_addr: producer port address
+ */
+static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
+{
+	RTE_SET_USED(hw);
+
+	/* To ensure outstanding HCWs reach the device, read the PP address. IA
+	 * memory ordering prevents reads from passing older writes, and the
+	 * mfence also ensures this.
+	 */
+	rte_mb();
+
+	*(volatile u64 *)pp_addr;
+}
+
+/**
+ * DLB2_HW_ERR() - log an error message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_ERR(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_ERR(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/**
+ * DLB2_HW_DBG() - log an info message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_DBG(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_DEBUG(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/* The callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void *dlb2_complete_queue_map_unmap(void *__args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
+	int ret;
+
+	while (1) {
+		rte_spinlock_lock(&dlb2_dev->resource_mutex);
+
+		ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+		ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+		if (ret != 0) {
+			rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+			/* Relinquish the CPU so the application can process
+			 * its CQs, so this function doesn't deadlock.
+			 */
+			sched_yield();
+		} else {
+			break;
+		}
+	}
+
+	dlb2_dev->worker_launched = false;
+
+	rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+
+	return NULL;
+}
+
+
+/**
+ * os_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static inline void os_schedule_work(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+	pthread_t complete_queue_map_unmap_thread;
+	int ret;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
+				     "dlb_queue_unmap_waiter",
+				     NULL,
+				     dlb2_complete_queue_map_unmap,
+				     dlb2_dev);
+	if (ret)
+		DLB2_ERR(dlb2_dev,
+			 "Could not create queue complete map/unmap thread, err=%d\n",
+			 ret);
+	else
+		dlb2_dev->worker_launched = true;
+}
+
+/**
+ * os_worker_active() - query whether the map/unmap worker thread is active
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether a thread (launched by
+ * os_schedule_work()) is active. This function is used to determine
+ * whether or not to launch a worker thread.
+ */
+static inline bool os_worker_active(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	return dlb2_dev->worker_launched;
+}
+
+#endif /*  __DLB2_OSDEP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
new file mode 100644
index 0000000..423233b
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <rte_bitmap.h>
+#include <rte_string_fns.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include "../dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+	struct rte_bitmap *map;
+	unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+				    unsigned int len)
+{
+	struct dlb2_bitmap *bm;
+	void *mem;
+	uint32_t alloc_size;
+	uint32_t nbits = (uint32_t)len;
+
+	if (bitmap == NULL || nbits == 0)
+		return -EINVAL;
+
+	/* Allocate DLB2 bitmap control struct */
+	bm = rte_malloc("DLB2_PF",
+			sizeof(struct dlb2_bitmap),
+			RTE_CACHE_LINE_SIZE);
+
+	if (bm == NULL)
+		return -ENOMEM;
+
+	/* Allocate bitmap memory */
+	alloc_size = rte_bitmap_get_memory_footprint(nbits);
+	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
+	if (mem == NULL) {
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->map = rte_bitmap_init(len, mem, alloc_size);
+	if (bm->map == NULL) {
+		rte_free(mem);
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap == NULL)
+		return;
+
+	rte_free(bitmap->map);
+	rte_free(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++)
+		rte_bitmap_set(bitmap->map, i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	rte_bitmap_reset(bitmap->map);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set() - set a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
+				  unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_set(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+					unsigned int bit,
+					unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_set(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear() - clear a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
+				    unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_clear(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+					  unsigned int bit,
+					  unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_clear(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+						 unsigned int len)
+{
+	unsigned int i, j = 0;
+
+	if (bitmap  == NULL || bitmap->map == NULL || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			if (++j == len)
+				return i - j + 1;
+		} else {
+			j = 0;
+		}
+	}
+
+	/* No set bit range of length len? */
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			return i;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+	int weight = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			weight++;
+	}
+	return weight;
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				      bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+	int max_len = 0, len = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			len++;
+		} else {
+			if (len > max_len)
+				max_len = len;
+			len = 0;
+		}
+	}
+
+	if (len > max_len)
+		max_len = len;
+
+	return max_len;
+}
+
+/**
+ * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
+ * @dest: pointer to dlb2_bitmap structure, which will contain the results of
+ *	  the 'or' of src1 and src2.
+ * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
+ * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
+ *
+ * This function 'or's two bitmaps together and stores the result in a third
+ * bitmap. The source and destination bitmaps can be the same.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - One of the bitmaps is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
+				 struct dlb2_bitmap *src1,
+				 struct dlb2_bitmap *src2)
+{
+	unsigned int i, min;
+	int numset = 0;
+
+	if (dest  == NULL || dest->map == NULL ||
+	    src1  == NULL || src1->map == NULL ||
+	    src2  == NULL || src2->map == NULL)
+		return -EINVAL;
+
+	min = dest->len;
+	min = (min > src1->len) ? src1->len : min;
+	min = (min > src2->len) ? src2->len : min;
+
+	for (i = 0; i != min; i++) {
+		if  (rte_bitmap_get(src1->map, i) ||
+		     rte_bitmap_get(src2->map, i)) {
+			rte_bitmap_set(dest->map, i);
+			numset++;
+		} else {
+			rte_bitmap_clear(dest->map, i);
+		}
+	}
+
+	return numset;
+}
+
+#endif /*  __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
new file mode 100644
index 0000000..5531739
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_LIST_H
+#define __DLB2_OSDEP_LIST_H
+
+#include <rte_tailq.h>
+
+struct dlb2_list_entry {
+	TAILQ_ENTRY(dlb2_list_entry) node;
+};
+
+/* Dummy - just a struct definition */
+TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
+
+/* =================
+ * TAILQ Supplements
+ * =================
+ */
+
+#ifndef TAILQ_FOREACH_ENTRY
+#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)		\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter)						\
+		&& (ptr = container_of(iter, typeof(*(ptr)), name)); \
+	    (iter) = TAILQ_NEXT((iter), node))
+#endif
+
+#ifndef TAILQ_FOREACH_ENTRY_SAFE
+#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)	\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter) &&						\
+		(ptr = container_of(iter, typeof(*(ptr)), name)) &&\
+		((tvar) = TAILQ_NEXT((iter), node), 1);	\
+	    (iter) = (tvar))
+#endif
+
+/***********************/
+/*** List operations ***/
+/***********************/
+
+/**
+ * dlb2_list_init_head() - initialize the head of a list
+ * @head: list head
+ */
+static inline void dlb2_list_init_head(struct dlb2_list_head *head)
+{
+	TAILQ_INIT(head);
+}
+
+/**
+ * dlb2_list_add() - add an entry to a list
+ * @head: list head
+ * @entry: new list entry
+ */
+static inline void
+dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
+{
+	TAILQ_INSERT_TAIL(head, entry, node);
+}
+
+/**
+ * dlb2_list_del() - delete an entry from a list
+ * @entry: list entry
+ * @head: list head
+ */
+static inline void dlb2_list_del(struct dlb2_list_head *head,
+				 struct dlb2_list_entry *entry)
+{
+	TAILQ_REMOVE(head, entry, node);
+}
+
+/**
+ * dlb2_list_empty() - check if a list is empty
+ * @head: list head
+ *
+ * Return:
+ * Returns 1 if empty, 0 if not.
+ */
+static inline int dlb2_list_empty(struct dlb2_list_head *head)
+{
+	return TAILQ_EMPTY(head);
+}
+
+/**
+ * dlb2_list_splice() - splice a list
+ * @src_head: list to be added
+ * @ head: where src_head will be inserted
+ */
+static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
+				    struct dlb2_list_head *head)
+{
+	TAILQ_CONCAT(head, src_head, node);
+}
+
+/**
+ * DLB2_LIST_HEAD() - retrieve the head of the list
+ * @head: list head
+ * @type: type of the list variable
+ * @name: name of the list field within the containing struct
+ */
+#define DLB2_LIST_HEAD(head, type, name)                       \
+	(TAILQ_FIRST(&head) ?					\
+		container_of(TAILQ_FIRST(&head), type, name) :	\
+		NULL)
+
+/**
+ * DLB2_LIST_FOR_EACH() - iterate over a list
+ * @head: list head
+ * @ptr: pointer to struct containing a struct list
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ */
+#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
+	TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
+
+/**
+ * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
+ * an element is removed from the list while processing it.
+ * @ptr: pointer to struct containing a struct list
+ * @ptr_tmp: pointer to struct containing a struct list (temporary)
+ * @head: list head
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ * @iter_tmp: iterator variable (temporary)
+ */
+#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
+	TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
+
+#endif /*  __DLB2_OSDEP_LIST_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
new file mode 100644
index 0000000..0a48f7e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_TYPES_H
+#define __DLB2_OSDEP_TYPES_H
+
+#include <linux/types.h>
+
+#include <inttypes.h>
+#include <ctype.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Types for user mode PF PMD */
+typedef uint8_t         u8;
+typedef int8_t          s8;
+typedef uint16_t        u16;
+typedef int16_t         s16;
+typedef uint32_t        u32;
+typedef int32_t         s32;
+typedef uint64_t        u64;
+
+#define __iomem
+
+/* END types for user mode PF PMD */
+
+#endif /* __DLB2_OSDEP_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
new file mode 100644
index 0000000..43ecad4
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
@@ -0,0 +1,2527 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "dlb2_osdep_types.h"
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+	struct {
+		u32 isr_pend : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+	struct {
+		u32 vf0_reset_in_progress : 1;
+		u32 vf1_reset_in_progress : 1;
+		u32 vf2_reset_in_progress : 1;
+		u32 vf3_reset_in_progress : 1;
+		u32 vf4_reset_in_progress : 1;
+		u32 vf5_reset_in_progress : 1;
+		u32 vf6_reset_in_progress : 1;
+		u32 vf7_reset_in_progress : 1;
+		u32 vf8_reset_in_progress : 1;
+		u32 vf9_reset_in_progress : 1;
+		u32 vf10_reset_in_progress : 1;
+		u32 vf11_reset_in_progress : 1;
+		u32 vf12_reset_in_progress : 1;
+		u32 vf13_reset_in_progress : 1;
+		u32 vf14_reset_in_progress : 1;
+		u32 vf15_reset_in_progress : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+	struct {
+		u32 vec_mask : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+	struct {
+		u32 func_vf_bar_dis : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+	struct {
+		u32 total_vas : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+	struct {
+		u32 total_dir_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+	struct {
+		u32 total_ldb_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+	struct {
+		u32 total_dir_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+	struct {
+		u32 total_ldb_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+	struct {
+		u32 total_dir_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+	struct {
+		u32 total_ldb_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 meas : 1;
+		u32 debug : 7;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 cq_int_rearm : 1;
+		u32 dsi_error : 1;
+		u32 rsvd0 : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 rsvd0 : 3;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+	struct {
+		u32 qid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+	struct {
+		u32 vqid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+	struct {
+		u32 qid : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 debug : 8;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 isz : 1;
+		u32 dsi_error : 1;
+		u32 dlbrsvd : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 vf_synd0_parity : 1;
+		u32 vf_synd1_parity : 1;
+		u32 vf_synd2_parity : 1;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+	struct {
+		u32 sn_cfg_v : 1;
+		u32 fid_cfg_v : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS  0
+#define DLB2_CQ_ISR_MODE_MSI  1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI  3
+union dlb2_sys_ldb_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+	struct {
+		u32 keep_pf_ppid : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+	struct {
+		u32 illegal_hcw : 1;
+		u32 illegal_pp : 1;
+		u32 illegal_pasid : 1;
+		u32 illegal_qid : 1;
+		u32 disabled_qid : 1;
+		u32 illegal_ldb_qid_cfg : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+	struct {
+		u32 msix_0_ack : 1;
+		u32 msix_1_ack : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+	struct {
+		u32 msix_0_passthru : 1;
+		u32 msix_1_passthru : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED     0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+	struct {
+		u32 mode : 1;
+		u32 poll_mode : 1;
+		u32 poll_mask : 1;
+		u32 poll_lock : 1;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+	struct {
+		u32 cq : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 alarm : 1;
+		u32 cwd : 1;
+		u32 vf_pf_mb : 1;
+		u32 rsvd0 : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+	struct {
+		u32 qid_fid_limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+	struct {
+		u32 compress_code : 3;
+		u32 rsvd0 : 29;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+	(DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+	struct {
+		u32 count : 15;
+		u32 rsvd0 : 17;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+	struct {
+		u32 sn : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+	struct {
+		u32 mode : 3;
+		u32 slot : 4;
+		u32 rsvz0 : 1;
+		u32 grp : 1;
+		u32 rsvz1 : 1;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+	struct {
+		u32 en : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+	struct {
+		u32 write_pointer : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+	struct {
+		u32 base : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+	struct {
+		u32 limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+	struct {
+		u32 pop_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+	struct {
+		u32 push_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+	struct {
+		u32 depth : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+	struct {
+		u32 write_pointer : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 dlb_cor_alarm_enable : 1;
+		u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+		u32 cfg_64bytes_qe_dir_cq_mode : 1;
+		u32 pad_write_ldb : 1;
+		u32 pad_write_dir : 1;
+		u32 pad_first_write_ldb : 1;
+		u32 pad_first_write_dir : 1;
+		u32 rsvz0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+	struct {
+		u32 egress_credit_status_empty : 1;
+		u32 egress_credit_status_afull : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+		u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+		u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+		u32 chp_rop_pipe_credit_status_empty : 1;
+		u32 chp_rop_pipe_credit_status_afull : 1;
+		u32 qed_to_cq_pipe_credit_status_empty : 1;
+		u32 qed_to_cq_pipe_credit_status_afull : 1;
+		u32 egress_lsp_token_credit_status_empty : 1;
+		u32 egress_lsp_token_credit_status_afull : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 rsvz0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+	struct {
+		u32 sn_mode_0 : 3;
+		u32 rszv0 : 5;
+		u32 sn_mode_1 : 3;
+		u32 rszv1 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+	struct {
+		u32 unit_single_step_mode : 1;
+		u32 rr_en : 1;
+		u32 rszv0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+	struct {
+		u32 prio : 24;
+		u32 v : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+	struct {
+		u32 qid_p0 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p1 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p2 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p3 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+	struct {
+		u32 qid_p4 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p5 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p6 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p7 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+	struct {
+		u32 token_depth_select : 4;
+		u32 disable_wb_opt : 1;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+	struct {
+		u32 token_count : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+	struct {
+		u32 thresh : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+	(DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+	(DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+	struct {
+		u32 depth : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+	struct {
+		u32 cq : 8;
+		u32 qidix : 3;
+		u32 value : 1;
+		u32 nalb_haswork_v : 1;
+		u32 rlist_haswork_v : 1;
+		u32 slist_haswork_v : 1;
+		u32 inflight_ok_v : 1;
+		u32 aqed_nfull_v : 1;
+		u32 rsvz0 : 15;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+	struct {
+		u32 transfer : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+	struct {
+		u32 bw_range : 9;
+		u32 rsvz0 : 22;
+		u32 no_extra_credit : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+	struct {
+		u32 disab_atq_empty_arb : 1;
+		u32 inc_tok_unit_idle : 1;
+		u32 disab_rlist_pri : 1;
+		u32 inc_cmp_unit_idle : 1;
+		u32 rsvz0 : 2;
+		u32 dir_single_op : 1;
+		u32 dir_half_bw : 1;
+		u32 dir_single_out : 1;
+		u32 dir_disab_multi : 1;
+		u32 atq_single_op : 1;
+		u32 atq_half_bw : 1;
+		u32 atq_single_out : 1;
+		u32 atq_disab_multi : 1;
+		u32 dirrpl_single_op : 1;
+		u32 dirrpl_half_bw : 1;
+		u32 dirrpl_single_out : 1;
+		u32 lbrpl_single_op : 1;
+		u32 lbrpl_half_bw : 1;
+		u32 lbrpl_single_out : 1;
+		u32 ldb_single_op : 1;
+		u32 ldb_half_bw : 1;
+		u32 ldb_disab_multi : 1;
+		u32 atm_single_sch : 1;
+		u32 atm_single_cmp : 1;
+		u32 ldb_ce_tog_arb : 1;
+		u32 rsvz1 : 1;
+		u32 smon0_valid_sel : 2;
+		u32 smon0_value_sel : 1;
+		u32 smon0_compare_sel : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+	struct {
+		u32 chp_pf_reset_done : 1;
+		u32 rop_pf_reset_done : 1;
+		u32 lsp_pf_reset_done : 1;
+		u32 nalb_pf_reset_done : 1;
+		u32 ap_pf_reset_done : 1;
+		u32 dp_pf_reset_done : 1;
+		u32 qed_pf_reset_done : 1;
+		u32 dqed_pf_reset_done : 1;
+		u32 aqed_pf_reset_done : 1;
+		u32 sys_pf_reset_done : 1;
+		u32 pf_reset_active : 1;
+		u32 flrsm_state : 7;
+		u32 rsvd0 : 13;
+		u32 dlb_proc_reset_done : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+	struct {
+		u32 chp_pipeidle : 1;
+		u32 rop_pipeidle : 1;
+		u32 lsp_pipeidle : 1;
+		u32 nalb_pipeidle : 1;
+		u32 ap_pipeidle : 1;
+		u32 dp_pipeidle : 1;
+		u32 qed_pipeidle : 1;
+		u32 dqed_pipeidle : 1;
+		u32 aqed_pipeidle : 1;
+		u32 sys_pipeidle : 1;
+		u32 chp_unit_idle : 1;
+		u32 rop_unit_idle : 1;
+		u32 lsp_unit_idle : 1;
+		u32 nalb_unit_idle : 1;
+		u32 ap_unit_idle : 1;
+		u32 dp_unit_idle : 1;
+		u32 qed_unit_idle : 1;
+		u32 dqed_unit_idle : 1;
+		u32 aqed_unit_idle : 1;
+		u32 sys_unit_idle : 1;
+		u32 rsvd1 : 4;
+		u32 mstr_cfg_ring_idle : 1;
+		u32 mstr_cfg_mstr_idle : 1;
+		u32 mstr_flr_clkreq_b : 1;
+		u32 mstr_proc_idle : 1;
+		u32 mstr_proc_idle_masked : 1;
+		u32 rsvd0 : 2;
+		u32 dlb_func_idle : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+	struct {
+		u32 prochot : 1;
+		u32 pgcb_dlb_idle : 1;
+		u32 pgcb_dlb_pg_rdy_ack_b : 1;
+		u32 pmsm_pgcb_req_b : 1;
+		u32 pgbc_pmc_pg_req_b : 1;
+		u32 pmc_pgcb_pg_ack_b : 1;
+		u32 pmc_pgcb_fet_en_b : 1;
+		u32 pgcb_fet_en_b : 1;
+		u32 rsvz0 : 1;
+		u32 rsvz1 : 1;
+		u32 fuse_force_on : 1;
+		u32 fuse_proc_disable : 1;
+		u32 rsvz2 : 1;
+		u32 rsvz3 : 1;
+		u32 pm_fsm_d0tod3_ok : 1;
+		u32 pm_fsm_d3tod0_ok : 1;
+		u32 dlb_in_d3 : 1;
+		u32 rsvz4 : 7;
+		u32 pmsm : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+	struct {
+		u32 disable : 1;
+		u32 rsvz0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+	struct {
+		u32 isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+	struct {
+		u32 pf_isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+	struct {
+		u32 isr_pend : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+	struct {
+		u32 reset_in_progress : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+	struct {
+		u32 vf_msi_isr : 32;
+	} field;
+	u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
new file mode 100644
index 0000000..503fdf3
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
@@ -0,0 +1,1913 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_osdep_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_start_domain() - start a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: start domain arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int dlb2_hw_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp,
+			 bool vdev_request,
+			 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_request,
+		    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding unmap procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_finish_map_qid_procedures() - finish any pending map procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding map procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ *					interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+			  int port_id,
+			  bool is_ldb,
+			  bool vdev_request,
+			  unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+					 u32 *ldb_interrupts,
+					 u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+				 u32 *ldb_interrupts,
+				 u32 *dir_interrupts);
+
+/**
+ * dlb2_read_vf_intr_status() - read the VF interrupt status register
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function can be called from a VF's interrupt handler to determine
+ * which interrupts have fired. The first 31 bits correspond to CQ interrupt
+ * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
+ *
+ * Return:
+ * Returns a bit vector indicating which interrupt vectors are active.
+ */
+u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_intr_status() - ack VF interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function ACKs a VF's interrupts. Its interrupts argument should be the
+ * value returned by dlb2_read_vf_intr_status().
+ */
+void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function clears the VF's MSI interrupt pending register. Its interrupts
+ * argument should be contain the MSI vectors to ACK. For example, if MSI MME
+ * is in mode 0, then one bit 0 should ever be set.
+ */
+void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When done processing the PF mailbox request, this function unsets
+ * the PF's mailbox ISR register.
+ */
+void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
+
+/**
+ * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
+ *					vdevs
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the vdev->PF ISR fires, this function can be called to determine which
+ * vdev(s) are requesting service. This bitvector must be passed to
+ * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
+ * vdevs.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested service.
+ */
+u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ *
+ * When done processing all VF mailbox requests, this function unsets the VF's
+ * mailbox ISR register.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
+ *				    FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the VF FLR ISR fires, this function can be called to determine which
+ * VF(s) are requesting FLRs. This bitvector must passed to
+ * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
+ */
+u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF FLR requests, this function unsets the VF's FLR
+ * ISR register.
+ */
+void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF requests, this function communicates to the
+ * hardware that processing is complete.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
+			     u32 mbox_bitvec,
+			     u32 flr_bitvec);
+
+/**
+ * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the watchdog timer interrupt cause registers to
+ * determine which port(s) had a watchdog timeout, and notifies the
+ * application(s) that own the port(s).
+ */
+void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads and logs the alarm syndrome, then acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_request,
+			      unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of resources in use by the PF or a VF. It
+ * fills in the fields that args points to, except the following:
+ * - max_contiguous_atomic_inflights
+ * - max_contiguous_hist_list_entries
+ * - max_contiguous_ldb_credits
+ * - max_contiguous_dir_credits
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
+				   struct dlb2_get_num_resources_args *arg,
+				   bool vdev_request,
+				   unsigned int vdev_id);
+
+/**
+ * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
+ *				       the PF
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function sends a VF->PF mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the PF has processed
+ * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
+ * when the PF has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
+ *				 request
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * processing a VF->PF mailbox request. It should only be called after sending
+ * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vf_flr_complete() - check the status of a VF FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * executing the VF FLR. It should only be called after setting the VF's FLR
+ * bit.
+ */
+bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
+ *					vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function sends a PF->vdev mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the vdev has
+ * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
+ * determine when the vdev has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
+ *			       request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function returns a boolean indicating whether the vdev has finished
+ * processing a PF->vdev mailbox request. It should only be called after
+ * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
+			     unsigned int vf_id,
+			     void *data,
+			     int len);
+
+/**
+ * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
+			       unsigned int vf_id,
+			       void *data,
+			       int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_reset_vdev() - reset the hardware owned by a virtual device
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function resets the hardware owned by a vdev, by resetting the vdev's
+ * domains one by one.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function returns whether or not the vdev's resource assignments are
+ * locked. If locked, no resources can be added to or subtracted from the
+ * group.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_lock_vdev() - lock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sets a flag indicating that the vdev is using its resources.
+ * When the vdev is locked, its resource assignment cannot be changed.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_unlock_vdev() - unlock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function unlocks the vdev's resource assignment, allowing it to be
+ * modified.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of scheduling domains to assign to this vdev
+ *
+ * This function assigns num scheduling domains to the specified vdev. If the
+ * vdev already has domains assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB queues to assign to this vdev
+ *
+ * This function assigns num LDB queues to the specified vdev. If the vdev
+ * already has LDB queues assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports to the specified vdev. If the vdev
+ * already has LDB ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @cos: class-of-service ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports from class-of-service cos to the
+ * specified vdev. If the vdev already has LDB ports from this class-of-service
+ * assigned, this existing assignment is adjusted accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
+				   u32 id,
+				   u32 cos,
+				   u32 num);
+
+/**
+ * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR ports to assign to this vdev
+ *
+ * This function assigns num DIR ports to the specified vdev. If the vdev
+ * already has DIR ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB credit credits to assign to this vdev
+ *
+ * This function assigns num LDB credit to the specified vdev. If the vdev
+ * already has LDB credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR credits to assign to this vdev
+ *
+ * This function assigns num DIR credit to the specified vdev. If the vdev
+ * already has DIR credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of history list entries to assign to this vdev
+ *
+ * This function assigns num history list entries to the specified vdev. If the
+ * vdev already has history list entries assigned, this existing assignment is
+ * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
+ * this function may fail if a sufficiently large contiguous chunk is not
+ * available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of atomic inflights to assign to this vdev
+ *
+ * This function assigns num atomic inflights to the specified vdev. If the vdev
+ * already has atomic inflights assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of entries, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function takes any resources currently assigned to the vdev and
+ * reassigns them to the PF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_notify_vf() - send an alarm to a VF
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID
+ * @notification: notification
+ *
+ * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
+ *
+ * Return:
+ * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
+ */
+int dlb2_notify_vf(struct dlb2_hw *hw,
+		   unsigned int vf_id,
+		   u32 notification);
+
+/**
+ * dlb2_vdev_in_use() - query whether a virtual device is in use
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sends a mailbox request to the vdev to query whether the vdev
+ * is in use.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
+ * an internal error occurs.
+ */
+int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+enum dlb2_virt_mode {
+	DLB2_VIRT_NONE,
+	DLB2_VIRT_SRIOV,
+	DLB2_VIRT_SIOV,
+
+	/* NUM_DLB2_VIRT_MODES must be last */
+	NUM_DLB2_VIRT_MODES,
+};
+
+/**
+ * dlb2_hw_set_virt_mode() - set the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: either none, SR-IOV, or Scalable IOV.
+ *
+ * This function sets the virtualization mode of the device. This controls
+ * whether the device uses a software or hardware mailbox.
+ *
+ * This should be called by the PF driver when either SR-IOV or Scalable IOV is
+ * selected as the virtualization mechanism, and by the VF/VDEV driver during
+ * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
+ *
+ * Errors:
+ * EINVAL - Invalid mode.
+ */
+int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
+
+/**
+ * dlb2_hw_get_virt_mode() - get the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function gets the virtualization mode of the device.
+ */
+enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_register_sw_mbox() - register a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
+ * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
+ * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
+ * @inject_arg: user argument for pf2vdev_inject callback.
+ *
+ * When Scalable IOV is enabled, the VDCM must register a software mailbox for
+ * every virtual device during vdev creation.
+ *
+ * This function notifies the driver to use a software mailbox using the
+ * provided pointers, instead of the device's hardware mailbox. When the driver
+ * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
+ * go to the software mailbox instead of the hardware one. This is used in
+ * Scalable IOV virtualization.
+ */
+void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
+			      unsigned int vdev_id,
+			      u32 *vdev2pf_mbox,
+			      u32 *pf2vdev_mbox,
+			      void (*pf2vdev_inject)(void *),
+			      void *inject_arg);
+
+/**
+ * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function notifies the driver to stop using a previously registered
+ * software mailbox.
+ */
+void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ * @addr_lo: least-significant 32 bits of address.
+ * @data: 32 data bits.
+ *
+ * This sets up the CQ's IMS entry with the provided address and data values.
+ * This function should only be called if the device is configured for Scalable
+ * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
+ * not needed.
+ */
+void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb,
+				u32 addr_lo,
+				u32 data);
+
+/**
+ * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ *
+ * This clears the CQ's IMS entry, reverting it to its reset state.
+ */
+void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb);
+
+/**
+ * dlb2_hw_register_pasid() - register a vdev's PASID
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @pasid: the vdev's PASID.
+ *
+ * This function stores the user-supplied PASID, and uses it when configuring
+ * the vdev's CQs.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+int dlb2_hw_register_pasid(struct dlb2_hw *hw,
+			   unsigned int vdev_id,
+			   unsigned int pasid);
+
+/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vf_request,
+				unsigned int vf_id);
+
+/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ *	to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ *	port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ *	    cause the total bandwidth across all classes of service to exceed
+ *	    100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+enum dlb2_wd_tmo {
+	/* 40s watchdog timeout */
+	DLB2_WD_TMO_40S,
+	/* 10s watchdog timeout */
+	DLB2_WD_TMO_10S,
+	/* 1s watchdog timeout */
+	DLB2_WD_TMO_1S,
+
+	/* Must be last */
+	NUM_DLB2_WD_TMOS,
+};
+
+/**
+ * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
+ *	caller-specified timeout.
+ * @hw: dlb2_hw handle for a particular device.
+ * @tmo: watchdog timeout.
+ *
+ * This function should be called during device initialization and after reset.
+ * The watchdog timer interrupt must also be enabled per-CQ, using either
+ * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid timeout.
+ */
+int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
+
+/**
+ * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid directed port ID.
+ */
+int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid load-balanced port ID.
+ */
+int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *	ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * load-balanced CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_ldb_cq_interrupt().
+ */
+void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical directed port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * directed CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_dir_cq_interrupt().
+ */
+void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+#endif /* __DLB2_RESOURCE_H */
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
new file mode 100644
index 0000000..3c51cab
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -0,0 +1,621 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "base/dlb2_resource.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_regs.h"
+#include "dlb2_main.h"
+#include "../dlb2_user.h"
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+
+#define PF_ID_ZERO 0	/* PF ONLY! */
+#define NO_OWNER_VF 0	/* PF ONLY! */
+#define NOT_VF_REQ false /* PF ONLY! */
+
+#define DLB2_PCI_CFG_SPACE_SIZE 256
+#define DLB2_PCI_CAP_POINTER 0x34
+#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
+#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
+#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
+#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
+#define DLB2_PCI_EXT_CAP_ID_ERR 1
+#define DLB2_PCI_ERR_UNCOR_MASK 8
+#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
+
+#define DLB2_PCI_EXP_DEVCTL 8
+#define DLB2_PCI_LNKCTL 16
+#define DLB2_PCI_SLTCTL 24
+#define DLB2_PCI_RTCTL 28
+#define DLB2_PCI_EXP_DEVCTL2 40
+#define DLB2_PCI_LNKCTL2 48
+#define DLB2_PCI_SLTCTL2 56
+#define DLB2_PCI_CMD 4
+#define DLB2_PCI_X_CMD 2
+#define DLB2_PCI_EXP_DEVSTA 10
+#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
+#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
+
+#define DLB2_PCI_CAP_ID_EXP       0x10
+#define DLB2_PCI_CAP_ID_MSIX      0x11
+#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
+#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
+#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
+
+#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
+#define DLB2_PCI_PRI_ALLOC_REQ           0xC
+#define DLB2_PCI_PRI_CTRL                0x4
+#define DLB2_PCI_MSIX_FLAGS              0x2
+#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
+#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
+#define DLB2_PCI_ERR_ROOT_STATUS         0x30
+#define DLB2_PCI_ERR_COR_STATUS          0x10
+#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
+#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
+#define DLB2_PCI_ACS_CAP                 0x4
+#define DLB2_PCI_ACS_CTRL                0x6
+#define DLB2_PCI_ACS_SV                  0x1
+#define DLB2_PCI_ACS_RR                  0x4
+#define DLB2_PCI_ACS_CR                  0x8
+#define DLB2_PCI_ACS_UF                  0x10
+#define DLB2_PCI_ACS_EC                  0x20
+
+/* Stubs: Allow building partial probe patch */
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	int ret = 0;
+	RTE_SET_USED(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+/* End stubs */
+
+static int
+dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint32_t hdr;
+	size_t sz;
+	int pos;
+
+	pos = DLB2_PCI_CFG_SPACE_SIZE;
+	sz = sizeof(hdr);
+
+	while (pos > 0xFF) {
+		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
+			return -1;
+
+		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
+			return pos;
+
+		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint8_t pos;
+	int ret;
+	uint16_t hdr;
+
+	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
+	pos &= 0xFC;
+
+	if (ret != 1)
+		return -1;
+
+	while (pos > 0x3F) {
+		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
+		if (ret != 2)
+			return -1;
+
+		if (DLB2_PCI_CAP_ID(hdr) == id)
+			return pos;
+
+		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
+			return -1;
+
+		pos = DLB2_PCI_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+	rte_spinlock_init(&dlb2_dev->resource_mutex);
+	rte_spinlock_init(&dlb2_dev->measurement_lock);
+
+	return 0;
+}
+
+static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
+{
+	u32 retries = 0;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
+		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+		union dlb2_cfg_mstr_cfg_pm_status pm_st;
+		u32 addr;
+
+		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+			break;
+
+		rte_delay_ms(1);
+	};
+
+	if (retries == DLB2_READY_RETRY_LIMIT) {
+		printf("[%s()] wait for device ready timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+struct dlb2_dev *
+dlb2_probe(struct rte_pci_device *pdev)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret = 0;
+
+	DLB2_INFO(dlb2_dev, "probe\n");
+
+	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
+			      RTE_CACHE_LINE_SIZE);
+
+	if (dlb2_dev == NULL) {
+		ret = -ENOMEM;
+		goto dlb2_dev_malloc_fail;
+	}
+
+	/* PCI Bus driver has already mapped bar space into process.
+	 * Save off our IO register and FUNC addresses.
+	 */
+
+	/* BAR 0 */
+	if (pdev->mem_resource[0].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
+	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.func_kva,
+		  (void *)dlb2_dev->hw.func_phys_addr,
+		  (void *)(pdev->mem_resource[0].len));
+
+	/* BAR 2 */
+	if (pdev->mem_resource[2].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
+	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.csr_kva,
+		  (void *)dlb2_dev->hw.csr_phys_addr,
+		  (void *)(pdev->mem_resource[2].len));
+
+	dlb2_dev->pdev = pdev;
+
+	/* PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb2_pf_enable_pm(dlb2_dev);
+
+	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb2_pf_reset(dlb2_dev);
+	if (ret)
+		goto dlb2_reset_fail;
+
+	ret = dlb2_pf_init_driver_state(dlb2_dev);
+	if (ret)
+		goto init_driver_state_fail;
+
+	ret = dlb2_resource_init(&dlb2_dev->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	return dlb2_dev;
+
+resource_init_fail:
+	dlb2_resource_free(&dlb2_dev->hw);
+init_driver_state_fail:
+dlb2_reset_fail:
+pci_mmap_bad_addr:
+wait_for_device_ready_fail:
+	rte_free(dlb2_dev);
+dlb2_dev_malloc_fail:
+	rte_errno = ret;
+	return NULL;
+}
+
+int
+dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
+{
+	int ret = 0;
+	int i = 0;
+	uint32_t dword[16];
+	uint16_t cmd;
+	off_t off;
+
+	uint16_t dev_ctl_word;
+	uint16_t dev_ctl2_word;
+	uint16_t lnk_word;
+	uint16_t lnk_word2;
+	uint16_t slt_word;
+	uint16_t slt_word2;
+	uint16_t rt_ctl_word;
+	uint32_t pri_reqs_dword;
+	uint16_t pri_ctrl_word;
+
+	int pcie_cap_offset;
+	int pri_cap_offset;
+	int msix_cap_offset;
+	int err_cap_offset;
+	int acs_cap_offset;
+	int wait_count;
+
+	uint16_t devsta_busy_word;
+	uint16_t devctl_word;
+
+	struct rte_pci_device *pdev = dlb2_dev->pdev;
+
+	/* Save PCI config state */
+
+	for (i = 0; i < 16; i++) {
+		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
+			return ret;
+	}
+
+	pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
+
+	if (pcie_cap_offset < 0) {
+		printf("[%s()] failed to find the pcie capability\n",
+		       __func__);
+		return pcie_cap_offset;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
+		dev_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
+		lnk_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
+		slt_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_RTCTL;
+	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
+		rt_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
+		dev_ctl2_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
+		lnk_word2 = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
+		slt_word2 = 0;
+
+	off = DLB2_PCI_EXT_CAP_ID_PRI;
+	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (pri_cap_offset >= 0) {
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
+			pri_reqs_dword = 0;
+	}
+
+	/* clear the PCI command register before issuing the FLR */
+
+	off = DLB2_PCI_CMD;
+	cmd = 0;
+	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+		printf("[%s()] failed to write the pci command\n",
+		       __func__);
+		return ret;
+	}
+
+	/* issue the FLR */
+	for (wait_count = 0; wait_count < 4; wait_count++) {
+		int sleep_time;
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
+		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to read the pci device status\n",
+			       __func__);
+			return ret;
+		}
+
+		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
+			break;
+
+		sleep_time = (1 << (wait_count)) * 100;
+		rte_delay_ms(sleep_time);
+	}
+
+	if (wait_count == 4) {
+		printf("[%s()] wait for pci pending transactions timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to read the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
+
+	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to write the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	rte_delay_ms(100);
+
+	/* Restore PCI config state */
+
+	if (pcie_cap_offset >= 0) {
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie device control at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_RTCTL;
+		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	if (pri_cap_offset >= 0) {
+		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
+
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
+		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ERR;
+	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (err_cap_offset >= 0) {
+		uint32_t tmp;
+
+		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	for (i = 16; i > 0; i--) {
+		off = (i - 1) * 4;
+		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_CMD;
+	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
+		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+			printf("[%s()] failed to write the pci command\n",
+			       __func__);
+			return ret;
+		}
+	}
+
+	msix_cap_offset = dlb2_pci_find_capability(pdev,
+						   DLB2_PCI_CAP_ID_MSIX);
+	if (msix_cap_offset >= 0) {
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
+			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ACS;
+	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (acs_cap_offset >= 0) {
+		uint16_t acs_cap, acs_ctrl, acs_mask;
+		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
+		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
+			acs_cap = 0;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
+		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
+		acs_ctrl |= (acs_cap & acs_mask);
+
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
+		acs_mask |= DLB2_PCI_ACS_EC;
+		acs_ctrl &= ~acs_mask;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
new file mode 100644
index 0000000..f082a94
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -0,0 +1,98 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+#endif
+
+#include "base/dlb2_hw_types.h"
+#include "../dlb2_user.h"
+
+#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
+
+struct dlb2_dev;
+
+struct dlb2_port_memory {
+	struct dlb2_list_head list;
+	void *cq_base;
+	bool valid;
+};
+
+struct dlb2_dev {
+	struct rte_pci_device *pdev;
+	struct dlb2_hw hw;
+	/* struct list_head list; */
+	struct device *dlb2_device;
+	bool domain_reset_failed;
+	/* The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	rte_spinlock_t resource_mutex;
+	rte_spinlock_t measurement_lock;
+	bool worker_launched;
+	u8 revision;
+};
+
+struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
+int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
+int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+
+#endif /* __DLB2_MAIN_H */
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
new file mode 100644
index 0000000..ae4d785
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -0,0 +1,196 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_devargs.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_pci.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+#include "dlb2_main.h"
+#include "base/dlb2_hw_types.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_resource.h"
+
+static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
+
+/* Stubs: Allow building partial probe patch */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(arg);
+	RTE_SET_USED(vdev_req);
+	RTE_SET_USED(vdev_id);
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+/* End stubs */
+
+static void
+dlb2_pf_iface_fn_ptrs_init(void)
+{
+/* flexible iface fn ptr assignments will go here */
+}
+
+/* PCI DEV HOOKS */
+static int
+dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
+{
+	int ret = 0;
+	struct rte_pci_device *pci_dev;
+	struct dlb2_devargs dlb2_args = {
+		.socket_id = rte_socket_id(),
+		.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+		.num_dir_credits_override = -1,
+		.qid_depth_thresholds = { {0} },
+		.cos_id = DLB2_COS_DEFAULT
+	};
+	struct dlb2_eventdev *dlb2;
+
+	DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
+		     eventdev->data->dev_id, eventdev->data->socket_id);
+
+	dlb2_pf_iface_fn_ptrs_init();
+
+	pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
+
+		/* Probe the DLB2 PF layer */
+		dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
+
+		if (dlb2->qm_instance.pf_dev == NULL) {
+			DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
+				     rte_errno);
+			ret = -rte_errno;
+			goto dlb2_probe_failed;
+		}
+
+		/* Were we invoked with runtime parameters? */
+		if (pci_dev->device.devargs) {
+			ret = dlb2_parse_params(pci_dev->device.devargs->args,
+						pci_dev->device.devargs->name,
+						&dlb2_args);
+			if (ret) {
+				DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
+					     ret, rte_errno);
+				goto dlb2_probe_failed;
+			}
+		}
+
+		ret = dlb2_primary_eventdev_probe(eventdev,
+						  event_dlb2_pf_name,
+						  &dlb2_args);
+	} else {
+		ret = dlb2_secondary_eventdev_probe(eventdev,
+						    event_dlb2_pf_name);
+	}
+	if (ret)
+		goto dlb2_probe_failed;
+
+	DLB2_LOG_INFO("DLB2 PF Probe success\n");
+
+	return 0;
+
+dlb2_probe_failed:
+
+	DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
+
+	return ret;
+}
+
+#define EVENTDEV_INTEL_VENDOR_ID 0x8086
+
+static const struct rte_pci_id pci_id_dlb2_map[] = {
+	{
+		RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
+			       PCI_DEVICE_ID_INTEL_DLB2_PF)
+	},
+	{
+		.vendor_id = 0,
+	},
+};
+
+static int
+event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
+		     struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
+					     sizeof(struct dlb2_eventdev),
+					     dlb2_eventdev_pci_init,
+					     event_dlb2_pf_name);
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+}
+
+static int
+event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_remove(pci_dev, NULL);
+
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+
+}
+
+static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
+	.id_table = pci_id_dlb2_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe = event_dlb2_pci_probe,
+	.remove = event_dlb2_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
-- 
2.6.4

[dpdk-dev] [PATCH v10 07/23] event/dlb2: add flexible interface

[Timothy McDaniel]

This commit introduces the flexible interface. This
interface allows the core code to operate in PF mode (direct
hardware access) or bifurcated mode (hardware configured via
kernel driver). This driver currently only supports PF mode
but bifurcated mode will be added in a future DPDK patch-set.
Note that the flexible interface is not used for data path
operations, and thus there are no performance concerns
related to the use of function pointers.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_iface.c | 28 ++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.h | 29 +++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build  |  1 +
 3 files changed, 58 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h

diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
new file mode 100644
index 0000000..0fc9991
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+
+#include "dlb2_priv.h"
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+void (*dlb2_iface_low_level_io_init)(void);
+
+int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+				     uint8_t *revision);
+
+void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+				   enum dlb2_cq_poll_modes *mode);
+
+int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
new file mode 100644
index 0000000..4fb416e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_IFACE_H_
+#define _DLB2_IFACE_H_
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+extern void (*dlb2_iface_low_level_io_init)(void);
+
+extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+					    uint8_t *revision);
+
+extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+					  enum dlb2_cq_poll_modes *mode);
+
+extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
+
+#endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 2855dd5..c2fb347 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -8,6 +8,7 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
 endif
 
 sources = files('dlb2.c',
+		'dlb2_iface.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c'
 )
-- 
2.6.4

[dpdk-dev] [PATCH v10 08/23] event/dlb2: add probe-time hardware init

[Timothy McDaniel]

This commit adds probe-time low level hardware
initialization.  It also adds probe-time init for both
primary and secondary DPDK processes.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 188 +++++++++++++++++++-
 drivers/event/dlb2/meson.build             |   3 +-
 drivers/event/dlb2/pf/base/dlb2_resource.c | 274 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  23 +--
 drivers/event/dlb2/pf/dlb2_main.h          |   1 -
 drivers/event/dlb2/pf/dlb2_pf.c            |  78 ++++++--
 6 files changed, 523 insertions(+), 44 deletions(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 1b11979..16653db 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -31,6 +31,7 @@
 #include <rte_string_fns.h>
 
 #include "dlb2_priv.h"
+#include "dlb2_iface.h"
 #include "dlb2_inline_fns.h"
 
 /*
@@ -40,10 +41,108 @@
 #if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
 #error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
 #endif
+static struct rte_event_dev_info evdev_dlb2_default_info = {
+	.driver_name = "", /* probe will set */
+	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
+	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
+	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
+#else
+	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
+#endif
+	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
+	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
+	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
+	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
+	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
+	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+	.max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
+	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
+			  RTE_EVENT_DEV_CAP_EVENT_QOS |
+			  RTE_EVENT_DEV_CAP_BURST_MODE |
+			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
+			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
+			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
+};
 
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/* override defaults with value(s) provided on command line */
+static void
+dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
+				 int *qid_depth_thresholds)
+{
+	int q;
+
+	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
+		if (qid_depth_thresholds[q] != 0)
+			dlb2->ev_queues[q].depth_threshold =
+				qid_depth_thresholds[q];
+	}
+}
+
+static int
+dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
+	int ret;
+
+	/* Query driver resources provisioned for this device */
+
+	ret = dlb2_iface_get_num_resources(handle,
+					   &dlb2->hw_rsrc_query_results);
+	if (ret) {
+		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
+		return ret;
+	}
+
+	/* Complete filling in device resource info returned to evdev app,
+	 * overriding any default values.
+	 * The capabilities (CAPs) were set at compile time.
+	 */
+
+	evdev_dlb2_default_info.max_event_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	evdev_dlb2_default_info.max_event_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	evdev_dlb2_default_info.max_num_events =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	/* Save off values used when creating the scheduling domain. */
+
+	handle->info.num_sched_domains =
+		dlb2->hw_rsrc_query_results.num_sched_domains;
+
+	handle->info.hw_rsrc_max.nb_events_limit =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	handle->info.hw_rsrc_max.num_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues +
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.num_ldb_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	handle->info.hw_rsrc_max.num_ldb_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	handle->info.hw_rsrc_max.num_dir_ports =
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.reorder_window_size =
+		dlb2->hw_rsrc_query_results.num_hist_list_entries;
+
+	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
+
+	return 0;
+}
+
 #define DLB2_BASE_10 10
 
 static int
@@ -235,14 +334,71 @@ set_qid_depth_thresh(const char *key __rte_unused,
 	return 0;
 }
 
+static void
+dlb2_entry_points_init(struct rte_eventdev *dev)
+{
+	RTE_SET_USED(dev);
+
+	/* Eventdev PMD entry points */
+}
+
 int
 dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    const char *name,
 			    struct dlb2_devargs *dlb2_args)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
-	RTE_SET_USED(dlb2_args);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	dlb2->event_dev = dev; /* backlink */
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	dlb2->max_num_events_override = dlb2_args->max_num_events;
+	dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
+	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
+					    &dlb2->revision);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_hardware_init(&dlb2->qm_instance);
+
+	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
+			     err);
+		return err;
+	}
+
+	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	dlb2_init_queue_depth_thresholds(dlb2,
+					 dlb2_args->qid_depth_thresholds.val);
 
 	return 0;
 }
@@ -251,8 +407,30 @@ int
 dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
 			      const char *name)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index c2fb347..d7390ec 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -10,7 +10,8 @@ endif
 sources = files('dlb2.c',
 		'dlb2_iface.c',
 		'pf/dlb2_main.c',
-		'pf/dlb2_pf.c'
+		'pf/dlb2_pf.c',
+		'pf/base/dlb2_resource.c'
 )
 
 headers = files()
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
new file mode 100644
index 0000000..6de8b95
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_mbox.h"
+#include "dlb2_osdep.h"
+#include "dlb2_osdep_bitmap.h"
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+	int i;
+
+	dlb2_list_init_head(&domain->used_ldb_queues);
+	dlb2_list_init_head(&domain->used_dir_pq_pairs);
+	dlb2_list_init_head(&domain->avail_ldb_queues);
+	dlb2_list_init_head(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+	int i;
+
+	dlb2_list_init_head(&rsrc->avail_domains);
+	dlb2_list_init_head(&rsrc->used_domains);
+	dlb2_list_init_head(&rsrc->avail_ldb_queues);
+	dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
+	arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
+	arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
+	arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb2_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	struct dlb2_list_entry *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. This is application
+	 * dependent, but the driver interleaves port IDs as much as possible
+	 * to reduce the likelihood of this. This initial allocation maximizes
+	 * the average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	/* Zero-out resource tracking data structures */
+	memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+	memset(&hw->pf, 0, sizeof(hw->pf));
+
+	dlb2_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+		dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+		dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_domains, list);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_ldb_queues, list);
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+		struct dlb2_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
+			      &port->func_list);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+				DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+	if (ret)
+		goto unwind;
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+					DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+		if (ret)
+			goto unwind;
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb2_resource_free(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+	r0.field.disable = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 3c51cab..210b3186 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -19,6 +19,7 @@
 #include "dlb2_main.h"
 #include "../dlb2_user.h"
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 
 #define PF_ID_ZERO 0	/* PF ONLY! */
@@ -72,27 +73,6 @@
 #define DLB2_PCI_ACS_UF                  0x10
 #define DLB2_PCI_ACS_EC                  0x20
 
-/* Stubs: Allow building partial probe patch */
-void dlb2_resource_free(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-int dlb2_resource_init(struct dlb2_hw *hw)
-{
-	int ret = 0;
-	RTE_SET_USED(hw);
-
-	return ret;
-}
-
-void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-/* End stubs */
-
 static int
 dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
 {
@@ -149,7 +129,6 @@ static int
 dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
 {
 	rte_spinlock_init(&dlb2_dev->resource_mutex);
-	rte_spinlock_init(&dlb2_dev->measurement_lock);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index f082a94..f3bee71 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -38,7 +38,6 @@ struct dlb2_dev {
 	 * hardware registers.
 	 */
 	rte_spinlock_t resource_mutex;
-	rte_spinlock_t measurement_lock;
 	bool worker_launched;
 	u8 revision;
 };
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index ae4d785..7d64309 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -32,6 +32,7 @@
 #include <rte_string_fns.h>
 
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 #include "dlb2_main.h"
 #include "base/dlb2_hw_types.h"
@@ -40,35 +41,82 @@
 
 static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
 
-/* Stubs: Allow building partial probe patch */
-int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
-			      struct dlb2_get_num_resources_args *arg,
-			      bool vdev_req,
-			      unsigned int vdev_id)
+static void
+dlb2_pf_low_level_io_init(void)
+{
+	int i;
+	/* Addresses will be initialized at port create */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		/* First directed ports */
+		dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
+
+		/* Now load balanced ports */
+		dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
+	}
+}
+
+static int
+dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
+{
+	RTE_SET_USED(handle);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+static int
+dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
+			   uint8_t *revision)
 {
-	RTE_SET_USED(hw);
-	RTE_SET_USED(arg);
-	RTE_SET_USED(vdev_req);
-	RTE_SET_USED(vdev_id);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	*revision = dlb2_dev->revision;
 
 	return 0;
 }
 
-void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+static void
+dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+	dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_num_resources_args *rsrcs)
 {
-	RTE_SET_USED(hw);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
 }
 
-void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+static int
+dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
+			 enum dlb2_cq_poll_modes *mode)
 {
-	RTE_SET_USED(hw);
+	RTE_SET_USED(handle);
+
+	*mode = DLB2_CQ_POLL_MODE_SPARSE;
+
+	return 0;
 }
-/* End stubs */
 
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
-/* flexible iface fn ptr assignments will go here */
+	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
+	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
+	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
+	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
+	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v10 09/23] event/dlb2: add xstats

[Timothy McDaniel]

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
Reviewed-by: Chen, Mike Ximing <mike.ximing.chen@intel.com>
---
 drivers/event/dlb2/dlb2.c        |   35 +-
 drivers/event/dlb2/dlb2_xstats.c | 1235 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build   |    1 +
 3 files changed, 1268 insertions(+), 3 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 16653db..925824e 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,6 +70,21 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/*
+ * DUMMY - added so that xstats path will compile/link.
+ * Will be replaced by real version in a subsequent
+ * patch.
+ */
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	RTE_SET_USED(dlb2);
+	RTE_SET_USED(queue);
+
+	return 0;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -337,9 +352,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
-	RTE_SET_USED(dev);
-
-	/* Eventdev PMD entry points */
+	/* Expose PMD's eventdev interface */
+	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dump             = dlb2_eventdev_dump,
+		.xstats_get       = dlb2_eventdev_xstats_get,
+		.xstats_get_names = dlb2_eventdev_xstats_get_names,
+		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
+		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+	};
+
+	dev->dev_ops = &dlb2_eventdev_entry_ops;
 }
 
 int
@@ -391,6 +413,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Complete xtstats runtime initialization */
+	err = dlb2_xstats_init(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
+		return err;
+	}
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
new file mode 100644
index 0000000..a274779
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_xstats.c
@@ -0,0 +1,1235 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eventdev.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+enum dlb2_xstats_type {
+	/* common to device and port */
+	rx_ok,				/**< Receive an event */
+	rx_drop,                        /**< Error bit set in received QE */
+	rx_interrupt_wait,		/**< Wait on an interrupt */
+	rx_umonitor_umwait,		/**< Block using umwait */
+	tx_ok,				/**< Transmit an event */
+	total_polls,			/**< Call dequeue_burst */
+	zero_polls,			/**< Call dequeue burst and return 0 */
+	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
+	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
+	tx_nospc_inflight_max,		/**< Reach the new_event_threshold */
+	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
+	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
+	/* device specific */
+	nb_events_limit,
+	inflight_events,
+	ldb_pool_size,
+	dir_pool_size,
+	/* port specific */
+	tx_new,				/**< Send an OP_NEW event */
+	tx_fwd,				/**< Send an OP_FORWARD event */
+	tx_rel,				/**< Send an OP_RELEASE event */
+	tx_implicit_rel,		/**< Issue an implicit event release */
+	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED event */
+	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL event */
+	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event */
+	tx_sched_directed,		/**< Send a directed event */
+	tx_invalid,                     /**< Send an event with an invalid op */
+	outstanding_releases,		/**< # of releases a port owes */
+	max_outstanding_releases,	/**< max # of releases a port can owe */
+	rx_sched_ordered,		/**< Dequeue an ordered event */
+	rx_sched_unordered,		/**< Dequeue an unordered event */
+	rx_sched_atomic,		/**< Dequeue an atomic event */
+	rx_sched_directed,		/**< Dequeue an directed event */
+	rx_sched_invalid,               /**< Dequeue event sched type invalid */
+	/* common to port and queue */
+	is_configured,			/**< Port is configured */
+	is_load_balanced,		/**< Port is LDB */
+	hw_id,				/**< Hardware ID */
+	/* queue specific */
+	num_links,			/**< Number of ports linked */
+	sched_type,			/**< Queue sched type */
+	enq_ok,				/**< # events enqueued to the queue */
+	current_depth,			/**< Current queue depth */
+	depth_threshold,		/**< Programmed depth threshold */
+	depth_le50_threshold,
+	/**< Depth LE to 50% of the configured hardware threshold */
+	depth_gt50_le75_threshold,
+	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold */
+	depth_gt75_le100_threshold,
+	/**< Depth GT 75%. but LE to the configured hardware threshold */
+	depth_gt100_threshold
+	/**< Depth GT 100% of the configured hw threshold */
+};
+
+typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
+		uint16_t obj_idx, /* port or queue id */
+		enum dlb2_xstats_type stat, int extra_arg);
+
+enum dlb2_xstats_fn_type {
+	DLB2_XSTATS_FN_DEV,
+	DLB2_XSTATS_FN_PORT,
+	DLB2_XSTATS_FN_QUEUE
+};
+
+struct dlb2_xstats_entry {
+	struct rte_event_dev_xstats_name name;
+	uint64_t reset_value; /* an offset to be taken away to emulate resets */
+	enum dlb2_xstats_fn_type fn_id;
+	enum dlb2_xstats_type stat;
+	enum rte_event_dev_xstats_mode mode;
+	int extra_arg;
+	uint16_t obj_idx;
+	uint8_t reset_allowed; /* when set, this value can be reset */
+};
+
+/* Some device stats are simply a summation of the corresponding port values */
+static uint64_t
+dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
+			     int which_stat)
+{
+	int i;
+	uint64_t val = 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
+
+		if (!port->setup_done)
+			continue;
+
+		switch (which_stat) {
+		case rx_ok:
+			val += port->stats.traffic.rx_ok;
+			break;
+		case rx_drop:
+			val += port->stats.traffic.rx_drop;
+			break;
+		case rx_interrupt_wait:
+			val += port->stats.traffic.rx_interrupt_wait;
+			break;
+		case rx_umonitor_umwait:
+			val += port->stats.traffic.rx_umonitor_umwait;
+			break;
+		case tx_ok:
+			val += port->stats.traffic.tx_ok;
+			break;
+		case total_polls:
+			val += port->stats.traffic.total_polls;
+			break;
+		case zero_polls:
+			val += port->stats.traffic.zero_polls;
+			break;
+		case tx_nospc_ldb_hw_credits:
+			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
+			break;
+		case tx_nospc_dir_hw_credits:
+			val += port->stats.traffic.tx_nospc_dir_hw_credits;
+			break;
+		case tx_nospc_inflight_max:
+			val += port->stats.traffic.tx_nospc_inflight_max;
+			break;
+		case tx_nospc_new_event_limit:
+			val += port->stats.traffic.tx_nospc_new_event_limit;
+			break;
+		case tx_nospc_inflight_credits:
+			val += port->stats.traffic.tx_nospc_inflight_credits;
+			break;
+		default:
+			return -1;
+		}
+	}
+	return val;
+}
+
+static uint64_t
+get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
+	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	switch (type) {
+	case rx_ok:
+	case rx_drop:
+	case rx_interrupt_wait:
+	case rx_umonitor_umwait:
+	case tx_ok:
+	case total_polls:
+	case zero_polls:
+	case tx_nospc_ldb_hw_credits:
+	case tx_nospc_dir_hw_credits:
+	case tx_nospc_inflight_max:
+	case tx_nospc_new_event_limit:
+	case tx_nospc_inflight_credits:
+		return dlb2_device_traffic_stat_get(dlb2, type);
+	case nb_events_limit:
+		return dlb2->new_event_limit;
+	case inflight_events:
+		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
+	case ldb_pool_size:
+		return dlb2->num_ldb_credits;
+	case dir_pool_size:
+		return dlb2->num_dir_credits;
+	default: return -1;
+	}
+}
+
+static uint64_t
+get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
+
+	switch (type) {
+	case rx_ok: return ev_port->stats.traffic.rx_ok;
+
+	case rx_drop: return ev_port->stats.traffic.rx_drop;
+
+	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
+
+	case rx_umonitor_umwait:
+		return ev_port->stats.traffic.rx_umonitor_umwait;
+
+	case tx_ok: return ev_port->stats.traffic.tx_ok;
+
+	case total_polls: return ev_port->stats.traffic.total_polls;
+
+	case zero_polls: return ev_port->stats.traffic.zero_polls;
+
+	case tx_nospc_ldb_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
+
+	case tx_nospc_dir_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
+
+	case tx_nospc_inflight_max:
+		return ev_port->stats.traffic.tx_nospc_inflight_max;
+
+	case tx_nospc_new_event_limit:
+		return ev_port->stats.traffic.tx_nospc_new_event_limit;
+
+	case tx_nospc_inflight_credits:
+		return ev_port->stats.traffic.tx_nospc_inflight_credits;
+
+	case is_configured: return ev_port->setup_done;
+
+	case is_load_balanced: return !ev_port->qm_port.is_directed;
+
+	case hw_id: return ev_port->qm_port.id;
+
+	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
+
+	case tx_fwd: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
+
+	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
+
+	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
+
+	case tx_sched_ordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case tx_sched_unordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case tx_sched_atomic:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case tx_sched_directed:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case tx_invalid: return ev_port->stats.tx_invalid;
+
+	case outstanding_releases: return ev_port->outstanding_releases;
+
+	case max_outstanding_releases:
+		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	case rx_sched_ordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case rx_sched_unordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case rx_sched_atomic:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case rx_sched_directed:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
+
+	default: return -1;
+	}
+}
+
+static uint64_t
+dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
+{
+	int port = 0;
+	uint64_t tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
+
+	return tally;
+}
+
+static uint64_t
+dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
+{
+	int port = 0;
+	uint64_t enq_ok_tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
+
+	return enq_ok_tally;
+}
+
+static uint64_t
+get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_queue *ev_queue =
+		&dlb2->ev_queues[obj_idx];
+
+	switch (type) {
+	case is_configured: return ev_queue->setup_done;
+
+	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
+
+	case hw_id: return ev_queue->qm_queue.id;
+
+	case num_links: return ev_queue->num_links;
+
+	case sched_type: return ev_queue->qm_queue.sched_type;
+
+	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
+
+	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
+
+	case depth_threshold: return ev_queue->depth_threshold;
+
+	case depth_le50_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_LE50);
+
+	case depth_gt50_le75_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT50_LE75);
+
+	case depth_gt75_le100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT75_LE100);
+
+	case depth_gt100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT100);
+
+	default: return -1;
+	}
+}
+
+int
+dlb2_xstats_init(struct dlb2_eventdev *dlb2)
+{
+	/*
+	 * define the stats names and types. Used to build up the device
+	 * xstats array
+	 * There are multiple set of stats:
+	 *   - device-level,
+	 *   - per-port,
+	 *   - per-qid,
+	 *
+	 * For each of these sets, we have three parallel arrays, one for the
+	 * names, the other for the stat type parameter to be passed in the fn
+	 * call to get that stat. The third array allows resetting or not.
+	 * All these arrays must be kept in sync
+	 */
+	static const char * const dev_stats[] = {
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"nb_events_limit",
+		"inflight_events",
+		"ldb_pool_size",
+		"dir_pool_size",
+	};
+	static const enum dlb2_xstats_type dev_types[] = {
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		nb_events_limit,
+		inflight_events,
+		ldb_pool_size,
+		dir_pool_size,
+	};
+	/* Note: generated device stats are not allowed to be reset. */
+	static const uint8_t dev_reset_allowed[] = {
+		0, /* rx_ok */
+		0, /* rx_drop */
+		0, /* rx_interrupt_wait */
+		0, /* rx_umonitor_umwait */
+		0, /* tx_ok */
+		0, /* total_polls */
+		0, /* zero_polls */
+		0, /* tx_nospc_ldb_hw_credits */
+		0, /* tx_nospc_dir_hw_credits */
+		0, /* tx_nospc_inflight_max */
+		0, /* tx_nospc_new_event_limit */
+		0, /* tx_nospc_inflight_credits */
+		0, /* nb_events_limit */
+		0, /* inflight_events */
+		0, /* ldb_pool_size */
+		0, /* dir_pool_size */
+	};
+	static const char * const port_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"tx_new",
+		"tx_fwd",
+		"tx_rel",
+		"tx_implicit_rel",
+		"tx_sched_ordered",
+		"tx_sched_unordered",
+		"tx_sched_atomic",
+		"tx_sched_directed",
+		"tx_invalid",
+		"outstanding_releases",
+		"max_outstanding_releases",
+		"rx_sched_ordered",
+		"rx_sched_unordered",
+		"rx_sched_atomic",
+		"rx_sched_directed",
+		"rx_sched_invalid"
+	};
+	static const enum dlb2_xstats_type port_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		tx_new,
+		tx_fwd,
+		tx_rel,
+		tx_implicit_rel,
+		tx_sched_ordered,
+		tx_sched_unordered,
+		tx_sched_atomic,
+		tx_sched_directed,
+		tx_invalid,
+		outstanding_releases,
+		max_outstanding_releases,
+		rx_sched_ordered,
+		rx_sched_unordered,
+		rx_sched_atomic,
+		rx_sched_directed,
+		rx_sched_invalid
+	};
+	static const uint8_t port_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		1, /* rx_ok */
+		1, /* rx_drop */
+		1, /* rx_interrupt_wait */
+		1, /* rx_umonitor_umwait */
+		1, /* tx_ok */
+		1, /* total_polls */
+		1, /* zero_polls */
+		1, /* tx_nospc_ldb_hw_credits */
+		1, /* tx_nospc_dir_hw_credits */
+		1, /* tx_nospc_inflight_max */
+		1, /* tx_nospc_new_event_limit */
+		1, /* tx_nospc_inflight_credits */
+		1, /* tx_new */
+		1, /* tx_fwd */
+		1, /* tx_rel */
+		1, /* tx_implicit_rel */
+		1, /* tx_sched_ordered */
+		1, /* tx_sched_unordered */
+		1, /* tx_sched_atomic */
+		1, /* tx_sched_directed */
+		1, /* tx_invalid */
+		0, /* outstanding_releases */
+		0, /* max_outstanding_releases */
+		1, /* rx_sched_ordered */
+		1, /* rx_sched_unordered */
+		1, /* rx_sched_atomic */
+		1, /* rx_sched_directed */
+		1  /* rx_sched_invalid */
+	};
+
+	/* QID specific stats */
+	static const char * const qid_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"num_links",
+		"sched_type",
+		"enq_ok",
+		"current_depth",
+		"depth_threshold",
+		"depth_le50_threshold",
+		"depth_gt50_le75_threshold",
+		"depth_gt75_le100_threshold",
+		"depth_gt100_threshold",
+	};
+	static const enum dlb2_xstats_type qid_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		num_links,
+		sched_type,
+		enq_ok,
+		current_depth,
+		depth_threshold,
+		depth_le50_threshold,
+		depth_gt50_le75_threshold,
+		depth_gt75_le100_threshold,
+		depth_gt100_threshold,
+	};
+	static const uint8_t qid_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		0, /* num_links */
+		0, /* sched_type */
+		1, /* enq_ok */
+		0, /* current_depth */
+		0, /* depth_threshold */
+		1, /* depth_le50_threshold */
+		1, /* depth_gt50_le75_threshold */
+		1, /* depth_gt75_le100_threshold */
+		1, /* depth_gt100_threshold */
+	};
+
+	/* ---- end of stat definitions ---- */
+
+	/* check sizes, since a missed comma can lead to strings being
+	 * joined by the compiler.
+	 */
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
+
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_reset_allowed));
+
+	/* other vars */
+	const unsigned int count = RTE_DIM(dev_stats) +
+			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
+			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
+	unsigned int i, port, qid, stat_id = 0;
+
+	dlb2->xstats = rte_zmalloc_socket(NULL,
+			sizeof(dlb2->xstats[0]) * count, 0,
+			dlb2->qm_instance.info.socket_id);
+	if (dlb2->xstats == NULL)
+		return -ENOMEM;
+
+#define sname dlb2->xstats[stat_id].name.name
+	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
+		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
+			.fn_id = DLB2_XSTATS_FN_DEV,
+			.stat = dev_types[i],
+			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
+			.reset_allowed = dev_reset_allowed[i],
+		};
+		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
+	}
+	dlb2->xstats_count_mode_dev = stat_id;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
+		dlb2->xstats_offset_for_port[port] = stat_id;
+
+		uint32_t count_offset = stat_id;
+
+		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_PORT,
+				.obj_idx = port,
+				.stat = port_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_PORT,
+				.reset_allowed = port_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "port_%u_%s",
+				 port, port_stats[i]);
+		}
+
+		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_port = stat_id - dlb2->xstats_count_mode_dev;
+
+	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
+		uint32_t count_offset = stat_id;
+
+		dlb2->xstats_offset_for_qid[qid] = stat_id;
+
+		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_QUEUE,
+				.obj_idx = qid,
+				.stat = qid_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
+				.reset_allowed = qid_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "qid_%u_%s",
+				 qid, qid_stats[i]);
+		}
+
+		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_queue = stat_id -
+		(dlb2->xstats_count_mode_dev + dlb2->xstats_count_mode_port);
+#undef sname
+
+	dlb2->xstats_count = stat_id;
+
+	return 0;
+}
+
+void
+dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
+{
+	rte_free(dlb2->xstats);
+	dlb2->xstats_count = 0;
+}
+
+int
+dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstats_names,
+		unsigned int *ids, unsigned int size)
+{
+	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+	uint32_t start_offset = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			break;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			break;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
+		break;
+	default:
+		return -EINVAL;
+	};
+
+	if (xstats_mode_count > size || ids == NULL || xstats_names == NULL)
+		return xstats_mode_count;
+
+	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
+		if (dlb2->xstats[i].mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != dlb2->xstats[i].obj_idx)
+			continue;
+
+		xstats_names[xidx] = dlb2->xstats[i].name;
+		if (ids)
+			ids[xidx] = start_offset + xidx;
+		xidx++;
+	}
+	return xidx;
+}
+
+static int
+dlb2_xstats_update(struct dlb2_eventdev *dlb2,
+		enum rte_event_dev_xstats_mode mode,
+		uint8_t queue_port_id, const unsigned int ids[],
+		uint64_t values[], unsigned int n, const uint32_t reset)
+{
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			goto invalid_value;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			goto invalid_value;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		break;
+	default:
+		goto invalid_value;
+	};
+
+	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
+		dlb2_xstats_fn fn;
+
+		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != xs->obj_idx)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			goto invalid_value;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+
+		if (values)
+			values[xidx] = val;
+
+		if (xs->reset_allowed && reset)
+			xs->reset_value += val;
+
+		xidx++;
+	}
+
+	return xidx;
+
+invalid_value:
+	return -EINVAL;
+}
+
+int
+dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	const uint32_t reset = 0;
+
+	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
+				  reset);
+}
+
+uint64_t
+dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+				 const char *name, unsigned int *id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	dlb2_xstats_fn fn;
+
+	for (i = 0; i < dlb2->xstats_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (strncmp(xs->name.name, name,
+			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
+			if (id != NULL)
+				*id = i;
+
+			switch (xs->fn_id) {
+			case DLB2_XSTATS_FN_DEV:
+				fn = get_dev_stat;
+				break;
+			case DLB2_XSTATS_FN_PORT:
+				fn = get_port_stat;
+				break;
+			case DLB2_XSTATS_FN_QUEUE:
+				fn = get_queue_stat;
+				break;
+			default:
+				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
+					  xs->fn_id);
+				return (uint64_t)-1;
+			}
+
+			return fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+		}
+	}
+	if (id != NULL)
+		*id = (uint32_t)-1;
+	return (uint64_t)-1;
+}
+
+static void
+dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
+			uint32_t num)
+{
+	uint32_t i;
+	dlb2_xstats_fn fn;
+
+	for (i = start; i < start + num; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (!xs->reset_allowed)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			return;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
+		xs->reset_value = val;
+	}
+}
+
+static int
+dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
+			const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_QUEUE,
+					queue_id, ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	if (ids == NULL)
+		dlb2_xstats_reset_range(dlb2,
+			dlb2->xstats_offset_for_qid[queue_id],
+			dlb2->xstats_count_per_qid[queue_id]);
+
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
+		       const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+	int offset = dlb2->xstats_offset_for_port[port_id];
+	int nb_stat = dlb2->xstats_count_per_port[port_id];
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_PORT, port_id,
+					ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
+		      uint32_t nb_ids)
+{
+	uint32_t i;
+
+	if (ids) {
+		for (i = 0; i < nb_ids; i++) {
+			uint32_t id = ids[i];
+
+			if (id >= dlb2->xstats_count_mode_dev)
+				return -EINVAL;
+			dlb2_xstats_reset_range(dlb2, id, 1);
+		}
+	} else {
+		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
+			dlb2_xstats_reset_range(dlb2, i, 1);
+	}
+
+	return 0;
+}
+
+int
+dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+			   enum rte_event_dev_xstats_mode mode,
+			   int16_t queue_port_id,
+			   const uint32_t ids[],
+			   uint32_t nb_ids)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	uint32_t i;
+
+	/* handle -1 for queue_port_id here, looping over all ports/queues */
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
+			return -EINVAL;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+				if (dlb2_xstats_reset_port(dlb2, i,
+							   ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
+			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
+						   ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
+				if (dlb2_xstats_reset_queue(dlb2, i,
+							    ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
+			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
+						    ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	};
+
+	return 0;
+}
+
+void
+dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	int i;
+
+	if (f == NULL) {
+		DLB2_LOG_ERR("Invalid file pointer\n");
+		return;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (dlb2 == NULL) {
+		fprintf(f, "DLB2 Event device cannot be dumped!\n");
+		return;
+	}
+
+	if (!dlb2->configured)
+		fprintf(f, "DLB2 Event device is not configured\n");
+
+	handle = &dlb2->qm_instance;
+
+	fprintf(f, "================\n");
+	fprintf(f, "DLB2 Device Dump\n");
+	fprintf(f, "================\n");
+
+	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
+		dlb2->umwait_allowed ? "yes" : "no");
+
+	/* Generic top level device information */
+
+	fprintf(f, "device is configured and run state =");
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		fprintf(f, "STOPPED\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
+		fprintf(f, "STOPPING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
+		fprintf(f, "STARTING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
+		fprintf(f, "STARTED\n");
+	else
+		fprintf(f, "UNEXPECTED\n");
+
+	fprintf(f, "domain ID=%u, socket_id=%u, evdev=%p\n",
+		handle->domain_id, handle->info.socket_id, dlb2->event_dev);
+
+	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
+		dlb2->num_dir_ports, dlb2->num_dir_queues);
+
+	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
+		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
+
+	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
+		handle->cfg.resources.num_atomic_inflights,
+		handle->cfg.resources.num_hist_list_entries);
+
+	fprintf(f, "results from most recent hw resource query:\n");
+
+	fprintf(f, "\tnum_sched_domains = %u\n",
+		dlb2->hw_rsrc_query_results.num_sched_domains);
+
+	fprintf(f, "\tnum_ldb_queues = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_queues);
+
+	fprintf(f, "\tnum_ldb_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_ports);
+
+	fprintf(f, "\tnum_dir_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_ports);
+
+	fprintf(f, "\tnum_atomic_inflights = %u\n",
+		dlb2->hw_rsrc_query_results.num_atomic_inflights);
+
+	fprintf(f, "\tnum_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.num_hist_list_entries);
+
+	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
+
+	fprintf(f, "\tnum_ldb_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_credits);
+
+	fprintf(f, "\tnum_dir_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_credits);
+
+	/* Port level information */
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
+		int j;
+
+		if (!p->enq_configured)
+			fprintf(f, "Port_%d is not configured\n", i);
+
+		fprintf(f, "Port_%d\n", i);
+		fprintf(f, "=======\n");
+
+		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
+			p->id, p->setup_done);
+
+		fprintf(f, "\tconfig state=%d, port state=%d\n",
+			p->qm_port.config_state, p->qm_port.state);
+
+		fprintf(f, "\tport is %s\n",
+			p->qm_port.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\toutstanding releases=%u\n",
+			p->outstanding_releases);
+
+		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
+			p->inflight_max, p->inflight_credits);
+
+		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
+			p->credit_update_quanta, p->implicit_release);
+
+		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (p->link[j].valid)
+				fprintf(f, "id=%u prio=%u ",
+					p->link[j].queue_id,
+					p->link[j].priority);
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
+
+		fprintf(f, "\tcached_ldb_credits=%u\n",
+			p->qm_port.cached_ldb_credits);
+
+		fprintf(f, "\tldb_credits = %u\n",
+			p->qm_port.ldb_credits);
+
+		fprintf(f, "\tcached_dir_credits = %u\n",
+			p->qm_port.cached_dir_credits);
+
+		fprintf(f, "\tdir_credits = %u\n",
+			p->qm_port.dir_credits);
+
+		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
+			p->qm_port.gen_bit,
+			p->qm_port.cq_idx,
+			p->qm_port.cq_depth);
+
+		fprintf(f, "\tinterrupt armed=%d\n",
+			p->qm_port.int_armed);
+
+		fprintf(f, "\tPort statistics\n");
+
+		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
+			p->stats.traffic.rx_ok);
+
+		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
+			p->stats.traffic.rx_drop);
+
+		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
+			p->stats.traffic.rx_interrupt_wait);
+
+		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
+			p->stats.traffic.rx_umonitor_umwait);
+
+		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
+			p->stats.traffic.tx_ok);
+
+		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
+			p->stats.traffic.total_polls);
+
+		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
+			p->stats.traffic.zero_polls);
+
+		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_ldb_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_dir_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_max);
+
+		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_new_event_limit);
+
+		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_credits);
+
+		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
+
+		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
+
+		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
+
+		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
+			p->stats.tx_implicit_rel);
+
+		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
+			p->stats.tx_invalid);
+
+		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
+			p->stats.rx_sched_invalid);
+	}
+
+	/* Queue level information */
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
+		int j, k;
+
+		if (!q->setup_done)
+			fprintf(f, "Queue_%d is not configured\n", i);
+
+		fprintf(f, "Queue_%d\n", i);
+		fprintf(f, "========\n");
+
+		fprintf(f, "\tevqueue_%u is set up\n", q->id);
+
+		fprintf(f, "\tqueue is %s\n",
+			q->qm_queue.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
+
+		for (j = 0; j < dlb2->num_ports; j++) {
+			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
+
+			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++) {
+				if (p->link[k].valid &&
+				    p->link[k].queue_id == q->id)
+					fprintf(f, "id=%u prio=%u ",
+						p->id, p->link[k].priority);
+			}
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\tcurrent depth: %u events\n",
+			dlb2_get_queue_depth(dlb2, q));
+
+		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
+			q->qm_queue.num_qid_inflights, q->qm_queue.sched_type);
+	}
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index d7390ec..a80d7ab 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files('dlb2.c',
 		'dlb2_iface.c',
+		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c'
-- 
2.6.4

[dpdk-dev] [PATCH v10 10/23] event/dlb2: add infos get and configure

[Timothy McDaniel]

Add support for configuring the DLB2 hardware.
In particular, this patch configures the DLB2
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  116 +
 drivers/event/dlb2/dlb2.c                  |  313 +++
 drivers/event/dlb2/dlb2_iface.c            |    5 +
 drivers/event/dlb2/dlb2_iface.h            |    4 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 3237 ++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |   15 +
 drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
 7 files changed, 3734 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index e3091c9..5f6c486 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -32,9 +32,125 @@ detailed understanding of the hardware, but these details are important when
 writing high-performance code. This section describes the places where the
 eventdev API and DLB2 misalign.
 
+Scheduling Domain Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are 32 scheduling domainis the DLB2.
+When one is configured, it allocates load-balanced and
+directed queues, ports, credits, and other hardware resources. Some
+resource allocations are user-controlled -- the number of queues, for example
+-- and others, like credit pools (one directed and one load-balanced pool per
+scheduling domain), are not.
+
+The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
+setup argument and the per-port ``new_event_threshold`` argument apply as
+defined in the eventdev header file. The limit is applied to all enqueues,
+regardless of whether it will consume a directed or load-balanced credit.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Reconfiguration
+~~~~~~~~~~~~~~~
+
+The Eventdev API allows one to reconfigure a device, its ports, and its queues
+by first stopping the device, calling the configuration function(s), then
+restarting the device. The DLB2 does not support configuring an individual queue
+or port without first reconfiguring the entire device, however, so there are
+certain reconfiguration sequences that are valid in the eventdev API but not
+supported by the PMD.
+
+Specifically, the PMD supports the following configuration sequence:
+1. Configure and start the device
+2. Stop the device
+3. (Optional) Reconfigure the device
+4. (Optional) If step 3 is run:
+
+   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
+   b. The reconfigured port(s) lose their previous queue links.
+
+5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
+6. Restart the device. If the device is reconfigured in step 3 but one or more
+   of its ports or queues are not, the PMD will apply their previous
+   configuration (including port->queue links) at this time.
+
+The PMD does not support the following configuration sequences:
+1. Configure and start the device
+2. Stop the device
+3. Setup queue or setup port
+4. Start the device
+
+This sequence is not supported because the event device must be reconfigured
+before its ports or queues can be.
+
+Atomic Inflights Allocation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
+inflight event in a temporary buffer that is divided among load-balanced
+queues. If a queue's atomic buffer storage fills up, this can result in
+head-of-line-blocking. For example:
+
+- An LDB queue allocated N atomic buffer entries
+- All N entries are filled with events from flow X, which is pinned to CQ 0.
+
+Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
+scheduled. The likelihood of this case depends on the eventdev configuration,
+traffic behavior, event processing latency, potential for a worker to be
+interrupted or otherwise delayed, etc.
+
+By default, the PMD allocates 16 buffer entries for each load-balanced queue,
+which provides an even division across all 128 queues but potentially wastes
+buffer space (e.g. if not all queues are used, or aren't used for atomic
+scheduling).
+
+The PMD provides a dev arg to override the default per-queue allocation. To
+increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,atm_inflights=64
+
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 925824e..6798e66 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -85,6 +85,25 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
 	return 0;
 }
 
+static void
+dlb2_free_qe_mem(struct dlb2_port *qm_port)
+{
+	if (qm_port == NULL)
+		return;
+
+	rte_free(qm_port->qe4);
+	qm_port->qe4 = NULL;
+
+	rte_free(qm_port->int_arm_qe);
+	qm_port->int_arm_qe = NULL;
+
+	rte_free(qm_port->consume_qe);
+	qm_port->consume_qe = NULL;
+
+	rte_memzone_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz);
+	dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz = NULL;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -350,10 +369,304 @@ set_qid_depth_thresh(const char *key __rte_unused,
 }
 
 static void
+dlb2_eventdev_info_get(struct rte_eventdev *dev,
+		       struct rte_event_dev_info *dev_info)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret;
+
+	ret = dlb2_hw_query_resources(dlb2);
+	if (ret) {
+		const struct rte_eventdev_data *data = dev->data;
+
+		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+			     ret, data->dev_id);
+		/* fn is void, so fall through and return values set up in
+		 * probe
+		 */
+	}
+
+	/* Add num resources currently owned by this domain.
+	 * These would become available if the scheduling domain were reset due
+	 * to the application recalling eventdev_configure to *reconfigure* the
+	 * domain.
+	 */
+	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
+	evdev_dlb2_default_info.max_event_queues += dlb2->num_ldb_queues;
+	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
+
+	evdev_dlb2_default_info.max_event_queues =
+		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
+			RTE_EVENT_MAX_QUEUES_PER_DEV);
+
+	evdev_dlb2_default_info.max_num_events =
+		RTE_MIN(evdev_dlb2_default_info.max_num_events,
+			dlb2->max_num_events_override);
+
+	*dev_info = evdev_dlb2_default_info;
+}
+
+static int
+dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
+			    const struct dlb2_hw_rsrcs *resources_asked)
+{
+	int ret = 0;
+	struct dlb2_create_sched_domain_args *cfg;
+
+	if (resources_asked == NULL) {
+		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
+		ret = EINVAL;
+		goto error_exit;
+	}
+
+	/* Map generic qm resources to dlb2 resources */
+	cfg = &handle->cfg.resources;
+
+	/* DIR ports and queues */
+
+	cfg->num_dir_ports = resources_asked->num_dir_ports;
+
+	cfg->num_dir_credits = resources_asked->num_dir_credits;
+
+	/* LDB queues */
+
+	cfg->num_ldb_queues = resources_asked->num_ldb_queues;
+
+	/* LDB ports */
+
+	cfg->cos_strict = 0; /* Best effort */
+	cfg->num_cos_ldb_ports[0] = 0;
+	cfg->num_cos_ldb_ports[1] = 0;
+	cfg->num_cos_ldb_ports[2] = 0;
+	cfg->num_cos_ldb_ports[3] = 0;
+
+	switch (handle->cos_id) {
+	case DLB2_COS_0:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[0] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_1:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[1] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_2:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[2] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_3:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[3] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_DEFAULT:
+		/* all ldb ports are don't care ports from a cos perspective */
+		cfg->num_ldb_ports =
+			resources_asked->num_ldb_ports;
+		break;
+	}
+
+	cfg->num_ldb_credits =
+		resources_asked->num_ldb_credits;
+
+	cfg->num_atomic_inflights =
+		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
+		cfg->num_ldb_queues;
+
+	cfg->num_hist_list_entries = resources_asked->num_ldb_ports *
+		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d\n",
+		     cfg->num_ldb_queues,
+		     resources_asked->num_ldb_ports,
+		     cfg->num_dir_ports,
+		     cfg->num_atomic_inflights,
+		     cfg->num_hist_list_entries,
+		     cfg->num_ldb_credits,
+		     cfg->num_dir_credits);
+
+	/* Configure the QM */
+
+	ret = dlb2_iface_sched_domain_create(handle, cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: domain create failed, ret = %d, extra status: %s\n",
+			     ret,
+			     dlb2_error_strings[cfg->response.status]);
+
+		goto error_exit;
+	}
+
+	handle->domain_id = cfg->response.id;
+	handle->cfg.configured = true;
+
+error_exit:
+
+	return ret;
+}
+
+static void
+dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	enum dlb2_configuration_state config_state;
+	int i, j;
+
+	dlb2_iface_domain_reset(dlb2);
+
+	/* Free all dynamically allocated port memory */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
+
+	/* If reconfiguring, mark the device's queues and ports as "previously
+	 * configured." If the user doesn't reconfigure them, the PMD will
+	 * reapply their previous configuration when the device is started.
+	 */
+	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
+		DLB2_NOT_CONFIGURED;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		dlb2->ev_ports[i].qm_port.config_state = config_state;
+		/* Reset setup_done so ports can be reconfigured */
+		dlb2->ev_ports[i].setup_done = false;
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			dlb2->ev_ports[i].link[j].mapped = false;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++)
+		dlb2->ev_queues[i].qm_queue.config_state = config_state;
+
+	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
+		dlb2->ev_queues[i].setup_done = false;
+
+	dlb2->num_ports = 0;
+	dlb2->num_ldb_ports = 0;
+	dlb2->num_dir_ports = 0;
+	dlb2->num_queues = 0;
+	dlb2->num_ldb_queues = 0;
+	dlb2->num_dir_queues = 0;
+	dlb2->configured = false;
+}
+
+/* Note: 1 QM instance per QM device, QM instance/device == event device */
+static int
+dlb2_eventdev_configure(const struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
+	const struct rte_eventdev_data *data = dev->data;
+	const struct rte_event_dev_config *config = &data->dev_conf;
+	int ret;
+
+	/* If this eventdev is already configured, we must release the current
+	 * scheduling domain before attempting to configure a new one.
+	 */
+	if (dlb2->configured) {
+		dlb2_hw_reset_sched_domain(dev, true);
+
+		ret = dlb2_hw_query_resources(dlb2);
+		if (ret) {
+			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+				     ret, data->dev_id);
+			return ret;
+		}
+	}
+
+	if (config->nb_event_queues > rsrcs->num_queues) {
+		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_queues,
+			     rsrcs->num_queues);
+		return -EINVAL;
+	}
+	if (config->nb_event_ports > (rsrcs->num_ldb_ports
+			+ rsrcs->num_dir_ports)) {
+		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_ports,
+			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
+		return -EINVAL;
+	}
+	if (config->nb_events_limit > rsrcs->nb_events_limit) {
+		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_events_limit,
+			     rsrcs->nb_events_limit);
+		return -EINVAL;
+	}
+
+	if (config->event_dev_cfg & RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT)
+		dlb2->global_dequeue_wait = false;
+	else {
+		uint32_t timeout32;
+
+		dlb2->global_dequeue_wait = true;
+
+		/* note size mismatch of timeout vals in eventdev lib. */
+		timeout32 = config->dequeue_timeout_ns;
+
+		dlb2->global_dequeue_wait_ticks =
+			timeout32 * (rte_get_timer_hz() / 1E9);
+	}
+
+	/* Does this platform support umonitor/umwait? */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
+		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
+			DLB2_LOG_ERR("invalid value (%d) for RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
+				     RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
+			return -EINVAL;
+		}
+		dlb2->umwait_allowed = true;
+	}
+
+	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
+	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs->num_dir_ports;
+	/* 1 dir queue per dir port */
+	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports;
+
+	/* Scale down nb_events_limit by 4 for directed credits, since there
+	 * are 4x as many load-balanced credits.
+	 */
+	rsrcs->num_ldb_credits = 0;
+	rsrcs->num_dir_credits = 0;
+
+	if (rsrcs->num_ldb_queues)
+		rsrcs->num_ldb_credits = config->nb_events_limit;
+	if (rsrcs->num_dir_ports)
+		rsrcs->num_dir_credits = config->nb_events_limit / 4;
+	if (dlb2->num_dir_credits_override != -1)
+		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
+
+	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
+		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
+		return -ENODEV;
+	}
+
+	dlb2->new_event_limit = config->nb_events_limit;
+	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
+
+	/* Save number of ports/queues for this event dev */
+	dlb2->num_ports = config->nb_event_ports;
+	dlb2->num_queues = config->nb_event_queues;
+	dlb2->num_dir_ports = rsrcs->num_dir_ports;
+	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
+	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
+	dlb2->num_dir_queues = dlb2->num_dir_ports;
+	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
+	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
+	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
+	dlb2->max_dir_credits = rsrcs->num_dir_credits;
+
+	dlb2->configured = true;
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dev_infos_get    = dlb2_eventdev_info_get,
+		.dev_configure    = dlb2_eventdev_configure,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 0fc9991..a829b9b 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -26,3 +26,8 @@ int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
+
+int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				struct dlb2_create_sched_domain_args *args);
+
+void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4fb416e..6663dab 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -26,4 +26,8 @@ extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
 
+extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				 struct dlb2_create_sched_domain_args *args);
+
+extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 6de8b95..63a68a6 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -12,6 +12,27 @@
 #include "dlb2_regs.h"
 #include "dlb2_resource.h"
 
+#include "../../dlb2_priv.h"
+#include "../../dlb2_inline_fns.h"
+
+#define DLB2_DOM_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, domain_list, iter)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, func_list, iter)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, domain_list, it, it_tmp)
+
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, func_list, it, it_tmp)
+
 static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
 {
 	int i;
@@ -272,3 +293,3219 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
 
 	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
 }
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+	union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+	r0.field.count = domain->num_ldb_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+	r1.field.count = domain->num_dir_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+		       struct dlb2_function_resources *rsrcs,
+		       u32 domain_id,
+		       u32 cos_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	RTE_SET_USED(iter);
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+				   typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				   struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_ports,
+				   u32 cos_id,
+				   struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_ldb_port *port;
+
+		port = dlb2_get_next_ldb_port(hw, rsrcs,
+					      domain->id.phys_id, cos_id);
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_ports[cos_id],
+			      &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_ports[cos_id],
+			      &port->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_create_sched_domain_args *args,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      num,
+						      i,
+						      resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+					ret = __dlb2_attach_ldb_ports(hw,
+								      rsrcs,
+								      domain,
+								      1,
+								      cos_id,
+								      resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret < 0)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      1,
+						      j,
+						      resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 u32 num_ports,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+					   typeof(*port));
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_dir_pq_pairs, &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_dir_pq_pairs, &port->domain_list);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+					struct dlb2_hw_domain *domain,
+					u32 num_atomic_inflights,
+					struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+				     struct dlb2_hw_domain *domain,
+				     u32 num_hist_list_entries,
+				     struct dlb2_cmd_response *resp)
+{
+	struct dlb2_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb2_bitmap_find_set_bit_range(bitmap,
+						      num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+				  struct dlb2_function_resources *rsrcs,
+				  struct dlb2_hw_domain *domain,
+				  u32 num_queues,
+				  struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb2_ldb_queue *queue;
+
+		queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+					    typeof(*queue));
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_queues, &queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_queues, &queue->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+			     struct dlb2_function_resources *rsrcs,
+			     struct dlb2_hw_domain *domain,
+			     struct dlb2_create_sched_domain_args *args,
+			     struct dlb2_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb2_attach_ldb_queues(hw,
+				     rsrcs,
+				     domain,
+				     args->num_ldb_queues,
+				     resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_ports(hw,
+				    rsrcs,
+				    domain,
+				    args,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_ports(hw,
+				    rsrcs,
+				    domain,
+				    args->num_dir_ports,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_credits(rsrcs,
+				      domain,
+				      args->num_ldb_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_credits(rsrcs,
+				      domain,
+				      args->num_dir_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+						   domain,
+						   args->num_hist_list_entries,
+						   resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_atomic_inflights(rsrcs,
+					   domain,
+					   args->num_atomic_inflights,
+					   resp);
+	if (ret < 0)
+		return ret;
+
+	dlb2_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+				  struct dlb2_create_sched_domain_args *args,
+				  struct dlb2_cmd_response *resp)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb2_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+				  struct dlb2_create_sched_domain_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		    args->num_ldb_queues);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		    args->num_ldb_ports);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		    args->num_cos_ldb_ports[0]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		    args->cos_strict);
+	DLB2_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		    args->num_dir_ports);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		    args->num_atomic_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		    args->num_hist_list_entries);
+	DLB2_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		    args->num_ldb_credits);
+	DLB2_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		    args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ *	domain and its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+	if (ret)
+		return ret;
+
+	domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available domains\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (domain->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_domains contains configured domains.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	dlb2_init_domain_rsrc_lists(domain);
+
+	ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to verify args.\n",
+			    __func__);
+
+		return ret;
+	}
+
+	dlb2_list_del(&rsrcs->avail_domains, &domain->func_list);
+
+	dlb2_list_add(&rsrcs->used_domains, &domain->func_list);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+	DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+			     struct dlb2_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb2_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	int ret;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_disable(hw, port);
+
+		ret = dlb2_drain_dir_cq(hw, port);
+		if (ret < 0)
+			return ret;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_dir_pq_pair *queue)
+{
+	union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *queue)
+{
+	return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+				      struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		dlb2_movdir64b(pp_addr, hcw);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int ret, i;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			if (toggle_port)
+				dlb2_ldb_port_cq_disable(hw, port);
+
+			ret = dlb2_drain_ldb_cq(hw, port);
+			if (ret < 0)
+				return ret;
+
+			if (toggle_port)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+
+	return 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_atm_active r1;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_ldb_queue *queue)
+{
+	return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb2_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+					   struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to unmap domain queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = true;
+
+			dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+			   u32 id,
+			   bool vdev_req,
+			   unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iter1) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter2)
+			if (queue->id.virt_id == id)
+				return queue;
+	}
+
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue, iter1)
+		if (queue->id.virt_id == id)
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+						      u32 id,
+						      bool vdev_req,
+						      unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iteration;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	RTE_SET_USED(iteration);
+
+	if (id >= DLB2_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iteration)
+		if (domain->id.virt_id == id)
+			return domain;
+
+	return NULL;
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot,
+					   enum dlb2_qid_map_state new_state)
+{
+	enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb2_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB2_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB2_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB2_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB2_HW_DBG(hw,
+		    "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB2_HW_ERR(hw,
+		    "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+				enum dlb2_qid_map_state state,
+				int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+				      enum dlb2_qid_map_state state,
+				      struct dlb2_ldb_queue *queue,
+				      int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+					      struct dlb2_hw_domain *domain,
+					      struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain,
+					     struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+						struct dlb2_ldb_port *port,
+						int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 0;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 1;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+					struct dlb2_ldb_port *p,
+					struct dlb2_ldb_queue *q,
+					u8 priority)
+{
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_lsp_cq2qid0 r1;
+	union dlb2_atm_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix_00 r3;
+	union dlb2_lsp_qid2cqidix2_00 r4;
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+	    !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+	r0.field.v |= 1 << i;
+	r0.field.prio |= (priority & 0x7) << i * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+	else
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		r1.field.qid_p0 = q->id.phys_id;
+	if (i == 1 || i == 5)
+		r1.field.qid_p1 = q->id.phys_id;
+	if (i == 2 || i == 6)
+		r1.field.qid_p2 = q->id.phys_id;
+	if (i == 3 || i == 7)
+		r1.field.qid_p3 = q->id.phys_id;
+
+	if (i < 4)
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+	else
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r4.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+						  p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		r2.field.cq_p0 |= 1 << i;
+		r3.field.cq_p0 |= 1 << i;
+		r4.field.cq_p0 |= 1 << i;
+		break;
+
+	case 1:
+		r2.field.cq_p1 |= 1 << i;
+		r3.field.cq_p1 |= 1 << i;
+		r4.field.cq_p1 |= 1 << i;
+		break;
+
+	case 2:
+		r2.field.cq_p2 |= 1 << i;
+		r3.field.cq_p2 |= 1 << i;
+		r4.field.cq_p2 |= 1 << i;
+		break;
+
+	case 3:
+		r2.field.cq_p3 |= 1 << i;
+		r3.field.cq_p3 |= 1 << i;
+		r4.field.cq_p3 |= 1 << i;
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r3.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		    r4.val);
+
+	dlb2_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB2_QUEUE_MAPPED;
+
+	return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	/* Set the atomic scheduling haswork bit */
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.rlist_haswork_v = r0.field.count > 0;
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      u8 slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.rlist_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.nalb_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+					      struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+	r0.field.limit = queue->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+						struct dlb2_ldb_queue *queue)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+						struct dlb2_hw_domain *domain,
+						struct dlb2_ldb_port *port,
+						struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_lsp_qid_ldb_infl_cnt r0;
+	enum dlb2_qid_map_state state;
+	int slot, ret, i;
+	u8 prio;
+	RTE_SET_USED(iter);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: non-zero QID inflight count\n",
+			    __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			state = DLB2_QUEUE_MAPPED;
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+					 struct dlb2_ldb_port *port,
+					 struct dlb2_ldb_queue *queue,
+					 u8 priority)
+{
+	union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	int domain_id, slot, ret;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+		DLB2_HW_ERR(hw,
+			    "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		if (port->enabled)
+			dlb2_ldb_port_cq_enable(hw, port);
+
+		dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain,
+					struct dlb2_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		union dlb2_lsp_qid_ldb_infl_cnt r0;
+		struct dlb2_ldb_queue *queue;
+		int qid;
+
+		if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: unable to find queue %d\n",
+				    __func__, qid);
+			continue;
+		}
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count)
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb2_ldb_port_cq_disable(hw, port);
+
+		dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count) {
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+
+			dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port,
+				   struct dlb2_ldb_queue *queue)
+{
+	enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_atm_qid2cqidix_00 r1;
+	union dlb2_lsp_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix2_00 r3;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB2_QUEUE_MAPPED;
+	in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: QID %d isn't mapped\n",
+			    __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+	r0.field.v &= ~(1 << i);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		r1.field.cq_p0 &= ~(1 << i);
+		r2.field.cq_p0 &= ~(1 << i);
+		r3.field.cq_p0 &= ~(1 << i);
+		break;
+
+	case 1:
+		r1.field.cq_p1 &= ~(1 << i);
+		r2.field.cq_p1 &= ~(1 << i);
+		r3.field.cq_p1 &= ~(1 << i);
+		break;
+
+	case 2:
+		r1.field.cq_p2 &= ~(1 << i);
+		r2.field.cq_p2 &= ~(1 << i);
+		r3.field.cq_p2 &= ~(1 << i);
+		break;
+
+	case 3:
+		r1.field.cq_p3 &= ~(1 << i);
+		r2.field.cq_p3 &= ~(1 << i);
+		r3.field.cq_p3 &= ~(1 << i);
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+		    r1.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+		    r3.val);
+
+	dlb2_flush_csr(hw);
+
+	unmapped = DLB2_QUEUE_UNMAPPED;
+
+	return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_ldb_port *port,
+				 struct dlb2_ldb_queue *queue,
+				 u8 prio)
+{
+	if (domain->started)
+		return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   int slot)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb2_ldb_port_qid_map *map;
+		struct dlb2_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain,
+					  struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (r0.field.count > 0)
+		return false;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = false;
+
+			dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_dir_vpp_v r1;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		unsigned int offs;
+		u32 virt_id;
+
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+	}
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_ldb_vpp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			unsigned int offs;
+			u32 virt_id;
+
+			if (hw->virt_mode == DLB2_VIRT_SRIOV)
+				virt_id = port->id.virt_id;
+			else
+				virt_id = port->id.phys_id;
+
+			offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+			DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+	union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+				    r0.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+				    r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+	union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+			    r0.val);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+			    r1.val);
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+		union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+		union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+				    r1.val);
+
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID_V(idx),
+				    r2.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID2QID(idx),
+				    r3.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID_V(idx),
+				    r1.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID2QID(idx),
+				    r2.val);
+		}
+	}
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+					       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_sn_chk_enbl r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.en = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			int i;
+
+			for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+				if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					    __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		port->enabled = false;
+
+		dlb2_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	union dlb2_sys_dir_pp_v r1;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+			    r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_ldb_pp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_ldb_queue *queue;
+	int i;
+	RTE_SET_USED(iter);
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty ldb queue %d\n",
+				    __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port, iter) {
+			if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb2_ldb_cq_token_count(hw, ldb_port)) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to empty ldb port %d\n",
+					    __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir queue %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb2_dir_cq_token_count(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir port %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						   struct dlb2_ldb_port *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP2PP(offs),
+			    DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP_V(offs),
+			    DLB2_SYS_VF_LDB_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    DLB2_SYS_LDB_PP_V_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_DSBL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_BASE_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID0(port->id.phys_id),
+		    DLB2_LSP_CQ2QID0_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID1(port->id.phys_id),
+		    DLB2_LSP_CQ2QID1_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+		    DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			__dlb2_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+				       struct dlb2_dir_pq_pair *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_DSBL_RST);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_FMT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP2PP(offs),
+			    DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP_V(offs),
+			    DLB2_SYS_VF_DIR_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		__dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+			    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_ITS(queue_id),
+			    DLB2_SYS_LDB_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN(queue_id),
+			    DLB2_CHP_ORD_QID_SN_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+			    DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_V(queue_id),
+			    DLB2_SYS_LDB_QID_V_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_CFG_V(queue_id),
+			    DLB2_SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB2_CSR_WR(hw,
+				    offs[queue->sn_group],
+				    DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX2(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX2_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_ATM_QID2CQIDIX(queue_id, i),
+				    DLB2_ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb2_domain_reset_dir_port_registers(hw, domain);
+
+	dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_dir_pq_pair *tmp_dir_port;
+	struct dlb2_ldb_queue *tmp_ldb_queue;
+	struct dlb2_ldb_port *tmp_ldb_port;
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_list_head *list;
+	int ret, i;
+	RTE_SET_USED(tmp_dir_port);
+	RTE_SET_USED(tmp_ldb_queue);
+	RTE_SET_USED(tmp_ldb_port);
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list = &domain->used_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb2_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		dlb2_list_del(&domain->used_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list = &domain->avail_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		ldb_queue->owned = false;
+
+		dlb2_list_del(&domain->avail_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		list = &domain->used_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB2_QUEUE_UNMAPPED;
+
+			dlb2_list_del(&domain->used_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list = &domain->avail_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			ldb_port->owned = false;
+
+			dlb2_list_del(&domain->avail_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list = &domain->used_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		dlb2_list_del(&domain->used_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list = &domain->avail_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				    domain->hist_list_entry_base,
+				    domain->total_hist_list_entries);
+	if (ret) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			    __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	dlb2_list_del(&rsrcs->used_domains, &domain->func_list);
+	dlb2_list_add(&rsrcs->avail_domains, &domain->func_list);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain,
+					    struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_ldb_port *port;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (!dlb2_list_empty(&domain->used_ldb_ports[i]))
+			break;
+	}
+
+	if (i == DLB2_NUM_COS_DOMAINS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: No configured LDB ports\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb2_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+	 * prior to this in dlb2_domain_drain_mapped_queues().
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings != 0 ||
+		    dlb2_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ *	hardware resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain  == NULL || !domain->configured)
+		return -EINVAL;
+
+	/* Disable VPPs */
+	if (vdev_req) {
+		dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+		dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+	}
+
+	/* Disable CQ interrupts */
+	dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+	dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+	dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb2_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_mapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	dlb2_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb2_domain_disable_dir_cqs(hw, domain);
+
+	/* Disable PPs */
+	dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+	ret = dlb2_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Reset the QID and port state. */
+	dlb2_domain_reset_registers(hw, domain);
+
+	/* Hardware reset complete. Reset the domain's software state */
+	ret = dlb2_domain_reset_software_state(hw, domain);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 210b3186..285ad2c 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -598,3 +598,18 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
 
 	return 0;
 }
+
+int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+			    struct dlb2_create_sched_domain_args *args,
+			    struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_sched_domain(hw, args, resp, NOT_VF_REQ,
+					   PF_ID_ZERO);
+}
+
+int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 7d64309..9310f72 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -108,15 +108,59 @@ dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
 	return 0;
 }
 
+static int
+dlb2_pf_sched_domain_create(struct dlb2_hw_dev *handle,
+			    struct dlb2_create_sched_domain_args *arg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (dlb2_dev->domain_reset_failed) {
+		response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = dlb2_pf_create_sched_domain(&dlb2_dev->hw, arg, &response);
+	if (ret)
+		goto done;
+
+done:
+
+	arg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static void
+dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret;
+
+	dlb2_dev = (struct dlb2_dev *)dlb2->qm_instance.pf_dev;
+	ret = dlb2_pf_reset_domain(&dlb2_dev->hw, dlb2->qm_instance.domain_id);
+	if (ret)
+		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
 	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
 	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_domain_reset = dlb2_pf_domain_reset;
 	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
 	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v10 11/23] event/dlb2: add queue and port default conf

[Timothy McDaniel]

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 6798e66..ef1c000 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -661,12 +661,42 @@ dlb2_eventdev_configure(const struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
+				    uint8_t port_id,
+				    struct rte_event_port_conf *port_conf)
+{
+	RTE_SET_USED(port_id);
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	port_conf->new_event_threshold = dlb2->new_event_limit;
+	port_conf->dequeue_depth = 32;
+	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
+	port_conf->event_port_cfg = 0;
+}
+
+static void
+dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
+				     uint8_t queue_id,
+				     struct rte_event_queue_conf *queue_conf)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(queue_id);
+
+	queue_conf->nb_atomic_flows = 1024;
+	queue_conf->nb_atomic_order_sequences = 64;
+	queue_conf->event_queue_cfg = 0;
+	queue_conf->priority = 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v10 12/23] event/dlb2: add queue setup

[Timothy McDaniel]

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  73 +++--
 drivers/event/dlb2/dlb2.c                  | 312 +++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |  12 +
 drivers/event/dlb2/dlb2_iface.h            |  12 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 464 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  82 +++++
 7 files changed, 926 insertions(+), 39 deletions(-)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 5f6c486..b154cac 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -47,45 +47,40 @@ setup argument and the per-port ``new_event_threshold`` argument apply as
 defined in the eventdev header file. The limit is applied to all enqueues,
 regardless of whether it will consume a directed or load-balanced credit.
 
-Load-balanced and Directed Ports
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
-does not have the same concept, but it has a similar one: ports and queues that
-are singly-linked (i.e. linked to a single queue or port, respectively).
-
-The ``rte_event_dev_info_get()`` function reports the number of available
-event ports and queues (among other things). For the DLB2 PMD, max_event_ports
-and max_event_queues report the number of available load-balanced ports and
-queues, and max_single_link_event_port_queue_pairs reports the number of
-available directed ports and queues.
-
-When a scheduling domain is created in ``rte_event_dev_configure()``, the user
-specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
-control the total number of ports (load-balanced and directed) and the number
-of directed ports. Hence, the number of requested load-balanced ports is
-``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
-specifies the total number of queues (load-balanced and directed). The number
-of directed queues comes from ``nb_single_link_event_port_queues``, since
-directed ports and queues come in pairs.
-
-When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
-whether it should be configured as a directed (the flag is set) or a
-load-balanced (the flag is unset) port. Similarly, the
-``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
-whether it is a directed or load-balanced queue.
-
-Load-balanced ports can only be linked to load-balanced queues, and directed
-ports can only be linked to directed queues. Furthermore, directed ports can
-only be linked to a single directed queue (and vice versa), and that link
-cannot change after the eventdev is started.
-
-The eventdev API does not have a directed scheduling type. To support directed
-traffic, the dlb PMD detects when an event is being sent to a directed queue
-and overrides its scheduling type. Note that the originally selected scheduling
-type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
-will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
-port.
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
 
 Flow ID
 ~~~~~~~
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index ef1c000..48e7245 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -688,6 +688,317 @@ dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
 	queue_conf->priority = 0;
 }
 
+static int32_t
+dlb2_get_sn_allocation(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_allocation_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_set_sn_allocation(struct dlb2_eventdev *dlb2, int group, int num)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_set_sn_allocation_args cfg;
+	int ret;
+
+	cfg.num = num;
+	cfg.group = group;
+
+	ret = dlb2_iface_set_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: set_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int32_t
+dlb2_get_sn_occupancy(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_occupancy_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_occupancy(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_occupancy ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+/* Query the current sequence number allocations and, if they conflict with the
+ * requested LDB queue configuration, attempt to re-allocate sequence numbers.
+ * This is best-effort; if it fails, the PMD will attempt to configure the
+ * load-balanced queue and return an error.
+ */
+static void
+dlb2_program_sn_allocation(struct dlb2_eventdev *dlb2,
+			   const struct rte_event_queue_conf *queue_conf)
+{
+	int grp_occupancy[DLB2_NUM_SN_GROUPS];
+	int grp_alloc[DLB2_NUM_SN_GROUPS];
+	int i, sequence_numbers;
+
+	sequence_numbers = (int)queue_conf->nb_atomic_order_sequences;
+
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		int total_slots;
+
+		grp_alloc[i] = dlb2_get_sn_allocation(dlb2, i);
+		if (grp_alloc[i] < 0)
+			return;
+
+		total_slots = DLB2_MAX_LDB_SN_ALLOC / grp_alloc[i];
+
+		grp_occupancy[i] = dlb2_get_sn_occupancy(dlb2, i);
+		if (grp_occupancy[i] < 0)
+			return;
+
+		/* DLB has at least one available slot for the requested
+		 * sequence numbers, so no further configuration required.
+		 */
+		if (grp_alloc[i] == sequence_numbers &&
+		    grp_occupancy[i] < total_slots)
+			return;
+	}
+
+	/* None of the sequence number groups are configured for the requested
+	 * sequence numbers, so we have to reconfigure one of them. This is
+	 * only possible if a group is not in use.
+	 */
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		if (grp_occupancy[i] == 0)
+			break;
+	}
+
+	if (i == DLB2_NUM_SN_GROUPS) {
+		DLB2_LOG_ERR("[%s()] No groups with %d sequence_numbers are available or have free slots\n",
+		       __func__, sequence_numbers);
+		return;
+	}
+
+	/* Attempt to configure slot i with the requested number of sequence
+	 * numbers. Ignore the return value -- if this fails, the error will be
+	 * caught during subsequent queue configuration.
+	 */
+	dlb2_set_sn_allocation(dlb2, i, sequence_numbers);
+}
+
+static int32_t
+dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 const struct rte_event_queue_conf *evq_conf)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_queue *queue = &ev_queue->qm_queue;
+	struct dlb2_create_ldb_queue_args cfg;
+	int32_t ret;
+	uint32_t qm_qid;
+	int sched_type = -1;
+
+	if (evq_conf == NULL)
+		return -EINVAL;
+
+	if (evq_conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_ALL_TYPES) {
+		if (evq_conf->nb_atomic_order_sequences != 0)
+			sched_type = RTE_SCHED_TYPE_ORDERED;
+		else
+			sched_type = RTE_SCHED_TYPE_PARALLEL;
+	} else
+		sched_type = evq_conf->schedule_type;
+
+	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
+	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
+	cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences;
+
+	if (sched_type != RTE_SCHED_TYPE_ORDERED) {
+		cfg.num_sequence_numbers = 0;
+		cfg.num_qid_inflights = 2048;
+	}
+
+	/* App should set this to the number of hardware flows they want, not
+	 * the overall number of flows they're going to use. E.g. if app is
+	 * using 64 flows and sets compression to 64, best-case they'll get
+	 * 64 unique hashed flows in hardware.
+	 */
+	switch (evq_conf->nb_atomic_flows) {
+	/* Valid DLB2 compression levels */
+	case 64:
+	case 128:
+	case 256:
+	case 512:
+	case (1 * 1024): /* 1K */
+	case (2 * 1024): /* 2K */
+	case (4 * 1024): /* 4K */
+	case (64 * 1024): /* 64K */
+		cfg.lock_id_comp_level = evq_conf->nb_atomic_flows;
+		break;
+	default:
+		/* Invalid compression level */
+		cfg.lock_id_comp_level = 0; /* no compression */
+	}
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create LB event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	qm_qid = cfg.response.id;
+
+	/* Save off queue config for debug, resource lookups, and reconfig */
+	queue->num_qid_inflights = cfg.num_qid_inflights;
+	queue->num_atm_inflights = cfg.num_atomic_inflights;
+
+	queue->sched_type = sched_type;
+	queue->config_state = DLB2_CONFIGURED;
+
+	DLB2_LOG_DBG("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n",
+		     qm_qid,
+		     cfg.num_atomic_inflights,
+		     cfg.num_sequence_numbers,
+		     cfg.num_qid_inflights);
+
+	return qm_qid;
+}
+
+static int
+dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int32_t qm_qid;
+
+	if (queue_conf->nb_atomic_order_sequences)
+		dlb2_program_sn_allocation(dlb2, queue_conf);
+
+	qm_qid = dlb2_hw_create_ldb_queue(dlb2, ev_queue, queue_conf);
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the load-balanced queue\n");
+
+		return qm_qid;
+	}
+
+	dlb2->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int dlb2_num_dir_queues_setup(struct dlb2_eventdev *dlb2)
+{
+	int i, num = 0;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].setup_done &&
+		    dlb2->ev_queues[i].qm_queue.is_directed)
+			num++;
+	}
+
+	return num;
+}
+
+static void
+dlb2_queue_link_teardown(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue)
+{
+	struct dlb2_eventdev_port *ev_port;
+	int i, j;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (!ev_port->link[j].valid ||
+			    ev_port->link[j].queue_id != ev_queue->id)
+				continue;
+
+			ev_port->link[j].valid = false;
+			ev_port->num_links--;
+		}
+	}
+
+	ev_queue->num_links = 0;
+}
+
+static int
+dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
+			  uint8_t ev_qid,
+			  const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_queue *ev_queue;
+	int ret;
+
+	if (queue_conf == NULL)
+		return -EINVAL;
+
+	if (ev_qid >= dlb2->num_queues)
+		return -EINVAL;
+
+	ev_queue = &dlb2->ev_queues[ev_qid];
+
+	ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg &
+		RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+	ev_queue->id = ev_qid;
+	ev_queue->conf = *queue_conf;
+
+	if (!ev_queue->qm_queue.is_directed) {
+		ret = dlb2_eventdev_ldb_queue_setup(dev, ev_queue, queue_conf);
+	} else {
+		/* The directed queue isn't setup until link time, at which
+		 * point we know its directed port ID. Directed queue setup
+		 * will only fail if this queue is already setup or there are
+		 * no directed queues left to configure.
+		 */
+		ret = 0;
+
+		ev_queue->qm_queue.config_state = DLB2_NOT_CONFIGURED;
+
+		if (ev_queue->setup_done ||
+		    dlb2_num_dir_queues_setup(dlb2) == dlb2->num_dir_queues)
+			ret = -EINVAL;
+	}
+
+	/* Tear down pre-existing port->queue links */
+	if (!ret && dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_queue_link_teardown(dlb2, ev_queue);
+
+	if (!ret)
+		ev_queue->setup_done = true;
+
+	return ret;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -696,6 +1007,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a829b9b..4c07574 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -31,3 +31,15 @@ int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				struct dlb2_create_sched_domain_args *args);
 
 void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_ldb_queue_args *cfg);
+
+int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_sn_allocation_args *args);
+
+int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_set_sn_allocation_args *args);
+
+int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_occupancy_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 6663dab..4c88fe0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -30,4 +30,16 @@ extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				 struct dlb2_create_sched_domain_args *args);
 
 extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+extern int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_queue_args *cfg);
+
+extern int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_allocation_args *args);
+
+extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_set_sn_allocation_args *args);
+
+extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				  struct dlb2_get_sn_occupancy_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 63a68a6..e6ea0d7 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3509,3 +3509,467 @@ unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
 
 	return num;
 }
+
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_ldb_queue *queue,
+				     struct dlb2_create_ldb_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+	union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+	union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+	union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+	union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+	union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+	union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+	union dlb2_sys_ldb_qid_its r7 = { {0} };
+	union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+	union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+	union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+	union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+	union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+	union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+	struct dlb2_sn_group *sn_group;
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r3.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	r4.field.limit = args->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+	r5.field.limit = queue->aqed_limit;
+
+	if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+		r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+		    r5.val);
+
+	switch (args->lock_id_comp_level) {
+	case 64:
+		r6.field.compress_code = 1;
+		break;
+	case 128:
+		r6.field.compress_code = 2;
+		break;
+	case 256:
+		r6.field.compress_code = 3;
+		break;
+	case 512:
+		r6.field.compress_code = 4;
+		break;
+	case 1024:
+		r6.field.compress_code = 5;
+		break;
+	case 2048:
+		r6.field.compress_code = 6;
+		break;
+	case 4096:
+		r6.field.compress_code = 7;
+		break;
+	case 0:
+	case 65536:
+		r6.field.compress_code = 0;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+		    r6.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r7.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+		    r7.val);
+
+	r8.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+		    r8.val);
+
+	r9.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+		    r9.val);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	r10.field.qid_fid_limit = 512;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+		    r10.val);
+
+	/* Configure SNs */
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	r11.field.mode = sn_group->mode;
+	r11.field.slot = queue->sn_slot;
+	r11.field.grp  = sn_group->id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+	r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+	r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		r0.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+		r1.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+		r2.field.vqid = queue->id.virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+			    r2.val);
+	}
+
+	r13.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+				  struct dlb2_ldb_queue *queue,
+				  struct dlb2_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb2_sn_group_full(group)) {
+			slot = dlb2_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no sequence number slots available\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+				struct dlb2_hw_domain *domain,
+				struct dlb2_ldb_queue *queue,
+				struct dlb2_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_ldb_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (dlb2_list_empty(&domain->avail_ldb_queues)) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb2_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_ldb_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		    args->num_sequence_numbers);
+	DLB2_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		    args->num_qid_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		    args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+
+	dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			    __func__, __LINE__);
+		return ret;
+	}
+
+	dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_queues, &queue->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_queues, &queue->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+						unsigned int group_id,
+						unsigned long val)
+{
+	DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+	DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+	DLB2_HW_DBG(hw, "\tValue:    %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val)
+{
+	u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+	union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+	struct dlb2_sn_group *group;
+	int mode;
+
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	group = &hw->rsrcs.sn_groups[group_id];
+
+	/*
+	 * Once the first load-balanced queue using an SN group is configured,
+	 * the group cannot be changed.
+	 */
+	if (group->slot_use_bitmap != 0)
+		return -EPERM;
+
+	for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+		if (val == valid_allocations[mode])
+			break;
+
+	if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+		return -EINVAL;
+
+	group->mode = mode;
+	group->sequence_numbers_per_queue = val;
+
+	r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+	r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+	DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+	dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 285ad2c..14bb04b 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -613,3 +613,13 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
 {
 	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_ldb_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 9310f72..a6cd178 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -150,6 +150,84 @@ dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
 		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
 }
 
+static int
+dlb2_pf_ldb_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_ldb_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_ldb_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw_dev *handle,
+			 struct dlb2_get_sn_occupancy_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_number_occupancy(&dlb2_dev->hw,
+						       args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_numbers(&dlb2_dev->hw, args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_set_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_set_group_sequence_numbers(&dlb2_dev->hw, args->group,
+					      args->num);
+
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -161,6 +239,10 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
+	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
+	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
+	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v10 13/23] event/dlb2: add port setup

[Timothy McDaniel]

Configure the load balanced (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  75 +++
 drivers/event/dlb2/dlb2.c                  | 498 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   8 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 922 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  28 +
 drivers/event/dlb2/pf/dlb2_pf.c            | 180 ++++++
 7 files changed, 1719 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index b154cac..bbd6ac8 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -82,6 +82,81 @@ The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
 the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
 load-balanced queues can use the full 16-bit flow ID range.
 
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 48e7245..f0b13a4 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -999,6 +999,503 @@ dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
 	return ret;
 }
 
+static int
+dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_cq_pop_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_cq_pop_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL)	{
+		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->consume_qe = qe;
+
+	qe->qe_valid = 0;
+	qe->qe_frag = 0;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
+	 * and so on.
+	 */
+	qe->tokens = 0;	/* set at run time */
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_enqueue_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_enqueue_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL) {
+		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->int_arm_qe = qe;
+
+	/* V2 - INT ARM is CQ_TOKEN + FRAG */
+	qe->qe_valid = 0;
+	qe->qe_frag = 1;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
+{
+	int ret, sz;
+
+	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct dlb2_enqueue_qe);
+
+	qm_port->qe4 = rte_zmalloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
+
+	if (qm_port->qe4 == NULL) {
+		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	ret = dlb2_init_consume_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	return 0;
+
+error_exit:
+
+	dlb2_free_qe_mem(qm_port);
+
+	return ret;
+}
+
+static int
+dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_ldb_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_CQ_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	cfg.cq_history_list_size = DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	if (handle->cos_id == DLB2_COS_DEFAULT)
+		cfg.cos_id = 0;
+	else
+		cfg.cos_id = handle->cos_id;
+
+	cfg.cos_strict = 0;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* If there are no directed ports, the kernel driver will ignore this
+	 * port's directed credit settings. Don't use enqueue_depth if it would
+	 * require more directed credits than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2; /* back ptr */
+	/*
+	 * Allocate and init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_ldb_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(qm_port->cfg.ldb));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     qm_port->ldb_credits,
+		     qm_port->dir_credits);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
+
+	return ret;
+}
+
+static void
+dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port)
+{
+	struct dlb2_eventdev_queue *ev_queue;
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (!ev_port->link[i].valid)
+			continue;
+
+		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
+
+		ev_port->link[i].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+}
+
+static int
+dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (dlb2 == NULL || handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* Directed queues are configured at link time. */
+	cfg.queue_id = -1;
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* Don't use enqueue_depth if it would require more directed credits
+	 * than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2;  /* back ptr */
+
+	/*
+	 * Init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE to be aligned
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_dir_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = cfg.cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(qm_port->cfg.dir));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     dir_credit_high_watermark,
+		     ldb_credit_high_watermark);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_setup(struct rte_eventdev *dev,
+			 uint8_t ev_port_id,
+			 const struct rte_event_port_conf *port_conf)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_eventdev_port *ev_port;
+	int ret;
+
+	if (dev == NULL || port_conf == NULL) {
+		DLB2_LOG_ERR("Null parameter\n");
+		return -EINVAL;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
+		return -EINVAL;
+
+	if (port_conf->dequeue_depth >
+		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
+	    port_conf->enqueue_depth >
+		evdev_dlb2_default_info.max_event_port_enqueue_depth)
+		return -EINVAL;
+
+	ev_port = &dlb2->ev_ports[ev_port_id];
+	/* configured? */
+	if (ev_port->setup_done) {
+		DLB2_LOG_ERR("evport %d is already configured\n", ev_port_id);
+		return -EINVAL;
+	}
+
+	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
+		RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (!ev_port->qm_port.is_directed) {
+		ret = dlb2_hw_create_ldb_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the lB port ve portId=%d\n",
+				     ev_port_id);
+
+			return ret;
+		}
+	} else {
+		ret = dlb2_hw_create_dir_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the DIR port\n");
+			return ret;
+		}
+	}
+
+	/* Save off port config for reconfig */
+	ev_port->conf = *port_conf;
+
+	ev_port->id = ev_port_id;
+	ev_port->enq_configured = true;
+	ev_port->setup_done = true;
+	ev_port->inflight_max = port_conf->new_event_threshold;
+	ev_port->implicit_release = !(port_conf->event_port_cfg &
+		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
+	ev_port->outstanding_releases = 0;
+	ev_port->inflight_credits = 0;
+	ev_port->credit_update_quanta = RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
+	ev_port->dlb2 = dlb2; /* reverse link */
+
+	/* Tear down pre-existing port->queue links */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
+
+	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1009,6 +1506,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
+		.port_setup       = dlb2_eventdev_port_setup,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 4c07574..01818c7 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -43,3 +43,11 @@ int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				   struct dlb2_get_sn_occupancy_args *args);
+
+int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_dir_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4c88fe0..a6b923a 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -42,4 +42,12 @@ extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				  struct dlb2_get_sn_occupancy_args *args);
+
+extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_ldb_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_dir_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index e6ea0d7..a633d3e 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3973,3 +3973,925 @@ int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_ldb_port *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_pp2vas r0 = { {0} };
+	union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+		union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_ldb_port *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_ldb_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+	union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+	union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+	union dlb2_chp_hist_list_lim r5 = { {0} };
+	union dlb2_chp_hist_list_base r6 = { {0} };
+	union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+	union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+	union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+	union dlb2_sys_ldb_cq_at r10 = { {0} };
+	union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+	union dlb2_chp_ldb_cq2vas r12 = { {0} };
+	union dlb2_lsp_cq2priov r13 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r14.field.token_count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    r14.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	r5.field.limit = port->hist_list_entry_limit - 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+	r6.field.base = port->hist_list_entry_base;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	r7.field.limit = args->cq_history_list_size;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+	r8.field.push_ptr = r6.field.base;
+	r8.field.generation = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    r8.val);
+
+	r9.field.pop_ptr = r6.field.base;
+	r9.field.generation = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+	/* Disable the port's QID mappings */
+	r13.field.v = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+	return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_ldb_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb2_ldb_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+	if (ret < 0)
+		return ret;
+
+	dlb2_ldb_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_ldb_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		    args->cq_history_list_size);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+	DLB2_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		    args->cos_strict);
+}
+
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_ldb_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+		resp->status = DLB2_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		if (dlb2_list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	} else {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			if (!dlb2_list_empty(&domain->avail_ldb_ports[i]))
+				break;
+		}
+
+		if (i == DLB2_NUM_COS_DOMAINS) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ *	its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	int ret, cos_id, i;
+
+	dlb2_log_create_ldb_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->cos_strict) {
+		cos_id = args->cos_id;
+
+		port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+					  typeof(*port));
+	} else {
+		int idx;
+
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+			port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+						  typeof(*port));
+			if (port)
+				break;
+		}
+
+		cos_id = idx;
+	}
+
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (port->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_ldb_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_ports[cos_id], &port->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_ports[cos_id], &port->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_dir_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+}
+
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+			    bool vdev_req,
+			    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+
+	return NULL;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_dir_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the queue is already configured, validate
+	 * the queue ID, its domain, and whether the queue is configured.
+	 */
+	if (args->queue_id != -1) {
+		struct dlb2_dir_pq_pair *queue;
+
+		queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+						    vdev_req,
+						    domain);
+
+		if (queue == NULL || queue->domain_id.phys_id !=
+				domain->id.phys_id ||
+				!queue->queue_configured) {
+			resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the port's queue is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->queue_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_dir_pq_pair *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_dir_pp2vas r0 = { {0} };
+	union dlb2_sys_dir_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+		union dlb2_sys_dir_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_dir_pq_pair *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_dir_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+	union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+	union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+	union dlb2_sys_dir_cq_fmt r9 = { {0} };
+	union dlb2_sys_dir_cq_at r10 = { {0} };
+	union dlb2_sys_dir_cq_pasid r11 = { {0} };
+	union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r13.field.count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    r13.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.disable_wb_opt = 0;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	r9.field.keep_pf_ppid = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+	return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_dir_pq_pair *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_dir_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb2_dir_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+
+	if (ret < 0)
+		return ret;
+
+	dlb2_dir_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ *	and queue. The port/queue pair have the same ID and name.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *port;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (args->queue_id != -1)
+		port = dlb2_get_domain_used_dir_pq(args->queue_id,
+						   vdev_req,
+						   domain);
+	else
+		port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					  typeof(*port));
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_dir_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs, &port->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs, &port->domain_list);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 14bb04b..2a089f6 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -623,3 +623,31 @@ dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_ldb_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
+
+int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_dir_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a6cd178..2a82b8f 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -228,6 +228,184 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static void *
+dlb2_alloc_coherent_aligned(const struct rte_memzone **mz, uintptr_t *phys,
+			    size_t size, int align)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t core_id = rte_lcore_id();
+	unsigned int socket_id;
+
+	snprintf(mz_name, sizeof(mz_name) - 1, "event_dlb2_pf_%lx",
+		 (unsigned long)rte_get_timer_cycles());
+	if (core_id == (unsigned int)LCORE_ID_ANY)
+		core_id = rte_get_main_lcore();
+	socket_id = rte_lcore_to_socket_id(core_id);
+	*mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
+					 RTE_MEMZONE_IOVA_CONTIG, align);
+	if (*mz == NULL) {
+		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu bytes - %s\n",
+			     size, rte_strerror(rte_errno));
+		*phys = 0;
+		return NULL;
+	}
+	*phys = (*mz)->iova;
+	return (*mz)->addr;
+}
+
+static int
+dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_ldb_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret, cq_alloc_depth;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = false;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	cq_alloc_depth = RTE_MAX(cfg->cq_depth, DLB2_MIN_HARDWARE_CQ_DEPTH);
+	alloc_sz = cq_alloc_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].cq_base = (void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+	return ret;
+}
+
+static int
+dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_dir_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = true;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	alloc_sz = cfg->cq_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
+		(void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -240,6 +418,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v10 14/23] event/dlb2: add port link

[Timothy McDaniel]

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 302 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 633 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
 6 files changed, 1007 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index f0b13a4..a7e9a4c 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1496,6 +1496,307 @@ dlb2_eventdev_port_setup(struct rte_eventdev *dev,
 	return 0;
 }
 
+static int16_t
+dlb2_hw_map_ldb_qid_to_port(struct dlb2_hw_dev *handle,
+			    uint32_t qm_port_id,
+			    uint16_t qm_qid,
+			    uint8_t priority)
+{
+	struct dlb2_map_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	/* Build message */
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+	cfg.priority = EV_TO_DLB2_PRIO(priority);
+
+	ret = dlb2_iface_map_qid(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: map qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		DLB2_LOG_ERR("dlb2: grp=%d, qm_port=%d, qm_qid=%d prio=%d\n",
+			     handle->domain_id, cfg.port_id,
+			     cfg.qid,
+			     cfg.priority);
+	} else {
+		DLB2_LOG_DBG("dlb2: mapped queue %d to qm_port %d\n",
+			     qm_qid, qm_port_id);
+	}
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port,
+			  struct dlb2_eventdev_queue *ev_queue,
+			  uint8_t priority)
+{
+	int first_avail = -1;
+	int ret, i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid) {
+			if (ev_port->link[i].queue_id == ev_queue->id &&
+			    ev_port->link[i].priority == priority) {
+				if (ev_port->link[i].mapped)
+					return 0; /* already mapped */
+				first_avail = i;
+			}
+		} else if (first_avail == -1)
+			first_avail = i;
+	}
+	if (first_avail == -1) {
+		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
+			     ev_port->qm_port.id);
+		return -EINVAL;
+	}
+
+	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
+					  ev_port->qm_port.id,
+					  ev_queue->qm_queue.id,
+					  priority);
+
+	if (!ret)
+		ev_port->link[first_avail].mapped = true;
+
+	return ret;
+}
+
+static int32_t
+dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 int32_t qm_port_id)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_queue_args cfg;
+	int32_t ret;
+
+	/* The directed port is always configured before its queue */
+	cfg.port_id = qm_port_id;
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_dir_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	int32_t qm_qid;
+
+	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port->qm_port.id);
+
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the DIR queue\n");
+		return qm_qid;
+	}
+
+	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int
+dlb2_do_port_link(struct rte_eventdev *dev,
+		  struct dlb2_eventdev_queue *ev_queue,
+		  struct dlb2_eventdev_port *ev_port,
+		  uint8_t prio)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int err;
+
+	/* Don't link until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	if (ev_queue->qm_queue.is_directed)
+		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue, ev_port);
+	else
+		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue, prio);
+
+	if (err) {
+		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
+			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
+			     ev_queue->id, ev_port->id);
+
+		rte_errno = err;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
+			uint8_t queue_id,
+			bool link_exists,
+			int index)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	bool port_is_dir, queue_is_dir;
+
+	if (queue_id > dlb2->num_queues) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	ev_queue = &dlb2->ev_queues[queue_id];
+
+	if (!ev_queue->setup_done &&
+	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	port_is_dir = ev_port->qm_port.is_directed;
+	queue_is_dir = ev_queue->qm_queue.is_directed;
+
+	if (port_is_dir != queue_is_dir) {
+		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
+			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
+			     port_is_dir ? "DIR" : "LDB", ev_port->id);
+
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if there is space for the requested link */
+	if (!link_exists && index == -1) {
+		DLB2_LOG_ERR("no space for new link\n");
+		rte_errno = -ENOSPC;
+		return -1;
+	}
+
+	/* Check if the directed port is already linked */
+	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if the directed queue is already linked */
+	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
+			     ev_queue->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
+			const uint8_t queues[], const uint8_t priorities[],
+			uint16_t nb_links)
+
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i, j;
+
+	RTE_SET_USED(dev);
+
+	if (ev_port == NULL) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!ev_port->setup_done &&
+	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
+	 * queues pointer.
+	 */
+	if (nb_links == 0) {
+		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
+		     nb_links,
+		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
+		     ev_port->id);
+
+	for (i = 0; i < nb_links; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		uint8_t queue_id, prio;
+		bool found = false;
+		int index = -1;
+
+		queue_id = queues[i];
+		prio = priorities[i];
+
+		/* Check if the link already exists. */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].valid) {
+				if (ev_port->link[j].queue_id == queue_id) {
+					found = true;
+					index = j;
+					break;
+				}
+			} else if (index == -1) {
+				index = j;
+			}
+
+		/* could not link */
+		if (index == -1)
+			break;
+
+		/* Check if already linked at the requested priority */
+		if (found && ev_port->link[j].priority == prio)
+			continue;
+
+		if (dlb2_validate_port_link(ev_port, queue_id, found, index))
+			break; /* return index of offending queue */
+
+		ev_queue = &dlb2->ev_queues[queue_id];
+
+		if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+			break; /* return index of offending queue */
+
+		ev_queue->num_links++;
+
+		ev_port->link[index].queue_id = queue_id;
+		ev_port->link[index].priority = prio;
+		ev_port->link[index].valid = true;
+		/* Entry already exists?  If so, then must be prio change */
+		if (!found)
+			ev_port->num_links++;
+	}
+	return i;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1507,6 +1808,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_link        = dlb2_eventdev_port_link,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 01818c7..7d1ba73 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -51,3 +51,9 @@ int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 				  struct dlb2_create_dir_port_args *cfg,
 				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_dir_queue_args *cfg);
+
+int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+			  struct dlb2_map_qid_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index a6b923a..4eddd14 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -50,4 +50,10 @@ extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 struct dlb2_create_dir_port_args *cfg,
 					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+					struct dlb2_create_dir_queue_args *cfg);
+
+extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+				 struct dlb2_map_qid_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index a633d3e..14ad262 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -4895,3 +4895,636 @@ int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_dir_pq_pair *queue,
+				     struct dlb2_create_dir_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_dir_vasqid_v r0 = { {0} };
+	union dlb2_sys_dir_qid_its r1 = { {0} };
+	union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+	union dlb2_sys_dir_qid_v r5 = { {0} };
+
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r0.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r1.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+		    r1.val);
+
+	r2.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+		    r2.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		r3.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+		r4.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+	}
+
+	r5.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+	queue->queue_configured = true;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_dir_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_dir_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = dlb2_get_domain_used_dir_pq(args->port_id,
+						   vdev_req,
+						   domain);
+
+		if (port == NULL || port->domain_id.phys_id !=
+				domain->id.phys_id || !port->port_configured) {
+			resp->status = DLB2_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the queue's port is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->port_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->port_id != -1)
+		queue = dlb2_get_domain_used_dir_pq(args->port_id,
+						    vdev_req,
+						    domain);
+	else
+		queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					   typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &queue->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs,
+			      &queue->domain_list);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot,
+					      struct dlb2_map_qid_args *args)
+{
+	union dlb2_lsp_cq2priov r0;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+	r0.field.v |= 1 << slot;
+	r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+	dlb2_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+					      struct dlb2_ldb_queue *queue,
+					      struct dlb2_cmd_response *resp)
+{
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB2_QUEUE_UNMAPPED;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+			  bool vdev_req,
+			  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter)
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+			      bool vdev_req,
+			      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+
+		DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+	}
+
+	return NULL;
+}
+
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+				    u32 domain_id,
+				    struct dlb2_map_qid_args *args,
+				    struct dlb2_cmd_response *resp,
+				    bool vdev_req,
+				    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	struct dlb2_ldb_queue *queue;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB2_QID_PRIORITIES) {
+		resp->status = DLB2_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_map_qid_args *args,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	DLB2_HW_DBG(hw, "\tPriority:  %d\n",
+		    args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	int ret, i, id;
+	u8 prio;
+
+	dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_map_qid_args(hw,
+				       domain_id,
+				       args,
+				       resp,
+				       vdev_req,
+				       vdev_id);
+	if (ret)
+		return ret;
+
+	prio = args->priority;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	if (port->num_pending_removals)
+		dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].pending_priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+		if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb2_qid_map_state st;
+
+			if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+				DLB2_HW_ERR(hw,
+					    "[%s():%d] Internal error: port slot tracking failed\n",
+					    __func__, __LINE__);
+				return -EFAULT;
+			}
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb2_port_slot_state_transition(hw, port, queue,
+							      i, st);
+			if (ret)
+				return ret;
+
+			DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 2a089f6..a010e25 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -651,3 +651,13 @@ dlb2_pf_create_dir_port(struct dlb2_hw *hw,
 				       NOT_VF_REQ,
 				       PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_dir_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 2a82b8f..37be4b1 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -406,6 +406,54 @@ dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_dir_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_dir_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_dir_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
+		struct dlb2_map_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_map_qid(&dlb2_dev->hw,
+			      handle->domain_id,
+			      cfg,
+			      &response,
+			      false,
+			      0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -419,7 +467,9 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
 	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
+	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v10 15/23] event/dlb2: add port unlink and port unlinks in progress

[Timothy McDaniel]

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 163 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 283 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  51 ++++++
 5 files changed, 509 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index a7e9a4c..9357b23 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1797,6 +1797,166 @@ dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
 	return i;
 }
 
+static int16_t
+dlb2_hw_unmap_ldb_qid_from_port(struct dlb2_hw_dev *handle,
+				uint32_t qm_port_id,
+				uint16_t qm_qid)
+{
+	struct dlb2_unmap_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+
+	ret = dlb2_iface_unmap_qid(handle, &cfg);
+	if (ret < 0)
+		DLB2_LOG_ERR("dlb2: unmap qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_detach_ldb(struct dlb2_eventdev *dlb2,
+			    struct dlb2_eventdev_port *ev_port,
+			    struct dlb2_eventdev_queue *ev_queue)
+{
+	int ret, i;
+
+	/* Don't unlink until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid &&
+		    ev_port->link[i].queue_id == ev_queue->id)
+			break; /* found */
+	}
+
+	/* This is expected with eventdev API!
+	 * It blindly attemmpts to unmap all queues.
+	 */
+	if (i == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_LOG_DBG("dlb2: ignoring LB QID %d not mapped for qm_port %d.\n",
+			     ev_queue->qm_queue.id,
+			     ev_port->qm_port.id);
+		return 0;
+	}
+
+	ret = dlb2_hw_unmap_ldb_qid_from_port(&dlb2->qm_instance,
+					      ev_port->qm_port.id,
+					      ev_queue->qm_queue.id);
+	if (!ret)
+		ev_port->link[i].mapped = false;
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
+			  uint8_t queues[], uint16_t nb_unlinks)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (queues == NULL || nb_unlinks == 0) {
+		DLB2_LOG_DBG("dlb2: queues is NULL or nb_unlinks is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	/* FIXME: How to handle unlink on a directed port? */
+	if (ev_port->qm_port.is_directed) {
+		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
+			     ev_port->id);
+		rte_errno = 0;
+		return nb_unlinks; /* as if success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	for (i = 0; i < nb_unlinks; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		int ret, j;
+
+		if (queues[i] >= dlb2->num_queues) {
+			DLB2_LOG_ERR("dlb2: invalid queue id %d\n", queues[i]);
+			rte_errno = -EINVAL;
+			return i; /* return index of offending queue */
+		}
+
+		ev_queue = &dlb2->ev_queues[queues[i]];
+
+		/* Does a link exist? */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].queue_id == queues[i] &&
+			    ev_port->link[j].valid)
+				break;
+
+		if (j == DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+			continue;
+
+		ret = dlb2_event_queue_detach_ldb(dlb2, ev_port, ev_queue);
+		if (ret) {
+			DLB2_LOG_ERR("unlink err=%d for port %d queue %d\n",
+				     ret, ev_port->id, queues[i]);
+			rte_errno = -ENOENT;
+			return i; /* return index of offending queue */
+		}
+
+		ev_port->link[j].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+
+	return nb_unlinks;
+}
+
+static int
+dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
+				       void *event_port)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	struct dlb2_pending_port_unmaps_args cfg;
+	int ret;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	cfg.port_id = ev_port->qm_port.id;
+	dlb2 = ev_port->dlb2;
+	handle = &dlb2->qm_instance;
+	ret = dlb2_iface_pending_port_unmaps(handle, &cfg);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: num_unlinks_in_progress ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1809,6 +1969,9 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
 		.port_link        = dlb2_eventdev_port_link,
+		.port_unlink      = dlb2_eventdev_port_unlink,
+		.port_unlinks_in_progress =
+				    dlb2_eventdev_port_unlinks_in_progress,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 7d1ba73..bd241f3 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -57,3 +57,9 @@ int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 			  struct dlb2_map_qid_args *cfg);
+
+int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+			    struct dlb2_unmap_qid_args *cfg);
+
+int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4eddd14..e892036 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -56,4 +56,10 @@ extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 				 struct dlb2_map_qid_args *cfg);
+
+extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+				   struct dlb2_unmap_qid_args *cfg);
+
+extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 14ad262..4a3d96d 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5528,3 +5528,286 @@ int dlb2_hw_map_qid(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_unmap_qid_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+		DLB2_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			    hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+				      u32 domain_id,
+				      struct dlb2_unmap_qid_args *args,
+				      struct dlb2_cmd_response *resp,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	struct dlb2_ldb_port *port;
+	int slot;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			    __func__, args->qid);
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		return 0;
+
+	resp->status = DLB2_ST_INVALID_QID;
+	return -EINVAL;
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	bool unmap_complete;
+	int i, ret, id;
+
+	dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_unmap_qid_args(hw,
+					 domain_id,
+					 args,
+					 resp,
+					 vdev_req,
+					 vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+		st = DLB2_QUEUE_UNMAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		st = DLB2_QUEUE_UNMAP_IN_PROG;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB2_QUEUE_MAPPED;
+	if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: no available CQ slots\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asynchronous procedure. It requires
+	 * stopping the DLB2 from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb2_ldb_port_cq_disable(hw, port);
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete && !os_worker_active(hw))
+		os_schedule_work(hw);
+
+unmap_qid_done:
+	resp->status = 0;
+
+	return 0;
+}
+
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+				  struct dlb2_pending_port_unmaps_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+
+	dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 37be4b1..33ea73e 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -454,6 +454,55 @@ dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw_dev *handle,
+		  struct dlb2_unmap_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_unmap_qid(&dlb2_dev->hw,
+				handle->domain_id,
+				cfg,
+				&response,
+				false,
+				0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
+			    struct dlb2_pending_port_unmaps_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_pending_port_unmaps(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -470,6 +519,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
+	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v10 16/23] event/dlb2: add eventdev start

[Timothy McDaniel]

Add support for the eventdev start entry point.
We delay initializing some resources until
eventdev start, since the number of linked queues can be
used to determine if we are dealing with a ldb or dir resource.
If this is a device restart, then the previous configuration
will be reapplied.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 133 +++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   3 +
 drivers/event/dlb2/dlb2_iface.h            |   3 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 123 ++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +++
 drivers/event/dlb2/pf/dlb2_pf.c            |  25 ++++++
 6 files changed, 297 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 9357b23..fa6fa7b 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1957,6 +1957,138 @@ dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
 	return cfg.response.id;
 }
 
+static int
+dlb2_eventdev_reapply_configuration(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret, i;
+
+	/* If an event queue or port was previously configured, but hasn't been
+	 * reconfigured, reapply its original configuration.
+	 */
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+
+		ev_queue = &dlb2->ev_queues[i];
+
+		if (ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_queue_setup(dev, i, &ev_queue->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure queue %d", i);
+			return ret;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+
+		if (ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_port_setup(dev, i, &ev_port->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure ev_port %d",
+				     i);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_apply_port_links(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int i;
+
+	/* Perform requested port->queue links */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+		int j;
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			struct dlb2_eventdev_queue *ev_queue;
+			uint8_t prio, queue_id;
+
+			if (!ev_port->link[j].valid)
+				continue;
+
+			prio = ev_port->link[j].priority;
+			queue_id = ev_port->link[j].queue_id;
+
+			if (dlb2_validate_port_link(ev_port, queue_id, true, j))
+				return -EINVAL;
+
+			ev_queue = &dlb2->ev_queues[queue_id];
+
+			if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_start(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_start_domain_args cfg;
+	int ret, i;
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_ERR("bad state %d for dev_start\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return -EINVAL;
+	}
+	dlb2->run_state = DLB2_RUN_STATE_STARTING;
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	/* If the device was configured more than once, some event ports and/or
+	 * queues may need to be reconfigured.
+	 */
+	ret = dlb2_eventdev_reapply_configuration(dev);
+	if (ret)
+		return ret;
+
+	/* The DLB PMD delays port links until the device is started. */
+	ret = dlb2_eventdev_apply_port_links(dev);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		if (!dlb2->ev_ports[i].setup_done) {
+			DLB2_LOG_ERR("dlb2: port %d not setup", i);
+			return -ESTALE;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0) {
+			DLB2_LOG_ERR("dlb2: queue %d is not linked", i);
+			return -ENOLINK;
+		}
+	}
+
+	ret = dlb2_iface_sched_domain_start(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: sched_domain_start ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STARTED;
+	DLB2_LOG_DBG("dlb2: sched_domain_start completed OK\n");
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1964,6 +2096,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.dev_start        = dlb2_eventdev_start,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index bd241f3..a86191d 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -63,3 +63,6 @@ int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				     struct dlb2_start_domain_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index e892036..7ead374 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -62,4 +62,7 @@ extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				struct dlb2_start_domain_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 4a3d96d..c14a2f3 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5811,3 +5811,126 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 struct dlb2_cmd_response *resp,
+					 bool vdev_req,
+					 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_start_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 start domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_hw_start_domain() - Lock the domain configuration
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @arg: User-provided arguments (unused, here for ioctl callback template).
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int
+dlb2_hw_start_domain(struct dlb2_hw *hw,
+		     u32 domain_id,
+		     __attribute((unused)) struct dlb2_start_domain_args *arg,
+		     struct dlb2_cmd_response *resp,
+		     bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_queue;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+	RTE_SET_USED(arg);
+	RTE_SET_USED(iter);
+
+	dlb2_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb2_verify_start_domain_args(hw,
+					    domain_id,
+					    resp,
+					    vdev_req,
+					    vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB2 will drop all
+	 * incoming traffic to those queues.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, ldb_queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r0.val);
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+	}
+
+	dlb2_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index a010e25..aee8336 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -661,3 +661,13 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_start_domain(struct dlb2_hw *hw,
+		     u32 id,
+		     struct dlb2_start_domain_args *args,
+		     struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_start_domain(hw, id, args, resp, NOT_VF_REQ,
+				    PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 33ea73e..fff2e57 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -503,6 +503,30 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
 
 	return ret;
 }
+
+static int
+dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
+			   struct dlb2_start_domain_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_start_domain(&dlb2_dev->hw,
+				   handle->domain_id,
+				   cfg,
+				   &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -520,6 +544,7 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v10 17/23] event/dlb2: add enqueue and its burst variants

[Timothy McDaniel]

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst | 118 +++++++++
 drivers/event/dlb2/dlb2.c     | 540 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 658 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index bbd6ac8..aa8bf01 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -163,6 +163,124 @@ Flow ID
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Hardware Credits
+~~~~~~~~~~~~~~~~
+
+DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
+event storage, with each unit of storage represented by a credit. A port spends
+a credit to enqueue an event, and hardware refills the ports with credits as the
+events are scheduled to ports. Refills come from credit pools, and each port is
+a member of a load-balanced credit pool and a directed credit pool. The
+load-balanced credits are used to enqueue to load-balanced queues, and directed
+credits are used for directed queues.
+
+A DLB2 eventdev contains one load-balanced and one directed credit pool. These
+pools' sizes are controlled by the nb_events_limit field in struct
+rte_event_dev_config. The load-balanced pool is sized to contain
+nb_events_limit credits, and the directed pool is sized to contain
+nb_events_limit/4 credits. The directed pool size can be overridden with the
+num_dir_credits vdev argument, like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,num_dir_credits=<value>
+
+This can be used if the default allocation is too low or too high for the
+specific application needs. The PMD also supports a vdev arg that limits the
+max_num_events reported by rte_event_dev_info_get():
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,max_num_events=<value>
+
+By default, max_num_events is reported as the total available load-balanced
+credits. If multiple DLB2-based applications are being used, it may be desirable
+to control how many load-balanced credits each application uses, particularly
+when application(s) are written to configure nb_events_limit equal to the
+reported max_num_events.
+
+Each port is a member of both credit pools. A port's credit allocation is
+defined by its low watermark, high watermark, and refill quanta. These three
+parameters are calculated by the dlb PMD like so:
+
+- The load-balanced high watermark is set to the port's enqueue_depth.
+  The directed high watermark is set to the minimum of the enqueue_depth and
+  the directed pool size divided by the total number of ports.
+- The refill quanta is set to half the high watermark.
+- The low watermark is set to the minimum of 16 and the refill quanta.
+
+When the eventdev is started, each port is pre-allocated a high watermark's
+worth of credits. For example, if an eventdev contains four ports with enqueue
+depths of 32 and a load-balanced credit pool size of 4096, each port will start
+with 32 load-balanced credits, and there will be 3968 credits available to
+replenish the ports. Thus, a single port is not capable of enqueueing up to the
+nb_events_limit (without any events being dequeued), since the other ports are
+retaining their initial credit allocation; in short, all ports must enqueue in
+order to reach the limit.
+
+If a port attempts to enqueue and has no credits available, the enqueue
+operation will fail and the application must retry the enqueue. Credits are
+replenished asynchronously by the DLB2 hardware.
+
+Software Credits
+~~~~~~~~~~~~~~~~
+
+The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
+credit scheme on top of the hardware credit scheme in order to comply with
+the per-port backpressure described in the eventdev API.
+
+The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
+credit when it enqueues to a queue, and credits are later replenished after the
+events are dequeued and released.
+
+In the software credit scheme, a credit is consumed when a new (.op =
+RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
+replenished when the event is released from the system (either explicitly with
+RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
+
+In this model, an event is "in the system" from its first enqueue into eventdev
+until it is last dequeued. If the event goes through multiple event queues, it
+is still considered "in the system" while a worker thread is processing it.
+
+A port will fail to enqueue if the number of events in the system exceeds its
+``new_event_threshold`` (specified at port setup time). A port will also fail
+to enqueue if it lacks enough hardware credits to enqueue; load-balanced
+credits are used to enqueue to a load-balanced queue, and directed credits are
+used to enqueue to a directed queue.
+
+The out-of-credit situations are typically transient, and an eventdev
+application using the DLB2 ought to retry its enqueues if they fail.
+If enqueue fails, DLB2 PMD sets rte_errno as follows:
+
+- -ENOSPC: Credit exhaustion (either hardware or software)
+- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
+
+Depending on the pipeline the application has constructed, it's possible to
+enter a credit deadlock scenario wherein the worker thread lacks the credit
+to enqueue an event, and it must dequeue an event before it can recover the
+credit. If the worker thread retries its enqueue indefinitely, it will not
+make forward progress. Such deadlock is possible if the application has event
+"loops", in which an event in dequeued from queue A and later enqueued back to
+queue A.
+
+Due to this, workers should stop retrying after a time, release the events it
+is attempting to enqueue, and dequeue more events. It is important that the
+worker release the events and don't simply set them aside to retry the enqueue
+again later, because the port has limited history list size (by default, twice
+the port's dequeue_depth).
+
+Priority
+~~~~~~~~
+
+The DLB2 supports event priority and per-port queue service priority, as
+described in the eventdev header file. The DLB2 does not support 'global' event
+queue priority established at queue creation time.
+
+DLB2 supports 8 event and queue service priority levels. For both priority
+types, the PMD uses the upper three bits of the priority field to determine the
+DLB2 priority, discarding the 5 least significant bits. The 5 least significant
+event priority bits are not preserved when an event is enqueued.
+
 Reconfiguration
 ~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index fa6fa7b..faf45c8 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2089,6 +2089,540 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
 	return 0;
 }
 
+static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
+	{
+		/* Load-balanced cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
+	},
+	{
+		/* Directed cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
+	},
+};
+
+static inline uint32_t
+dlb2_port_credits_get(struct dlb2_port *qm_port,
+		      enum dlb2_hw_queue_types type)
+{
+	uint32_t credits = *qm_port->credit_pool[type];
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	if (unlikely(credits < batch_size))
+		batch_size = credits;
+
+	if (likely(credits &&
+		   __atomic_compare_exchange_n(
+			qm_port->credit_pool[type],
+			&credits, credits - batch_size, false,
+			__ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
+		return batch_size;
+	else
+		return 0;
+}
+
+static inline void
+dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port)
+{
+	uint16_t quanta = ev_port->credit_update_quanta;
+
+	if (ev_port->inflight_credits >= quanta * 2) {
+		/* Replenish credits, saving one quanta for enqueues */
+		uint16_t val = ev_port->inflight_credits - quanta;
+
+		__atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits -= val;
+	}
+}
+
+static inline int
+dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
+						__ATOMIC_SEQ_CST);
+	const int num = 1;
+
+	if (unlikely(ev_port->inflight_max < sw_inflights)) {
+		DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
+		rte_errno = -ENOSPC;
+		return 1;
+	}
+
+	if (ev_port->inflight_credits < num) {
+		/* check if event enqueue brings ev_port over max threshold */
+		uint32_t credit_update_quanta = ev_port->credit_update_quanta;
+
+		if (sw_inflights + credit_update_quanta >
+				dlb2->new_event_limit) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_new_event_limit,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+
+		__atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
+				   __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits += (credit_update_quanta);
+
+		if (ev_port->inflight_credits < num) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_inflight_credits,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_ldb_credits == 0)) {
+		qm_port->cached_ldb_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_LDB_QUEUE);
+		if (unlikely(qm_port->cached_ldb_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
+			1);
+			DLB2_LOG_DBG("ldb credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_dir_credits == 0)) {
+		qm_port->cached_dir_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_DIR_QUEUE);
+		if (unlikely(qm_port->cached_dir_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
+			1);
+			DLB2_LOG_DBG("dir credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static __rte_always_inline void
+dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
+	      struct process_local_port_data *port_data)
+{
+	dlb2_movdir64b(port_data->pp_addr, qe4);
+}
+
+static inline void
+dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
+		   bool do_sfence,
+		   struct process_local_port_data *port_data)
+{
+	/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
+	 * application writes complete before enqueueing the QE.
+	 */
+	if (do_sfence)
+		rte_wmb();
+
+	dlb2_pp_write(qm_port->qe4, port_data);
+}
+
+static inline void
+dlb2_event_build_hcws(struct dlb2_port *qm_port,
+		      const struct rte_event ev[],
+		      int num,
+		      uint8_t *sched_type,
+		      uint8_t *queue_id)
+{
+	struct dlb2_enqueue_qe *qe;
+	uint16_t sched_word[4];
+	__m128i sse_qe[2];
+	int i;
+
+	qe = qm_port->qe4;
+
+	sse_qe[0] = _mm_setzero_si128();
+	sse_qe[1] = _mm_setzero_si128();
+
+	switch (num) {
+	case 4:
+		/* Construct the metadata portion of two HCWs in one 128b SSE
+		 * register. HCW metadata is constructed in the SSE registers
+		 * like so:
+		 * sse_qe[0][63:0]:   qe[0]'s metadata
+		 * sse_qe[0][127:64]: qe[1]'s metadata
+		 * sse_qe[1][63:0]:   qe[2]'s metadata
+		 * sse_qe[1][127:64]: qe[3]'s metadata
+		 */
+
+		/* Convert the event operation into a command byte and store it
+		 * in the metadata:
+		 * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
+		 * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
+		 * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
+		 * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
+		 */
+#define DLB2_QE_CMD_BYTE 7
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[0].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[1].op],
+				DLB2_QE_CMD_BYTE + 8);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[2].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[3].op],
+				DLB2_QE_CMD_BYTE + 8);
+
+		/* Store priority, scheduling type, and queue ID in the sched
+		 * word array because these values are re-used when the
+		 * destination is a directed queue.
+		 */
+		sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
+				sched_type[0] << 8 |
+				queue_id[0];
+		sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
+				sched_type[1] << 8 |
+				queue_id[1];
+		sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
+				sched_type[2] << 8 |
+				queue_id[2];
+		sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
+				sched_type[3] << 8 |
+				queue_id[3];
+
+		/* Store the event priority, scheduling type, and queue ID in
+		 * the metadata:
+		 * sse_qe[0][31:16] = sched_word[0]
+		 * sse_qe[0][95:80] = sched_word[1]
+		 * sse_qe[1][31:16] = sched_word[2]
+		 * sse_qe[1][95:80] = sched_word[3]
+		 */
+#define DLB2_QE_QID_SCHED_WORD 1
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[0],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[1],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[2],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[3],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+
+		/* If the destination is a load-balanced queue, store the lock
+		 * ID. If it is a directed queue, DLB places this field in
+		 * bytes 10-11 of the received QE, so we format it accordingly:
+		 * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
+		 * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
+		 * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
+		 * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
+		 */
+#define DLB2_QE_LOCK_ID_WORD 2
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[0] == DLB2_SCHED_DIRECTED) ?
+					sched_word[0] : ev[0].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[1] == DLB2_SCHED_DIRECTED) ?
+					sched_word[1] : ev[1].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[2] == DLB2_SCHED_DIRECTED) ?
+					sched_word[2] : ev[2].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[3] == DLB2_SCHED_DIRECTED) ?
+					sched_word[3] : ev[3].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+
+		/* Store the event type and sub event type in the metadata:
+		 * sse_qe[0][15:0]  = flow_id[0]
+		 * sse_qe[0][79:64] = flow_id[1]
+		 * sse_qe[1][15:0]  = flow_id[2]
+		 * sse_qe[1][79:64] = flow_id[3]
+		 */
+#define DLB2_QE_EV_TYPE_WORD 0
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[0].sub_event_type << 8 |
+						ev[0].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[1].sub_event_type << 8 |
+						ev[1].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[2].sub_event_type << 8 |
+						ev[2].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[3].sub_event_type << 8 |
+						ev[3].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+
+		/* Store the metadata to memory (use the double-precision
+		 * _mm_storeh_pd because there is no integer function for
+		 * storing the upper 64b):
+		 * qe[0] metadata = sse_qe[0][63:0]
+		 * qe[1] metadata = sse_qe[0][127:64]
+		 * qe[2] metadata = sse_qe[1][63:0]
+		 * qe[3] metadata = sse_qe[1][127:64]
+		 */
+		_mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
+		_mm_storeh_pd((double *)&qe[1].u.opaque_data,
+			      (__m128d)sse_qe[0]);
+		_mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
+		_mm_storeh_pd((double *)&qe[3].u.opaque_data,
+			      (__m128d)sse_qe[1]);
+
+		qe[0].data = ev[0].u64;
+		qe[1].data = ev[1].u64;
+		qe[2].data = ev[2].u64;
+		qe[3].data = ev[3].u64;
+
+		break;
+	case 3:
+	case 2:
+	case 1:
+		/* At least one QE will be valid, so only zero out three */
+		qe[1].cmd_byte = 0;
+		qe[2].cmd_byte = 0;
+		qe[3].cmd_byte = 0;
+
+		for (i = 0; i < num; i++) {
+			qe[i].cmd_byte =
+				cmd_byte_map[qm_port->is_directed][ev[i].op];
+			qe[i].sched_type = sched_type[i];
+			qe[i].data = ev[i].u64;
+			qe[i].qid = queue_id[i];
+			qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
+			qe[i].lock_id = ev[i].flow_id;
+			if (sched_type[i] == DLB2_SCHED_DIRECTED) {
+				struct dlb2_msg_info *info =
+					(struct dlb2_msg_info *)&qe[i].lock_id;
+
+				info->qid = queue_id[i];
+				info->sched_type = DLB2_SCHED_DIRECTED;
+				info->priority = qe[i].priority;
+			}
+			qe[i].u.event_type.major = ev[i].event_type;
+			qe[i].u.event_type.sub = ev[i].sub_event_type;
+		}
+		break;
+	}
+}
+
+static inline int
+dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
+			struct dlb2_port *qm_port,
+			const struct rte_event ev[],
+			uint8_t *sched_type,
+			uint8_t *queue_id)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	uint16_t *cached_credits = NULL;
+	struct dlb2_queue *qm_queue;
+
+	ev_queue = &dlb2->ev_queues[ev->queue_id];
+	qm_queue = &ev_queue->qm_queue;
+	*queue_id = qm_queue->id;
+
+	/* Ignore sched_type and hardware credits on release events */
+	if (ev->op == RTE_EVENT_OP_RELEASE)
+		goto op_check;
+
+	if (!qm_queue->is_directed) {
+		/* Load balanced destination queue */
+
+		if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_ldb_credits;
+
+		switch (ev->sched_type) {
+		case RTE_SCHED_TYPE_ORDERED:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
+			if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
+				DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
+					     *queue_id);
+				rte_errno = -EINVAL;
+				return 1;
+			}
+			*sched_type = DLB2_SCHED_ORDERED;
+			break;
+		case RTE_SCHED_TYPE_ATOMIC:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
+			*sched_type = DLB2_SCHED_ATOMIC;
+			break;
+		case RTE_SCHED_TYPE_PARALLEL:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
+			if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
+				*sched_type = DLB2_SCHED_ORDERED;
+			else
+				*sched_type = DLB2_SCHED_UNORDERED;
+			break;
+		default:
+			DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
+			DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
+			rte_errno = -EINVAL;
+			return 1;
+		}
+	} else {
+		/* Directed destination queue */
+
+		if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_dir_credits;
+
+		DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
+
+		*sched_type = DLB2_SCHED_DIRECTED;
+	}
+
+op_check:
+	switch (ev->op) {
+	case RTE_EVENT_OP_NEW:
+		/* Check that a sw credit is available */
+		if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		ev_port->inflight_credits--;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_FORWARD:
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_RELEASE:
+		ev_port->inflight_credits++;
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+
+		/* Replenish s/w credits if enough are cached */
+		dlb2_replenish_sw_credits(dlb2, ev_port);
+		break;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+	if (ev->op != RTE_EVENT_OP_RELEASE) {
+		DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
+		DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
+	}
+#endif
+
+	return 0;
+}
+
+static inline uint16_t
+dlb2_event_enqueue_burst(void *event_port,
+			 const struct rte_event events[],
+			 uint16_t num)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct process_local_port_data *port_data;
+	int i, cnt;
+
+	RTE_ASSERT(ev_port->enq_configured);
+	RTE_ASSERT(events != NULL);
+
+	cnt = 0;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
+		int j = 0;
+
+		for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
+			const struct rte_event *ev = &events[i + j];
+
+			if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
+						    &sched_types[j],
+						    &queue_ids[j]))
+				break;
+		}
+
+		if (j == 0)
+			break;
+
+		dlb2_event_build_hcws(qm_port, &events[i], j,
+				      sched_types, queue_ids);
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+
+		if (j < DLB2_NUM_QES_PER_CACHE_LINE)
+			break;
+	}
+
+	return cnt;
+}
+
+static inline uint16_t
+dlb2_event_enqueue(void *event_port,
+		   const struct rte_event events[])
+{
+	return dlb2_event_enqueue_burst(event_port, events, 1);
+}
+
+static uint16_t
+dlb2_event_enqueue_new_burst(void *event_port,
+			     const struct rte_event events[],
+			     uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
+static uint16_t
+dlb2_event_enqueue_forward_burst(void *event_port,
+				 const struct rte_event events[],
+				 uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2112,7 +2646,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
 	};
 
+	/* Expose PMD's eventdev interface */
+
 	dev->dev_ops = &dlb2_eventdev_entry_ops;
+	dev->enqueue = dlb2_event_enqueue;
+	dev->enqueue_burst = dlb2_event_enqueue_burst;
+	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
+	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v10 18/23] event/dlb2: add dequeue and its burst variants

[Timothy McDaniel]

Add support for dequeue, dequeue_burst, ...

DLB2 does not currently support interrupts, but instead use
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB2 supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst |  21 ++
 drivers/event/dlb2/dlb2.c     | 783 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 804 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index aa8bf01..b9f57fd 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -314,6 +314,27 @@ The PMD does not support the following configuration sequences:
 This sequence is not supported because the event device must be reconfigured
 before its ports or queues can be.
 
+Deferred Scheduling
+~~~~~~~~~~~~~~~~~~~
+
+The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
+dequeued event before returning from rte_event_dequeue_burst(). This frees the
+corresponding entries in the CQ, which enables the DLB2 to schedule more events
+to it.
+
+To support applications seeking finer-grained scheduling control -- for example
+deferring scheduling to get the best possible priority scheduling and
+load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
+the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
+call. This mode only applies to load-balanced event ports with dequeue depth of
+1.
+
+To enable deferred scheduling, use the defer_sched vdev argument like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,defer_sched=on
+
 Atomic Inflights Allocation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index faf45c8..631fe52 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -26,6 +26,7 @@
 #include <rte_log.h>
 #include <rte_malloc.h>
 #include <rte_mbuf.h>
+#include <rte_power_intrinsics.h>
 #include <rte_prefetch.h>
 #include <rte_ring.h>
 #include <rte_string_fns.h>
@@ -2227,6 +2228,32 @@ dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
 	dlb2_movdir64b(port_data->pp_addr, qe4);
 }
 
+static inline int
+dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_cq_pop_qe *qe;
+
+	RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
+
+	qe = qm_port->consume_qe;
+
+	qe->tokens = num - 1;
+
+	/* No store fence needed since no pointer is being sent, and CQ token
+	 * pops can be safely reordered with other HCWs.
+	 */
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	dlb2_movntdq_single(port_data->pp_addr, qe);
+
+	DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
+
+	qm_port->owed_tokens = 0;
+
+	return 0;
+}
+
 static inline void
 dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
 		   bool do_sfence,
@@ -2623,9 +2650,756 @@ dlb2_event_enqueue_forward_burst(void *event_port,
 	return dlb2_event_enqueue_burst(event_port, events, num);
 }
 
+static inline void
+dlb2_port_credits_inc(struct dlb2_port *qm_port, int num)
+{
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	/* increment port credits, and return to pool if exceeds threshold */
+	if (!qm_port->is_directed) {
+		qm_port->cached_ldb_credits += num;
+		if (qm_port->cached_ldb_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_LDB_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_ldb_credits -= batch_size;
+		}
+	} else {
+		qm_port->cached_dir_credits += num;
+		if (qm_port->cached_dir_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_DIR_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_dir_credits -= batch_size;
+		}
+	}
+}
+
+static inline int
+dlb2_dequeue_wait(struct dlb2_eventdev *dlb2,
+		  struct dlb2_eventdev_port *ev_port,
+		  struct dlb2_port *qm_port,
+		  uint64_t timeout,
+		  uint64_t start_ticks)
+{
+	struct process_local_port_data *port_data;
+	uint64_t elapsed_ticks;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	elapsed_ticks = rte_get_timer_cycles() - start_ticks;
+
+	/* Wait/poll time expired */
+	if (elapsed_ticks >= timeout) {
+		return 1;
+	} else if (dlb2->umwait_allowed) {
+		volatile struct dlb2_dequeue_qe *cq_base;
+		union {
+			uint64_t raw_qe[2];
+			struct dlb2_dequeue_qe qe;
+		} qe_mask;
+		uint64_t expected_value;
+		volatile uint64_t *monitor_addr;
+
+		qe_mask.qe.cq_gen = 1; /* set mask */
+
+		cq_base = port_data->cq_base;
+		monitor_addr = (volatile uint64_t *)(volatile void *)
+			&cq_base[qm_port->cq_idx];
+		monitor_addr++; /* cq_gen bit is in second 64bit location */
+
+		if (qm_port->gen_bit)
+			expected_value = qe_mask.raw_qe[1];
+		else
+			expected_value = 0;
+
+		rte_power_monitor(monitor_addr, expected_value,
+				  qe_mask.raw_qe[1], timeout + start_ticks,
+				  sizeof(uint64_t));
+
+		DLB2_INC_STAT(ev_port->stats.traffic.rx_umonitor_umwait, 1);
+	} else {
+		uint64_t poll_interval = RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL;
+		uint64_t curr_ticks = rte_get_timer_cycles();
+		uint64_t init_ticks = curr_ticks;
+
+		while ((curr_ticks - start_ticks < timeout) &&
+		       (curr_ticks - init_ticks < poll_interval))
+			curr_ticks = rte_get_timer_cycles();
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
+			 struct dlb2_port *qm_port,
+			 struct rte_event *events,
+			 struct dlb2_dequeue_qe *qes,
+			 int cnt)
+{
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	int i, num, evq_id;
+
+	for (i = 0, num = 0; i < cnt; i++) {
+		struct dlb2_dequeue_qe *qe = &qes[i];
+		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
+			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+			[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+		};
+
+		/* Fill in event information.
+		 * Note that flow_id must be embedded in the data by
+		 * the app, such as the mbuf RSS hash field if the data
+		 * buffer is a mbuf.
+		 */
+		if (unlikely(qe->error)) {
+			DLB2_LOG_ERR("QE error bit ON\n");
+			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
+			dlb2_consume_qe_immediate(qm_port, 1);
+			continue; /* Ignore */
+		}
+
+		events[num].u64 = qe->data;
+		events[num].flow_id = qe->flow_id;
+		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
+		events[num].event_type = qe->u.event_type.major;
+		events[num].sub_event_type = qe->u.event_type.sub;
+		events[num].sched_type = sched_type_map[qe->sched_type];
+		events[num].impl_opaque = qe->qid_depth;
+
+		/* qid not preserved for directed queues */
+		if (qm_port->is_directed)
+			evq_id = ev_port->link[0].queue_id;
+		else
+			evq_id = qid_mappings[qe->qid];
+
+		events[num].queue_id = evq_id;
+		DLB2_INC_STAT(
+			ev_port->stats.queue[evq_id].qid_depth[qe->qid_depth],
+			1);
+		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type], 1);
+		num++;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num);
+
+	return num;
+}
+
+static inline int
+dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
+			      struct dlb2_port *qm_port,
+			      struct rte_event *events,
+			      struct dlb2_dequeue_qe *qes)
+{
+	int sched_type_map[] = {
+		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+	};
+	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	__m128i sse_evt[2];
+
+	/* In the unlikely case that any of the QE error bits are set, process
+	 * them one at a time.
+	 */
+	if (unlikely(qes[0].error || qes[1].error ||
+		     qes[2].error || qes[3].error))
+		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
+						 qes, num_events);
+
+	events[0].u64 = qes[0].data;
+	events[1].u64 = qes[1].data;
+	events[2].u64 = qes[2].data;
+	events[3].u64 = qes[3].data;
+
+	/* Construct the metadata portion of two struct rte_events
+	 * in one 128b SSE register. Event metadata is constructed in the SSE
+	 * registers like so:
+	 * sse_evt[0][63:0]:   event[0]'s metadata
+	 * sse_evt[0][127:64]: event[1]'s metadata
+	 * sse_evt[1][63:0]:   event[2]'s metadata
+	 * sse_evt[1][127:64]: event[3]'s metadata
+	 */
+	sse_evt[0] = _mm_setzero_si128();
+	sse_evt[1] = _mm_setzero_si128();
+
+	/* Convert the hardware queue ID to an event queue ID and store it in
+	 * the metadata:
+	 * sse_evt[0][47:40]   = qid_mappings[qes[0].qid]
+	 * sse_evt[0][111:104] = qid_mappings[qes[1].qid]
+	 * sse_evt[1][47:40]   = qid_mappings[qes[2].qid]
+	 * sse_evt[1][111:104] = qid_mappings[qes[3].qid]
+	 */
+#define DLB_EVENT_QUEUE_ID_BYTE 5
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[0].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[1].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[2].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[3].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+
+	/* Convert the hardware priority to an event priority and store it in
+	 * the metadata, while also returning the queue depth status
+	 * value captured by the hardware, storing it in impl_opaque, which can
+	 * be read by the application but not modified
+	 * sse_evt[0][55:48]   = DLB2_TO_EV_PRIO(qes[0].priority)
+	 * sse_evt[0][63:56]   = qes[0].qid_depth
+	 * sse_evt[0][119:112] = DLB2_TO_EV_PRIO(qes[1].priority)
+	 * sse_evt[0][127:120] = qes[1].qid_depth
+	 * sse_evt[1][55:48]   = DLB2_TO_EV_PRIO(qes[2].priority)
+	 * sse_evt[1][63:56]   = qes[2].qid_depth
+	 * sse_evt[1][119:112] = DLB2_TO_EV_PRIO(qes[3].priority)
+	 * sse_evt[1][127:120] = qes[3].qid_depth
+	 */
+#define DLB_EVENT_PRIO_IMPL_OPAQUE_WORD 3
+#define DLB_BYTE_SHIFT 8
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[0].priority) |
+			(qes[0].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[1].priority) |
+			(qes[1].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[2].priority) |
+			(qes[2].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[3].priority) |
+			(qes[3].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+
+	/* Write the event type, sub event type, and flow_id to the event
+	 * metadata.
+	 * sse_evt[0][31:0]   = qes[0].flow_id |
+	 *			qes[0].u.event_type.major << 28 |
+	 *			qes[0].u.event_type.sub << 20;
+	 * sse_evt[0][95:64]  = qes[1].flow_id |
+	 *			qes[1].u.event_type.major << 28 |
+	 *			qes[1].u.event_type.sub << 20;
+	 * sse_evt[1][31:0]   = qes[2].flow_id |
+	 *			qes[2].u.event_type.major << 28 |
+	 *			qes[2].u.event_type.sub << 20;
+	 * sse_evt[1][95:64]  = qes[3].flow_id |
+	 *			qes[3].u.event_type.major << 28 |
+	 *			qes[3].u.event_type.sub << 20;
+	 */
+#define DLB_EVENT_EV_TYPE_DW 0
+#define DLB_EVENT_EV_TYPE_SHIFT 28
+#define DLB_EVENT_SUB_EV_TYPE_SHIFT 20
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[0].flow_id |
+			qes[0].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[0].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[1].flow_id |
+			qes[1].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[1].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[2].flow_id |
+			qes[2].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[2].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[3].flow_id |
+			qes[3].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT  |
+			qes[3].u.event_type.sub << DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+
+	/* Write the sched type to the event metadata. 'op' and 'rsvd' are not
+	 * set:
+	 * sse_evt[0][39:32]  = sched_type_map[qes[0].sched_type] << 6
+	 * sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6
+	 * sse_evt[1][39:32]  = sched_type_map[qes[2].sched_type] << 6
+	 * sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6
+	 */
+#define DLB_EVENT_SCHED_TYPE_BYTE 4
+#define DLB_EVENT_SCHED_TYPE_SHIFT 6
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[0].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[1].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[2].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[3].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+
+	/* Store the metadata to the event (use the double-precision
+	 * _mm_storeh_pd because there is no integer function for storing the
+	 * upper 64b):
+	 * events[0].event = sse_evt[0][63:0]
+	 * events[1].event = sse_evt[0][127:64]
+	 * events[2].event = sse_evt[1][63:0]
+	 * events[3].event = sse_evt[1][127:64]
+	 */
+	_mm_storel_epi64((__m128i *)&events[0].event, sse_evt[0]);
+	_mm_storeh_pd((double *)&events[1].event, (__m128d) sse_evt[0]);
+	_mm_storel_epi64((__m128i *)&events[2].event, sse_evt[1]);
+	_mm_storeh_pd((double *)&events[3].event, (__m128d) sse_evt[1]);
+
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[0].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[1].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[2].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[3].sched_type], 1);
+
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[0].queue_id].
+			qid_depth[qes[0].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[1].queue_id].
+			qid_depth[qes[1].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[2].queue_id].
+			qid_depth[qes[2].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[3].queue_id].
+			qid_depth[qes[3].qid_depth],
+		1);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num_events);
+
+	return num_events;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe_sparse(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe)
+{
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	uint8_t xor_mask[2] = {0x0F, 0x00};
+	const uint8_t and_mask = 0x0F;
+	__m128i *qes = (__m128i *)qe;
+	uint8_t gen_bits, gen_bit;
+	uintptr_t addr[4];
+	uint16_t idx;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+
+	idx = qm_port->cq_idx;
+
+	/* Load the next 4 QEs */
+	addr[0] = (uintptr_t)&cq_addr[idx];
+	addr[1] = (uintptr_t)&cq_addr[(idx +  4) & qm_port->cq_depth_mask];
+	addr[2] = (uintptr_t)&cq_addr[(idx +  8) & qm_port->cq_depth_mask];
+	addr[3] = (uintptr_t)&cq_addr[(idx + 12) & qm_port->cq_depth_mask];
+
+	/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */
+	rte_prefetch0(&cq_addr[(idx + 16) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 20) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 24) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 28) & qm_port->cq_depth_mask]);
+
+	/* Correct the xor_mask for wrap-around QEs */
+	gen_bit = qm_port->gen_bit;
+	xor_mask[gen_bit] ^= !!((idx +  4) > qm_port->cq_depth_mask) << 1;
+	xor_mask[gen_bit] ^= !!((idx +  8) > qm_port->cq_depth_mask) << 2;
+	xor_mask[gen_bit] ^= !!((idx + 12) > qm_port->cq_depth_mask) << 3;
+
+	/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is
+	 * valid, then QEs[0:N-1] are too.
+	 */
+	qes[3] = _mm_load_si128((__m128i *)(void *)addr[3]);
+	rte_compiler_barrier();
+	qes[2] = _mm_load_si128((__m128i *)(void *)addr[2]);
+	rte_compiler_barrier();
+	qes[1] = _mm_load_si128((__m128i *)(void *)addr[1]);
+	rte_compiler_barrier();
+	qes[0] = _mm_load_si128((__m128i *)(void *)addr[0]);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[gen_bit];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask;
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline void
+dlb2_inc_cq_idx(struct dlb2_port *qm_port, int cnt)
+{
+	uint16_t idx = qm_port->cq_idx_unmasked + cnt;
+
+	qm_port->cq_idx_unmasked = idx;
+	qm_port->cq_idx = idx & qm_port->cq_depth_mask;
+	qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) & 0x1;
+}
+
+static int
+dlb2_event_release(struct dlb2_eventdev *dlb2,
+		   uint8_t port_id,
+		   int n)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_eventdev_port *ev_port;
+	struct dlb2_port *qm_port;
+	int i, cnt;
+
+	if (port_id > dlb2->num_ports) {
+		DLB2_LOG_ERR("Invalid port id %d in dlb2-event_release\n",
+			     port_id);
+		rte_errno = -EINVAL;
+		return rte_errno;
+	}
+
+	ev_port = &dlb2->ev_ports[port_id];
+	qm_port = &ev_port->qm_port;
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	cnt = 0;
+
+	if (qm_port->is_directed) {
+		cnt = n;
+		goto sw_credit_update;
+	}
+
+	for (i = 0; i < n; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		int j;
+
+		/* Zero-out QEs */
+		qm_port->qe4[0].cmd_byte = 0;
+		qm_port->qe4[1].cmd_byte = 0;
+		qm_port->qe4[2].cmd_byte = 0;
+		qm_port->qe4[3].cmd_byte = 0;
+
+		for (j = 0; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++)
+			qm_port->qe4[j].cmd_byte = DLB2_COMP_CMD_BYTE;
+
+		qm_port->issued_releases += j;
+
+		if (j == 0)
+			break;
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+	}
+
+sw_credit_update:
+	/* each release returns one credit */
+	if (!ev_port->outstanding_releases) {
+		DLB2_LOG_ERR("Unrecoverable application error. Outstanding releases underflowed.\n");
+		rte_errno = -ENOTRECOVERABLE;
+		return rte_errno;
+	}
+
+	ev_port->outstanding_releases -= cnt;
+	ev_port->inflight_credits += cnt;
+
+	/* Replenish s/w credits if enough releases are performed */
+	dlb2_replenish_sw_credits(dlb2, ev_port);
+	return 0;
+}
+
+static inline int16_t
+dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
+		       struct dlb2_eventdev_port *ev_port,
+		       struct rte_event *events,
+		       uint16_t max_num,
+		       uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe_sparse(qm_port, qes);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail << 2);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							      qm_port,
+							      &events[num],
+							      &qes[0]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							 qm_port,
+							 &events[num],
+							 &qes[0],
+							 num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe,
+	     uint8_t *offset)
+{
+	uint8_t xor_mask[2][4] = { {0x0F, 0x0E, 0x0C, 0x08},
+				   {0x00, 0x01, 0x03, 0x07} };
+	uint8_t and_mask[4] = {0x0F, 0x0E, 0x0C, 0x08};
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	__m128i *qes = (__m128i *)qe;
+	uint64_t *cache_line_base;
+	uint8_t gen_bits;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	cq_addr = &cq_addr[qm_port->cq_idx];
+
+	cache_line_base = (void *)(((uintptr_t)cq_addr) & ~0x3F);
+	*offset = ((uintptr_t)cq_addr & 0x30) >> 4;
+
+	/* Load the next CQ cache line from memory. Pack these reads as tight
+	 * as possible to reduce the chance that DLB invalidates the line while
+	 * the CPU is reading it. Read the cache line backwards to ensure that
+	 * if QE[N] (N > 0) is valid, then QEs[0:N-1] are too.
+	 *
+	 * (Valid QEs start at &qe[offset])
+	 */
+	qes[3] = _mm_load_si128((__m128i *)&cache_line_base[6]);
+	qes[2] = _mm_load_si128((__m128i *)&cache_line_base[4]);
+	qes[1] = _mm_load_si128((__m128i *)&cache_line_base[2]);
+	qes[0] = _mm_load_si128((__m128i *)&cache_line_base[0]);
+
+	/* Evict the cache line ASAP */
+	rte_cldemote(cache_line_base);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[qm_port->gen_bit][*offset];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask[*offset];
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline int16_t
+dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
+		struct dlb2_eventdev_port *ev_port,
+		struct rte_event *events,
+		uint16_t max_num,
+		uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t offset;
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe(qm_port, qes, &offset);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							     qm_port,
+							     &events[num],
+							     &qes[offset]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							qm_port,
+							&events[num],
+							&qes[offset],
+							num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static uint16_t
+dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
+			 uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue(void *event_port, struct rte_event *ev, uint64_t wait)
+{
+	return dlb2_event_dequeue_burst(event_port, ev, 1, wait);
+}
+
+static uint16_t
+dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
+				uint16_t num, uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
+			  uint64_t wait)
+{
+	return dlb2_event_dequeue_burst_sparse(event_port, ev, 1, wait);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
+	struct dlb2_eventdev *dlb2;
+
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
@@ -2653,6 +3427,15 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	dev->enqueue_burst = dlb2_event_enqueue_burst;
 	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
 	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
+
+	dlb2 = dev->data->dev_private;
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE) {
+		dev->dequeue = dlb2_event_dequeue_sparse;
+		dev->dequeue_burst = dlb2_event_dequeue_burst_sparse;
+	} else {
+		dev->dequeue = dlb2_event_dequeue;
+		dev->dequeue_burst = dlb2_event_dequeue_burst;
+	}
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v10 19/23] event/dlb2: add eventdev stop and close

[Timothy McDaniel]

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 257 +++++++++++++++++++++++++++--
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c |  91 ++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  52 ++++++
 5 files changed, 396 insertions(+), 16 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 631fe52..4b8f45f 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -71,21 +71,6 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
-/*
- * DUMMY - added so that xstats path will compile/link.
- * Will be replaced by real version in a subsequent
- * patch.
- */
-uint32_t
-dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
-		     struct dlb2_eventdev_queue *queue)
-{
-	RTE_SET_USED(dlb2);
-	RTE_SET_USED(queue);
-
-	return 0;
-}
-
 static void
 dlb2_free_qe_mem(struct dlb2_port *qm_port)
 {
@@ -1878,7 +1863,6 @@ dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
 		return 0; /* Ignore and return success */
 	}
 
-	/* FIXME: How to handle unlink on a directed port? */
 	if (ev_port->qm_port.is_directed) {
 		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
 			     ev_port->id);
@@ -3396,6 +3380,245 @@ dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
 }
 
 static void
+dlb2_flush_port(struct rte_eventdev *dev, int port_id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	eventdev_stop_flush_t flush;
+	struct rte_event ev;
+	uint8_t dev_id;
+	void *arg;
+	int i;
+
+	flush = dev->dev_ops->dev_stop_flush;
+	dev_id = dev->data->dev_id;
+	arg = dev->data->dev_stop_flush_arg;
+
+	while (rte_event_dequeue_burst(dev_id, port_id, &ev, 1, 0)) {
+		if (flush)
+			flush(dev_id, ev, arg);
+
+		if (dlb2->ev_ports[port_id].qm_port.is_directed)
+			continue;
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+	}
+
+	/* Enqueue any additional outstanding releases */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = dlb2->ev_ports[port_id].outstanding_releases; i > 0; i--)
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+}
+
+static uint32_t
+dlb2_get_ldb_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_ldb_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_ldb_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_ldb_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static uint32_t
+dlb2_get_dir_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_dir_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_dir_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_dir_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	if (queue->qm_queue.is_directed)
+		return dlb2_get_dir_queue_depth(dlb2, queue);
+	else
+		return dlb2_get_ldb_queue_depth(dlb2, queue);
+}
+
+static bool
+dlb2_queue_is_empty(struct dlb2_eventdev *dlb2,
+		    struct dlb2_eventdev_queue *queue)
+{
+	return dlb2_get_queue_depth(dlb2, queue) == 0;
+}
+
+static bool
+dlb2_linked_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0)
+			continue;
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static bool
+dlb2_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static void
+dlb2_drain(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_port *ev_port = NULL;
+	uint8_t dev_id;
+	int i;
+
+	dev_id = dev->data->dev_id;
+
+	while (!dlb2_linked_queues_empty(dlb2)) {
+		/* Flush all the ev_ports, which will drain all their connected
+		 * queues.
+		 */
+		for (i = 0; i < dlb2->num_ports; i++)
+			dlb2_flush_port(dev, i);
+	}
+
+	/* The queues are empty, but there may be events left in the ports. */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_flush_port(dev, i);
+
+	/* If the domain's queues are empty, we're done. */
+	if (dlb2_queues_empty(dlb2))
+		return;
+
+	/* Else, there must be at least one unlinked load-balanced queue.
+	 * Select a load-balanced port with which to drain the unlinked
+	 * queue(s).
+	 */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		if (!ev_port->qm_port.is_directed)
+			break;
+	}
+
+	if (i == dlb2->num_ports) {
+		DLB2_LOG_ERR("internal error: no LDB ev_ports\n");
+		return;
+	}
+
+	rte_errno = 0;
+	rte_event_port_unlink(dev_id, ev_port->id, NULL, 0);
+
+	if (rte_errno) {
+		DLB2_LOG_ERR("internal error: failed to unlink ev_port %d\n",
+			     ev_port->id);
+		return;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		uint8_t qid, prio;
+		int ret;
+
+		if (dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			continue;
+
+		qid = i;
+		prio = 0;
+
+		/* Link the ev_port to the queue */
+		ret = rte_event_port_link(dev_id, ev_port->id, &qid, &prio, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to link ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+
+		/* Flush the queue */
+		while (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			dlb2_flush_port(dev, ev_port->id);
+
+		/* Drain any extant events in the ev_port. */
+		dlb2_flush_port(dev, ev_port->id);
+
+		/* Unlink the ev_port from the queue */
+		ret = rte_event_port_unlink(dev_id, ev_port->id, &qid, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to unlink ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+	}
+}
+
+static void
+dlb2_eventdev_stop(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_DBG("Internal error: already stopped\n");
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	} else if (dlb2->run_state != DLB2_RUN_STATE_STARTED) {
+		DLB2_LOG_ERR("Internal error: bad state %d for dev_stop\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPING;
+
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	dlb2_drain(dev);
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPED;
+}
+
+static int
+dlb2_eventdev_close(struct rte_eventdev *dev)
+{
+	dlb2_hw_reset_sched_domain(dev, false);
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3405,6 +3628,8 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.dev_start        = dlb2_eventdev_start,
+		.dev_stop         = dlb2_eventdev_stop,
+		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a86191d..5471dd8 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -66,3 +66,9 @@ int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				     struct dlb2_start_domain_args *cfg);
+
+int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 7ead374..b508eb0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -65,4 +65,10 @@ extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				struct dlb2_start_domain_args *cfg);
+
+extern int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+extern int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index c14a2f3..ae5ef2f 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5934,3 +5934,94 @@ dlb2_hw_start_domain(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_dir_queue_depth(hw, queue);
+
+	return 0;
+}
+
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+
+	dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index fff2e57..632c4e0 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -527,6 +527,56 @@ dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_ldb_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_ldb_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_dir_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret = 0;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_dir_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -544,6 +594,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth;
+	dlb2_iface_get_dir_queue_depth = dlb2_pf_get_dir_queue_depth;
 	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v10 20/23] event/dlb2: add PMD's token pop public interface

[Timothy McDaniel]

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/api/doxy-api-index.md         |  3 +-
 doc/api/doxy-api.conf.in          |  1 +
 drivers/event/dlb2/dlb2.c         | 65 +++++++++++++++++++++++++++++++----
 drivers/event/dlb2/dlb2_priv.h    |  3 ++
 drivers/event/dlb2/meson.build    |  6 ++--
 drivers/event/dlb2/rte_pmd_dlb2.c | 39 +++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2.h | 72 +++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/version.map    |  6 ++++
 8 files changed, 186 insertions(+), 9 deletions(-)
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index b865a51..e1836dc 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -52,7 +52,8 @@ The public API headers are grouped by topics:
   [dpaa2_mempool]      (@ref rte_dpaa2_mempool.h),
   [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
-  [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
+  [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h),
+  [dlb2]               (@ref rte_pmd_dlb2.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index c5b01a1..535321f 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -7,6 +7,7 @@ USE_MDFILE_AS_MAINPAGE  = @TOPDIR@/doc/api/doxy-api-index.md
 INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/drivers/bus/vdev \
                           @TOPDIR@/drivers/crypto/scheduler \
+                          @TOPDIR@/drivers/event/dlb2 \
                           @TOPDIR@/drivers/mempool/dpaa2 \
                           @TOPDIR@/drivers/net/ark \
                           @TOPDIR@/drivers/net/bnxt \
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 4b8f45f..e8b4446 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1197,7 +1197,7 @@ dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
 	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
 
 	qm_port->dequeue_depth = dequeue_depth;
-
+	qm_port->token_pop_thresh = dequeue_depth;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -1365,6 +1365,8 @@ dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
 
 	qm_port->dequeue_depth = dequeue_depth;
 
+	/* Directed ports are auto-pop, by default. */
+	qm_port->token_pop_mode = AUTO_POP;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -2253,6 +2255,18 @@ dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
 }
 
 static inline void
+dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
+{
+	struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
+	int num = qm_port->owed_tokens;
+
+	qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
+	qe[idx].tokens = num - 1;
+
+	qm_port->owed_tokens = 0;
+}
+
+static inline void
 dlb2_event_build_hcws(struct dlb2_port *qm_port,
 		      const struct rte_event ev[],
 		      int num,
@@ -2600,6 +2614,14 @@ dlb2_event_enqueue_burst(void *event_port,
 		dlb2_event_build_hcws(qm_port, &events[i], j,
 				      sched_types, queue_ids);
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
@@ -2608,6 +2630,11 @@ dlb2_event_enqueue_burst(void *event_port,
 			break;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
 	return cnt;
 }
 
@@ -3085,11 +3112,25 @@ dlb2_event_release(struct dlb2_eventdev *dlb2,
 		if (j == 0)
 			break;
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
 sw_credit_update:
 	/* each release returns one credit */
 	if (!ev_port->outstanding_releases) {
@@ -3173,8 +3214,8 @@ dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3300,8 +3341,8 @@ dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3316,6 +3357,7 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 			 uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3331,6 +3373,9 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3350,6 +3395,7 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 				uint16_t num, uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3365,6 +3411,9 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3669,7 +3718,7 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    struct dlb2_devargs *dlb2_args)
 {
 	struct dlb2_eventdev *dlb2;
-	int err;
+	int err, i;
 
 	dlb2 = dev->data->dev_private;
 
@@ -3719,6 +3768,10 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Initialize each port's token pop mode */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
+		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 61567a6..b73cf3f 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -12,6 +12,7 @@
 #include <rte_config.h>
 #include "dlb2_user.h"
 #include "dlb2_log.h"
+#include "rte_pmd_dlb2.h"
 
 #ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
 #define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
@@ -290,6 +291,7 @@ struct dlb2_port {
 	bool gen_bit;
 	uint16_t dir_credits;
 	uint32_t dequeue_depth;
+	enum dlb2_token_pop_mode token_pop_mode;
 	union dlb2_port_config cfg;
 	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
 	uint16_t cached_ldb_credits;
@@ -298,6 +300,7 @@ struct dlb2_port {
 	bool int_armed;
 	uint16_t owed_tokens;
 	int16_t issued_releases;
+	int16_t token_pop_thresh;
 	int cq_depth;
 	uint16_t cq_idx;
 	uint16_t cq_idx_unmasked;
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index a80d7ab..2af72e2 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,3 +1,4 @@
+
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
@@ -12,9 +13,10 @@ sources = files('dlb2.c',
 		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
-		'pf/base/dlb2_resource.c'
+		'pf/base/dlb2_resource.c',
+		'rte_pmd_dlb2.c'
 )
 
-headers = files()
+headers = files('rte_pmd_dlb2.h')
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
new file mode 100644
index 0000000..b09b585
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "rte_pmd_dlb2.h"
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode)
+{
+	struct dlb2_eventdev *dlb2;
+	struct rte_eventdev *dev;
+
+	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+	dev = &rte_eventdevs[dev_id];
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (mode >= NUM_TOKEN_POP_MODES)
+		return -EINVAL;
+
+	/* The event device must be configured, but not yet started */
+	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
+		return -EINVAL;
+
+	/* The token pop mode must be set before configuring the port */
+	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
+		return -EINVAL;
+
+	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
new file mode 100644
index 0000000..74399db
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+/*!
+ *  @file      rte_pmd_dlb2.h
+ *
+ *  @brief     DLB PMD-specific functions
+ */
+
+#ifndef _RTE_PMD_DLB2_H_
+#define _RTE_PMD_DLB2_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Selects the token pop mode for a DLB2 port.
+ */
+enum dlb2_token_pop_mode {
+	/* Pop the CQ tokens immediately after dequeueing. */
+	AUTO_POP,
+	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
+	 * Supported on load-balanced ports only.
+	 */
+	DELAYED_POP,
+	/* Pop the CQ tokens during next dequeue operation. */
+	DEFERRED_POP,
+
+	/* NUM_TOKEN_POP_MODES must be last */
+	NUM_TOKEN_POP_MODES
+};
+
+/*!
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Configure the token pop mode for a DLB2 port. By default, all ports use
+ * AUTO_POP. This function must be called before calling rte_event_port_setup()
+ * for the port, but after calling rte_event_dev_configure().
+ *
+ * @param dev_id
+ *    The identifier of the event device.
+ * @param port_id
+ *    The identifier of the event port.
+ * @param mode
+ *    The token pop mode.
+ *
+ * @return
+ * - 0: Success
+ * - EINVAL: Invalid dev_id, port_id, or mode
+ * - EINVAL: The DLB2 is not configured, is already running, or the port is
+ *   already setup
+ */
+
+__rte_experimental
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_DLB2_H_ */
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
index 4a76d1d..b1e4dff 100644
--- a/drivers/event/dlb2/version.map
+++ b/drivers/event/dlb2/version.map
@@ -1,3 +1,9 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmd_dlb2_set_token_pop_mode;
+};
-- 
2.6.4

[dpdk-dev] [PATCH v10 21/23] event/dlb2: add PMD self-tests

[Timothy McDaniel]

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 app/test/test_eventdev.c           |    7 +
 drivers/event/dlb2/dlb2.c          |    1 +
 drivers/event/dlb2/dlb2_selftest.c | 1524 ++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build     |    3 +-
 4 files changed, 1534 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c

diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
index 62019c1..bcfaa53 100644
--- a/app/test/test_eventdev.c
+++ b/app/test/test_eventdev.c
@@ -1030,6 +1030,12 @@ test_eventdev_selftest_dpaa2(void)
 	return test_eventdev_selftest_impl("event_dpaa2", "");
 }
 
+static int
+test_eventdev_selftest_dlb2(void)
+{
+	return test_eventdev_selftest_impl("dlb2_event", "");
+}
+
 REGISTER_TEST_COMMAND(eventdev_common_autotest, test_eventdev_common);
 REGISTER_TEST_COMMAND(eventdev_selftest_sw, test_eventdev_selftest_sw);
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
@@ -1037,3 +1043,4 @@ REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
 		test_eventdev_selftest_octeontx2);
 REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2, test_eventdev_selftest_dpaa2);
+REGISTER_TEST_COMMAND(eventdev_selftest_dlb2, test_eventdev_selftest_dlb2);
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index e8b4446..86299dd 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3692,6 +3692,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
 		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+		.dev_selftest     = test_dlb2_eventdev,
 	};
 
 	/* Expose PMD's eventdev interface */
diff --git a/drivers/event/dlb2/dlb2_selftest.c b/drivers/event/dlb2/dlb2_selftest.c
new file mode 100644
index 0000000..e300067
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_selftest.c
@@ -0,0 +1,1524 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <unistd.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_eventdev.h>
+#include <rte_pause.h>
+
+#include "dlb2_priv.h"
+#include "rte_pmd_dlb2.h"
+
+#define MAX_PORTS 32
+#define MAX_QIDS 32
+#define DEFAULT_NUM_SEQ_NUMS 64
+
+static struct rte_mempool *eventdev_func_mempool;
+static int evdev;
+
+struct test {
+	struct rte_mempool *mbuf_pool;
+	int nb_qids;
+};
+
+/* initialization and config */
+static inline int
+init(struct test *t, int nb_queues, int nb_ports)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	memset(t, 0, sizeof(*t));
+
+	t->mbuf_pool = eventdev_func_mempool;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = nb_queues;
+	config.nb_event_ports = nb_ports;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0)
+		printf("%d: Error configuring device\n", __LINE__);
+
+	return ret;
+}
+
+static inline int
+create_ports(int num_ports)
+{
+	int i;
+
+	if (num_ports > MAX_PORTS)
+		return -1;
+
+	for (i = 0; i < num_ports; i++) {
+		struct rte_event_port_conf conf;
+
+		if (rte_event_port_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default port conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		if (rte_event_port_setup(evdev, i, &conf) < 0) {
+			printf("%d: Error setting up port %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+create_lb_qids(struct test *t, int num_qids, uint32_t flags)
+{
+	int i;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		struct rte_event_queue_conf conf;
+
+		if (rte_event_queue_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default queue conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		conf.schedule_type = flags;
+
+		if (conf.schedule_type == RTE_SCHED_TYPE_PARALLEL)
+			conf.nb_atomic_order_sequences = 0;
+		else
+			conf.nb_atomic_order_sequences = DEFAULT_NUM_SEQ_NUMS;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+static inline int
+create_atomic_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ATOMIC);
+}
+
+/* destruction */
+static inline void
+cleanup(void)
+{
+	int ret = 0;
+
+	rte_event_dev_stop(evdev);
+	ret = rte_event_dev_close(evdev);
+
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+};
+
+static inline int
+enqueue_timeout(uint8_t port_id, struct rte_event *ev, uint64_t tmo_us)
+{
+	const uint64_t start = rte_get_timer_cycles();
+	const uint64_t ticks = (tmo_us * rte_get_timer_hz()) / 1E6;
+
+	while ((rte_get_timer_cycles() - start) < ticks) {
+		if (rte_event_enqueue_burst(evdev, port_id, ev, 1) == 1)
+			return 0;
+
+		if (rte_errno != -ENOSPC) {
+			printf("enqueue_burst returned rte_errno %d\n",
+			       rte_errno);
+			return -1;
+		}
+	}
+	printf("%s time out\n", __func__);
+	return -1;
+}
+
+static void
+flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused)
+{
+	rte_pktmbuf_free(event.mbuf);
+}
+
+static int
+test_stop_flush(struct test *t) /* test to check we can properly flush events */
+{
+	struct rte_event ev;
+	uint32_t dequeue_depth;
+	unsigned int i, count;
+	uint8_t queue_id;
+
+	ev.op = RTE_EVENT_OP_NEW;
+
+	if (init(t, 2, 1) < 0 ||
+	    create_ports(1) < 0 ||
+	    create_atomic_qids(t, 2) < 0) {
+		printf("%d: Error initializing device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_port_link(evdev, 0, NULL, NULL, 0) != 2) {
+		printf("%d: Error linking queues to the port\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: Error with start call\n", __LINE__);
+		goto err;
+	}
+
+	/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked
+	 * queue.
+	 */
+	queue_id = 1;
+
+	if (rte_event_port_unlink(evdev, 0, &queue_id, 1) != 1) {
+		printf("%d: Error unlinking queue 1 from port\n", __LINE__);
+		goto err;
+	}
+
+	count = rte_mempool_avail_count(t->mbuf_pool);
+
+	if (rte_event_port_attr_get(evdev,
+				    0,
+				    RTE_EVENT_PORT_ATTR_DEQ_DEPTH,
+				    &dequeue_depth)) {
+		printf("%d: Error retrieveing dequeue depth\n", __LINE__);
+		goto err;
+	}
+
+	/* Send QEs to queue 0 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 0;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Send QEs to queue 1 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 1;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Now the DLB is scheduling events from the port to the IQ, and at
+	 * least one event should be remaining in each queue.
+	 */
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, flush, NULL)) {
+		printf("%d: Error installing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+
+	if (count != rte_mempool_avail_count(t->mbuf_pool)) {
+		printf("%d: Error executing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, NULL, NULL)) {
+		printf("%d: Error uninstalling the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	return 0;
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_single_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Create a directed port */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) == 0) {
+		printf("%d: port 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Create a load-balanced port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a directed queue */
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed queue */
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) == 0) {
+		printf("%d: queue 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a load-balanced queue */
+	queue_conf.event_queue_cfg = 0;
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to link directed and load-balanced resources */
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 0 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 1 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Link ports to queues */
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+#define NUM_LDB_PORTS 64
+#define NUM_LDB_QUEUES 32
+
+static int
+test_info_get(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	config.nb_event_ports = info.max_event_ports;
+	config.nb_event_queues = NUM_LDB_QUEUES + info.max_event_ports / 2;
+	config.nb_single_link_event_port_queues = info.max_event_ports / 2;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		goto err;
+	}
+
+	/* The DLB2 PMD only reports load-balanced ports and queues in its
+	 * info_get function. Confirm that these values don't include the
+	 * directed port or queue counts.
+	 */
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret) {
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+		return -1;
+	}
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+static int
+test_reconfiguration_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 2 LDB queues */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+			printf("%d: P0->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P1->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+			printf("%d: P1->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Configure device with 2 DIR ports and 2 DIR queues */
+	config.nb_single_link_event_port_queues = 2;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q1 */
+	queue_id = 1;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: P0->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Link P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: P1->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_queues = 5;
+	config.nb_event_ports = 5;
+	config.nb_single_link_event_port_queues = 1;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < config.nb_event_queues - 1; i++) {
+		port_conf.event_port_cfg = 0;
+		queue_conf.event_queue_cfg = 0;
+
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: P%d->Q%d link expected to succeed\n",
+			       __LINE__, i, i);
+			goto err;
+		}
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+		printf("%d: port %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+		printf("%d: queue %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	queue_id = i;
+
+	if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+		printf("%d: P%d->Q%d link expected to succeed\n",
+		       __LINE__, i, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_ports += 1;
+
+	/* Reconfigure device with 1 more load-balanced port */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Configure the new port */
+	if (rte_event_port_setup(evdev, config.nb_event_ports - 1,
+				 &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_load_balanced_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_directed_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 DIR port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (ev.queue_id != 0) {
+		printf("%d: invalid dequeued event queue ID (%d)\n",
+		       __LINE__, ev.queue_id);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_deferred_sched(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	const int num_events = 128;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 1 queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 1, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.dequeue_depth = 1;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.schedule_type = RTE_SCHED_TYPE_PARALLEL;
+	queue_conf.nb_atomic_order_sequences = 0;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 128 NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_PARALLEL;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue one event from port 0 */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue (and release) all other events from port 1. Deferred
+	 * scheduling ensures no other events are scheduled to port 0 without a
+	 * subsequent rte_event_dequeue_burst() call.
+	 */
+	for (i = 0; i < num_events - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 1, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		if (rte_event_enqueue_burst(evdev, 1, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_delayed_pop(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	int ret, i, num_events;
+	struct rte_event ev;
+	uint8_t queue_id;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DELAYED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	num_events = 2 * port_conf.dequeue_depth;
+
+	/* Enqueue 2 * dequeue_depth NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue dequeue_depth events but only release dequeue_depth - 2.
+	 * Delayed pop won't perform the pop and no more events will be
+	 * scheduled.
+	 */
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = 0; i < port_conf.dequeue_depth - 2; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	ret = rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout);
+	if (ret != 0) {
+		printf("%d: event dequeue expected to fail (ret = %d)\n",
+		       __LINE__, ret);
+		goto err;
+	}
+
+	/* Release one more event. This will trigger the token pop, and
+	 * dequeue_depth - 1 more events will be scheduled to the device.
+	 */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: RELEASE enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 0) {
+		printf("%d: event dequeue expected to fail\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+do_selftest(void)
+{
+	struct test t;
+	int ret;
+
+	/* Only create mbuf pool once, reuse for each test run */
+	if (!eventdev_func_mempool) {
+		eventdev_func_mempool =
+			rte_pktmbuf_pool_create("EVENTDEV_DLB2_ST_POOL",
+						(1 << 12), /* 4k buffers */
+						32 /*MBUF_CACHE_SIZE*/,
+						0,
+						512, /* use very small mbufs */
+						rte_socket_id());
+		if (!eventdev_func_mempool) {
+			printf("ERROR creating mempool\n");
+			goto test_fail;
+		}
+	}
+	t.mbuf_pool = eventdev_func_mempool;
+
+	printf("*** Running Stop Flush test...\n");
+	ret = test_stop_flush(&t);
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Single Link test...\n");
+	ret = test_single_link();
+	if (ret != 0) {
+		printf("ERROR - Single Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Info Get test...\n");
+	ret = test_info_get();
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Reconfiguration Link test...\n");
+	ret = test_reconfiguration_link();
+	if (ret != 0) {
+		printf("ERROR - Reconfiguration Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Load-Balanced Traffic test...\n");
+	ret = test_load_balanced_traffic();
+	if (ret != 0) {
+		printf("ERROR - Load-Balanced Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Directed Traffic test...\n");
+	ret = test_directed_traffic();
+	if (ret != 0) {
+		printf("ERROR - Directed Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Deferred Scheduling test...\n");
+	ret = test_deferred_sched();
+	if (ret != 0) {
+		printf("ERROR - Deferred Scheduling test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Delayed Pop test...\n");
+	ret = test_delayed_pop();
+	if (ret != 0) {
+		printf("ERROR - Delayed Pop test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	return 0;
+
+test_fail:
+	return -1;
+}
+
+int
+test_dlb2_eventdev(void)
+{
+	const char *dlb2_eventdev_name = "dlb2_event";
+	uint8_t num_evdevs = rte_event_dev_count();
+	int i, ret = 0;
+	int found = 0, skipped = 0, passed = 0, failed = 0;
+	struct rte_event_dev_info info;
+
+	for (i = 0; found + skipped < num_evdevs && i < RTE_EVENT_MAX_DEVS;
+	     i++) {
+		ret = rte_event_dev_info_get(i, &info);
+		if (ret < 0)
+			continue;
+
+		/* skip non-dlb2 event devices */
+		if (strncmp(info.driver_name, dlb2_eventdev_name,
+			    sizeof(*info.driver_name)) != 0) {
+			skipped++;
+			continue;
+		}
+
+		evdev = rte_event_dev_get_dev_id(info.driver_name);
+		if (evdev < 0) {
+			printf("Could not get dev_id for eventdev with name %s, i=%d\n",
+			       info.driver_name, i);
+			skipped++;
+			continue;
+		}
+		found++;
+		printf("Running selftest on eventdev %s\n", info.driver_name);
+		ret = do_selftest();
+		if (ret == 0) {
+			passed++;
+			printf("Selftest passed for eventdev %s\n",
+			       info.driver_name);
+		} else {
+			failed++;
+			printf("Selftest failed for eventdev %s, err=%d\n",
+			       info.driver_name, ret);
+		}
+	}
+
+	printf("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n",
+	       found, skipped, passed, failed);
+	return ret;
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 2af72e2..bcda3dd 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -14,7 +14,8 @@ sources = files('dlb2.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c',
-		'rte_pmd_dlb2.c'
+		'rte_pmd_dlb2.c',
+		'dlb2_selftest.c'
 )
 
 headers = files('rte_pmd_dlb2.h')
-- 
2.6.4

[dpdk-dev] [PATCH v10 22/23] event/dlb2: add queue and port release

[Timothy McDaniel]

DLB does not support reconfiguring individual queues
or ports on the fly. The entire device must be reconfigured.
Previously allocated port QE and memzone memory
is freed in this patch.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 86299dd..248296d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3668,6 +3668,28 @@ dlb2_eventdev_close(struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(id);
+
+	/* This function intentionally left blank. */
+}
+
+static void
+dlb2_eventdev_port_release(void *port)
+{
+	struct dlb2_eventdev_port *ev_port = port;
+	struct dlb2_port *qm_port;
+
+	if (ev_port) {
+		qm_port = &ev_port->qm_port;
+		if (qm_port->config_state == DLB2_CONFIGURED)
+			dlb2_free_qe_mem(qm_port);
+	}
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3681,8 +3703,10 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
+		.queue_release    = dlb2_eventdev_queue_release,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_release     = dlb2_eventdev_port_release,
 		.port_link        = dlb2_eventdev_port_link,
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
-- 
2.6.4

[dpdk-dev] [PATCH v10 23/23] event/dlb2: add timeout ticks entry point

[Timothy McDaniel]

Adds the timeout ticks conversion function.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 248296d..d42e48b 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3689,6 +3689,18 @@ dlb2_eventdev_port_release(void *port)
 	}
 }
 
+static int
+dlb2_eventdev_timeout_ticks(struct rte_eventdev *dev, uint64_t ns,
+			    uint64_t *timeout_ticks)
+{
+	RTE_SET_USED(dev);
+	uint64_t cycles_per_ns = rte_get_timer_hz() / 1E9;
+
+	*timeout_ticks = ns * cycles_per_ns;
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -3711,6 +3723,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
 				    dlb2_eventdev_port_unlinks_in_progress,
+		.timeout_ticks    = dlb2_eventdev_timeout_ticks,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v11 00/23] Add DLB2 PMD

[Timothy McDaniel]

The following patch series adds support for a new eventdev PMD. The DLB2
PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
hardware.
The DLB2 is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB2 hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB2 ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a
directed queue. Additionally, even though LDB ports may link multiple
queues,
the number of queues that may be linked is limited by hardware.

While reviewing the code, please be aware that this PMD has full
control over the DLB2 hardware. Intel will be extending the DLB2 PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.
Note that the DLB2 hardware is a successor of the DLB hardware, and
as such is structured similarly, both in terms of code layout and
implementation.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Changes since V10
=================
- converted printfs in dlb2/pf/dlb2_main.c to DLB2_LOG
- fixed a repeated word error in dlb2/pf/base/osdep_bitmap.h
- caught up with marking the patches that Gage reviewed

Changes since V9
================
Address comments from David Marchand:
- this patch-set is based on Nov 1, 2020 dpdk-next-eventdev
- fix docs build (doxy-api.conf.in and doxy-api-index.md)
- restore blank line in MAINTAINERS file
- move dlb announcement in release_20_11.rst after ethdev
- use headers = files() for exported meson public headers
- fix a typo in 'add documentation ..." commit message
- use eal version of cldemote
- convert a couple of printfs to LOG messages
- all patches build incrementally (gcc), and checkpatches reports
  success
- I am not able to run clang locally. If clang errors are still
  present I will ask IT to install clang on a build server tomorrow.

Changes since V8:
=================
- Fixed format errors in doc/api/doxy-api-index.md
- Delayed introduction of dlb2_consume_qe_immediate until
  add-dequeue-and-its-burst-variants.patch
- Delayed introduction of dlb2_construct_token_pop_qe until 
  add-PMD-s-token-pop-public-interface.patch

Changes since V7:
================
- fix CENTOS build error: use __m128i instead of __v2di with
  _mm_stream_si128
- deleted unused dlb2_umwait and dlb2_umonitor functions

Changes since V6:
=================
- removed unused function, fixing build error
- fixed typo in port_setup commit message
- this patch series is based on dpdk-next-eventdev

Changes since V5:
================
- convert to use rte_power_monitor patches
- replace __builtin_ia32_movntdq() with _mm_stream_si128()
- remove unused functions in dlb_selftest.c
- workaround for RHEL 7 gcc brace bug

Changes since V4:
================
- moved introduction of dlb in relnotes_20_11 to first patch in series
- fixed underlines in dlb.rst that were too short
- note that the code still uses its private byte-encoded versions of
  umonitor/umwait, rather than the new functions in the power
  patch that are built on top of those intrinsics. This is intentional.

Changes since V3:
================
- updated MAINTAINERS file to alphabetically insert DLB
- don't create RTE_ symbols in pmd
- converted to use version.map scheme
- converted to use .._master_lcore instead of .._main_lcore
- this patch set is based on dpdk-next-eventdev

Changes since V2:
================
- fixed meson conditional build. Moved test into driver’s meson.build
  file instead of event/meson.build
- documentation is populated as associated code is introduced
- add log_register in add dynamic logging patch
- rename RTE_xxx symbol(s) as DLB2_xxx
- replaced function ptr enqueue_four with direct call to movdir64b
- remove unused port_pages
- broke up probe patch into 3 smaller patches for easier review
- changed param order of movdir64b/movntdq to match intrinsics
- added self to MAINTAINERS files
- squashed announcement of availability into last patch in series
- correct spelling errors and delete repeated words
- DPDK_21.0 -> DPDK 21 in map file
- add experimental banner to public structs and APIs

Changes since V1:
=================
- implement changes requested in code reviews by Gage Eads and Mike
  Chen
- fix a memzone leak
- convert to use eal rte-cpuflags patch from Liang Ma

Known Issues:
- the documentation contained in dlb2.rst is not complete, and
  should be updated to include command line parameters (class of service,
  etc ...).

Depends-on: patch-82202 ("eventdev: increase MAX QUEUES PER DEV to 255")

Timothy McDaniel (23):
  event/dlb2: add documentation and meson build infrastructure
  event/dlb2: add dynamic logging
  event/dlb2: add private data structures and constants
  event/dlb2: add definitions shared with LKM or shared code
  event/dlb2: add inline functions
  event/dlb2: add eventdev probe
  event/dlb2: add flexible interface
  event/dlb2: add probe-time hardware init
  event/dlb2: add xstats
  event/dlb2: add infos get and configure
  event/dlb2: add queue and port default conf
  event/dlb2: add queue setup
  event/dlb2: add port setup
  event/dlb2: add port link
  event/dlb2: add port unlink and port unlinks in progress
  event/dlb2: add eventdev start
  event/dlb2: add enqueue and its burst variants
  event/dlb2: add dequeue and its burst variants
  event/dlb2: add eventdev stop and close
  event/dlb2: add PMD's token pop public interface
  event/dlb2: add PMD self-tests
  event/dlb2: add queue and port release
  event/dlb2: add timeout ticks entry point

 MAINTAINERS                                    |    5 +
 app/test/test_eventdev.c                       |    7 +
 config/rte_config.h                            |    7 +
 doc/api/doxy-api-index.md                      |    3 +-
 doc/api/doxy-api.conf.in                       |    1 +
 doc/guides/eventdevs/dlb2.rst                  |  365 ++
 doc/guides/eventdevs/index.rst                 |    1 +
 doc/guides/rel_notes/release_20_11.rst         |    5 +
 drivers/event/dlb2/dlb2.c                      | 3947 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c                |   74 +
 drivers/event/dlb2/dlb2_iface.h                |   74 +
 drivers/event/dlb2/dlb2_inline_fns.h           |   33 +
 drivers/event/dlb2/dlb2_log.h                  |   25 +
 drivers/event/dlb2/dlb2_priv.h                 |  578 +++
 drivers/event/dlb2/dlb2_selftest.c             | 1524 ++++++
 drivers/event/dlb2/dlb2_user.h                 |  679 +++
 drivers/event/dlb2/dlb2_xstats.c               | 1235 +++++
 drivers/event/dlb2/meson.build                 |   23 +
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 +++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 +
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c     | 6027 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  673 +++
 drivers/event/dlb2/pf/dlb2_main.h              |   97 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  728 +++
 drivers/event/dlb2/rte_pmd_dlb2.c              |   39 +
 drivers/event/dlb2/rte_pmd_dlb2.h              |   72 +
 drivers/event/dlb2/version.map                 |    9 +
 drivers/event/meson.build                      |    3 +-
 34 files changed, 22467 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
 create mode 100644 drivers/event/dlb2/dlb2_log.h
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c
 create mode 100644 drivers/event/dlb2/dlb2_user.h
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
 create mode 100644 drivers/event/dlb2/version.map

-- 
2.6.4

[dpdk-dev] [PATCH v11 01/23] event/dlb2: add documentation and meson build infrastructure

[Timothy McDaniel]

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux 64 bit X86 platforms at this time.

Adds announcement of availability for the new driver
for Intel Dynamic Load Balancer 2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 MAINTAINERS                            |  5 +++++
 config/rte_config.h                    |  7 ++++++
 doc/guides/eventdevs/dlb2.rst          | 40 ++++++++++++++++++++++++++++++++++
 doc/guides/eventdevs/index.rst         |  1 +
 doc/guides/rel_notes/release_20_11.rst |  5 +++++
 drivers/event/dlb2/meson.build         | 15 +++++++++++++
 drivers/event/dlb2/version.map         |  3 +++
 drivers/event/meson.build              |  3 ++-
 8 files changed, 78 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 5b390d1..c67fa74 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1176,6 +1176,11 @@ Cavium OCTEON TX timvf
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 F: drivers/event/octeontx/timvf_*
 
+Intel DLB2
+M: Timothy McDaniel <timothy.mcdaniel@intel.com>
+F: drivers/event/dlb2/
+F: doc/guides/eventdevs/dlb2.rst
+
 Marvell OCTEON TX2
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 M: Jerin Jacob <jerinj@marvell.com>
diff --git a/config/rte_config.h b/config/rte_config.h
index b78c6aa..64aa333 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -135,4 +135,11 @@
 /* QEDE PMD defines */
 #define RTE_LIBRTE_QEDE_FW ""
 
+/* DLB2 defines */
+#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
+#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
+#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
+#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
+
 #endif /* _RTE_CONFIG_H_ */
diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
new file mode 100644
index 0000000..e3091c9
--- /dev/null
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -0,0 +1,40 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Driver for the Intel® Dynamic Load Balancer (DLB2)
+==================================================
+
+The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
+
+Prerequisites
+-------------
+
+Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
+the basic DPDK environment.
+
+Configuration
+-------------
+
+The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
+device access. To use this operation mode, the PCIe PF device must be bound
+to a DPDK-compatible VFIO driver, such as vfio-pci.
+
+Eventdev API Notes
+------------------
+
+The DLB2 provides the functions of a DPDK event device; specifically, it
+supports atomic, ordered, and parallel scheduling events from queues to ports.
+However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
+features are abstracted by the PMD such as directed ports.
+
+In general the dlb PMD is designed for ease-of-use and does not require a
+detailed understanding of the hardware, but these details are important when
+writing high-performance code. This section describes the places where the
+eventdev API and DLB2 misalign.
+
+Flow ID
+~~~~~~~
+
+The flow ID field is preserved in the event when it is scheduled in the
+DLB2.
+
diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
index bb66a5e..738788d 100644
--- a/doc/guides/eventdevs/index.rst
+++ b/doc/guides/eventdevs/index.rst
@@ -11,6 +11,7 @@ application through the eventdev API.
     :maxdepth: 2
     :numbered:
 
+    dlb2
     dpaa
     dpaa2
     dsw
diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
index 88b9086..e1b9cc1 100644
--- a/doc/guides/rel_notes/release_20_11.rst
+++ b/doc/guides/rel_notes/release_20_11.rst
@@ -295,6 +295,11 @@ New Features
   Added performance tuning arguments to allow tuning the scheduler for
   better throughtput in high core count use cases.
 
+* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
+
+  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
+  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
+
 * **Updated ioat rawdev driver**
 
   The ioat rawdev driver has been updated and enhanced. Changes include:
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
new file mode 100644
index 0000000..68cb0ae
--- /dev/null
+++ b/drivers/event/dlb2/meson.build
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2020 Intel Corporation
+
+if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
+        build = false
+        reason = 'only supported on ARCH_X86_64 Linux'
+        subdir_done()
+endif
+
+sources = files(
+)
+
+headers = files()
+
+deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
new file mode 100644
index 0000000..4a76d1d
--- /dev/null
+++ b/drivers/event/dlb2/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index a7dac99..c433c10 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -5,7 +5,8 @@ if is_windows
 	subdir_done()
 endif
 
-drivers = ['dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw', 'dsw']
+drivers = ['dlb2', 'dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw',
+	   'dsw']
 if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
-- 
2.6.4

[dpdk-dev] [PATCH v11 02/23] event/dlb2: add dynamic logging

[Timothy McDaniel]

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c      |  7 +++++++
 drivers/event/dlb2/dlb2_log.h  | 25 +++++++++++++++++++++++++
 drivers/event/dlb2/meson.build |  3 +--
 3 files changed, 33 insertions(+), 2 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_log.h

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
new file mode 100644
index 0000000..2082dd6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2.c
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <rte_log.h>
+
+RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
new file mode 100644
index 0000000..dc1481e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_log.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_EVDEV_LOG_H_
+#define _DLB2_EVDEV_LOG_H_
+
+extern int eventdev_dlb2_log_level;
+
+/* Dynamic logging */
+#define DLB2_LOG_IMPL(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
+		__func__, ##args)
+
+#define DLB2_LOG_INFO(fmt, args...) \
+	DLB2_LOG_IMPL(INFO, fmt, ## args)
+
+#define DLB2_LOG_ERR(fmt, args...) \
+	DLB2_LOG_IMPL(ERR, fmt, ## args)
+
+/* remove debug logs at compile time unless actually debugging */
+#define DLB2_LOG_DBG(fmt, args...) \
+	RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
+
+#endif /* _DLB2_EVDEV_LOG_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 68cb0ae..6a569c9 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,8 +7,7 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files(
-)
+sources = files('dlb2.c')
 
 headers = files()
 
-- 
2.6.4

[dpdk-dev] [PATCH v11 03/23] event/dlb2: add private data structures and constants

[Timothy McDaniel]

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_priv.h | 575 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 575 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_priv.h

diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
new file mode 100644
index 0000000..61567a6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -0,0 +1,575 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_PRIV_H_
+#define _DLB2_PRIV_H_
+
+#include <emmintrin.h>
+#include <stdbool.h>
+
+#include <rte_eventdev.h>
+#include <rte_config.h>
+#include "dlb2_user.h"
+#include "dlb2_log.h"
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
+#else
+#define DLB2_INC_STAT(_stat, _incr_val)
+#endif
+
+#define EVDEV_DLB2_NAME_PMD dlb2_event
+
+/*  command line arg strings */
+#define NUMA_NODE_ARG "numa_node"
+#define DLB2_MAX_NUM_EVENTS "max_num_events"
+#define DLB2_NUM_DIR_CREDITS "num_dir_credits"
+#define DEV_ID_ARG "dev_id"
+#define DLB2_DEFER_SCHED_ARG "defer_sched"
+#define DLB2_QID_DEPTH_THRESH_ARG "qid_depth_thresh"
+#define DLB2_COS_ARG "cos"
+
+/* Begin HW related defines and structs */
+
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_VFS 16
+#define DLB2_MAX_NUM_LDB_QUEUES 32
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS 64
+#define DLB2_MAX_NUM_DIR_QUEUES 64
+#define DLB2_MAX_NUM_FLOWS (64 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDITS (8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS (2 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_DIR_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQOS_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_MAX_DEVICE_PATH 32
+#define DLB2_MIN_DEQUEUE_TIMEOUT_NS 1
+/* Note: "- 1" here to support the timeout range check in eventdev_autotest */
+#define DLB2_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1)
+#define DLB2_SW_CREDIT_BATCH_SZ 32
+#define DLB2_NUM_SN_GROUPS 2
+#define DLB2_MAX_LDB_SN_ALLOC 1024
+#define DLB2_MAX_QUEUE_DEPTH_THRESHOLD 8191
+
+/* 2048 total hist list entries and 64 total ldb ports, which
+ * makes for 2048/64 == 32 hist list entries per port. However, CQ
+ * depth must be a power of 2 and must also be >= HIST LIST entries.
+ * As a result we just limit the maximum dequeue depth to 32.
+ */
+#define DLB2_MIN_CQ_DEPTH 1
+#define DLB2_MAX_CQ_DEPTH 32
+#define DLB2_MIN_HARDWARE_CQ_DEPTH 8
+#define DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \
+	DLB2_MAX_CQ_DEPTH
+
+/*
+ * Static per queue/port provisioning values
+ */
+#define DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 64
+
+#define CQ_BASE(is_dir) ((is_dir) ? DLB2_DIR_CQ_BASE : DLB2_LDB_CQ_BASE)
+#define CQ_SIZE(is_dir) ((is_dir) ? DLB2_DIR_CQ_MAX_SIZE : \
+				    DLB2_LDB_CQ_MAX_SIZE)
+#define PP_BASE(is_dir) ((is_dir) ? DLB2_DIR_PP_BASE : DLB2_LDB_PP_BASE)
+
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+
+#define DLB2_NUM_QES_PER_CACHE_LINE 4
+
+#define DLB2_MAX_ENQUEUE_DEPTH 64
+#define DLB2_MIN_ENQUEUE_DEPTH 4
+
+#define DLB2_NAME_SIZE 64
+
+#define DLB2_1K 1024
+#define DLB2_2K (2 * DLB2_1K)
+#define DLB2_4K (4 * DLB2_1K)
+#define DLB2_16K (16 * DLB2_1K)
+#define DLB2_32K (32 * DLB2_1K)
+#define DLB2_1MB (DLB2_1K * DLB2_1K)
+#define DLB2_16MB (16 * DLB2_1MB)
+
+/* Use the upper 3 bits of the event priority to select the DLB2 priority */
+#define EV_TO_DLB2_PRIO(x) ((x) >> 5)
+#define DLB2_TO_EV_PRIO(x) ((x) << 5)
+
+enum dlb2_hw_port_types {
+	DLB2_LDB_PORT,
+	DLB2_DIR_PORT,
+	DLB2_NUM_PORT_TYPES /* Must be last */
+};
+
+enum dlb2_hw_queue_types {
+	DLB2_LDB_QUEUE,
+	DLB2_DIR_QUEUE,
+	DLB2_NUM_QUEUE_TYPES /* Must be last */
+};
+
+#define PORT_TYPE(p) ((p)->is_directed ? DLB2_DIR_PORT : DLB2_LDB_PORT)
+
+/* Do not change - must match hardware! */
+enum dlb2_hw_sched_type {
+	DLB2_SCHED_ATOMIC = 0,
+	DLB2_SCHED_UNORDERED,
+	DLB2_SCHED_ORDERED,
+	DLB2_SCHED_DIRECTED,
+	/* DLB2_NUM_HW_SCHED_TYPES must be last */
+	DLB2_NUM_HW_SCHED_TYPES
+};
+
+struct dlb2_hw_rsrcs {
+	int32_t nb_events_limit;
+	uint32_t num_queues;		/* Total queues (lb + dir) */
+	uint32_t num_ldb_queues;	/* Number of available ldb queues */
+	uint32_t num_ldb_ports;         /* Number of load balanced ports */
+	uint32_t num_dir_ports;         /* Number of directed ports */
+	uint32_t num_ldb_credits;       /* Number of load balanced credits */
+	uint32_t num_dir_credits;       /* Number of directed credits */
+	uint32_t reorder_window_size;   /* Size of reorder window */
+};
+
+struct dlb2_hw_resource_info {
+	/**> Max resources that can be provided */
+	struct dlb2_hw_rsrcs hw_rsrc_max;
+	int num_sched_domains;
+	uint32_t socket_id;
+};
+
+enum dlb2_enqueue_type {
+	/**>
+	 * New : Used to inject a new packet into the QM.
+	 */
+	DLB2_ENQ_NEW,
+	/**>
+	 * Forward : Enqueues a packet, and
+	 *  - if atomic: release any lock it holds in the QM
+	 *  - if ordered: release the packet for egress re-ordering
+	 */
+	DLB2_ENQ_FWD,
+	/**>
+	 * Enqueue Drop : Release an inflight packet. Must be called with
+	 * event == NULL. Used to drop a packet.
+	 *
+	 * Note that all packets dequeued from a load-balanced port must be
+	 * released, either with DLB2_ENQ_DROP or DLB2_ENQ_FWD.
+	 */
+	DLB2_ENQ_DROP,
+
+	/* marker for array sizing etc. */
+	_DLB2_NB_ENQ_TYPES
+};
+
+/* hw-specific format - do not change */
+
+struct dlb2_event_type {
+	uint8_t major:4;
+	uint8_t unused:4;
+	uint8_t sub;
+};
+
+union dlb2_opaque_data {
+	uint16_t opaque_data;
+	struct dlb2_event_type event_type;
+};
+
+struct dlb2_msg_info {
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+};
+
+#define DLB2_NEW_CMD_BYTE 0x08
+#define DLB2_FWD_CMD_BYTE 0x0A
+#define DLB2_COMP_CMD_BYTE 0x02
+#define DLB2_POP_CMD_BYTE 0x01
+#define DLB2_NOOP_CMD_BYTE 0x00
+
+/* hw-specific format - do not change */
+struct dlb2_enqueue_qe {
+	uint64_t data;
+	/* Word 3 */
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	/* Word 4 */
+	uint16_t lock_id;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_cq_pop_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t tokens:10;
+	uint16_t rsvd2:6;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_dequeue_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t flow_id:16; /* was pp_id in v1 */
+	uint8_t debug;
+	uint8_t cq_gen:1;
+	uint8_t qid_depth:2; /* 2 bits in v2 */
+	uint8_t rsvd1:2;
+	uint8_t error:1;
+	uint8_t rsvd2:2;
+};
+
+union dlb2_port_config {
+	struct dlb2_create_ldb_port_args ldb;
+	struct dlb2_create_dir_port_args dir;
+};
+
+enum dlb2_port_state {
+	PORT_CLOSED,
+	PORT_STARTED,
+	PORT_STOPPED
+};
+
+enum dlb2_configuration_state {
+	/* The resource has not been configured */
+	DLB2_NOT_CONFIGURED,
+	/* The resource was configured, but the device was stopped */
+	DLB2_PREV_CONFIGURED,
+	/* The resource is currently configured */
+	DLB2_CONFIGURED
+};
+
+struct dlb2_port {
+	uint32_t id;
+	bool is_directed;
+	bool gen_bit;
+	uint16_t dir_credits;
+	uint32_t dequeue_depth;
+	union dlb2_port_config cfg;
+	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
+	uint16_t cached_ldb_credits;
+	uint16_t ldb_credits;
+	uint16_t cached_dir_credits;
+	bool int_armed;
+	uint16_t owed_tokens;
+	int16_t issued_releases;
+	int cq_depth;
+	uint16_t cq_idx;
+	uint16_t cq_idx_unmasked;
+	uint16_t cq_depth_mask;
+	uint16_t gen_bit_shift;
+	enum dlb2_port_state state;
+	enum dlb2_configuration_state config_state;
+	int num_mapped_qids;
+	uint8_t *qid_mappings;
+	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
+	struct dlb2_enqueue_qe *int_arm_qe;
+	struct dlb2_cq_pop_qe *consume_qe;
+	struct dlb2_eventdev *dlb2; /* back ptr */
+	struct dlb2_eventdev_port *ev_port; /* back ptr */
+};
+
+/* Per-process per-port mmio and memory pointers */
+struct process_local_port_data {
+	uint64_t *pp_addr;
+	struct dlb2_dequeue_qe *cq_base;
+	const struct rte_memzone *mz;
+	bool mmaped;
+};
+
+struct dlb2_eventdev;
+
+struct dlb2_config {
+	int configured;
+	int reserved;
+	uint32_t num_ldb_credits;
+	uint32_t num_dir_credits;
+	struct dlb2_create_sched_domain_args resources;
+};
+
+enum dlb2_cos {
+	DLB2_COS_DEFAULT = -1,
+	DLB2_COS_0 = 0,
+	DLB2_COS_1,
+	DLB2_COS_2,
+	DLB2_COS_3
+};
+
+struct dlb2_hw_dev {
+	struct dlb2_config cfg;
+	struct dlb2_hw_resource_info info;
+	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
+	uint32_t domain_id;
+	enum dlb2_cos cos_id;
+	rte_spinlock_t resource_lock; /* for MP support */
+} __rte_cache_aligned;
+
+/* End HW related defines and structs */
+
+/* Begin DLB2 PMD Eventdev related defines and structs */
+
+#define DLB2_MAX_NUM_QUEUES \
+	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
+
+#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS + DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
+
+/** Structure to hold the queue to port link establishment attributes */
+
+struct dlb2_event_queue_link {
+	uint8_t queue_id;
+	uint8_t priority;
+	bool mapped;
+	bool valid;
+};
+
+struct dlb2_traffic_stats {
+	uint64_t rx_ok;
+	uint64_t rx_drop;
+	uint64_t rx_interrupt_wait;
+	uint64_t rx_umonitor_umwait;
+	uint64_t tx_ok;
+	uint64_t total_polls;
+	uint64_t zero_polls;
+	uint64_t tx_nospc_ldb_hw_credits;
+	uint64_t tx_nospc_dir_hw_credits;
+	uint64_t tx_nospc_inflight_max;
+	uint64_t tx_nospc_new_event_limit;
+	uint64_t tx_nospc_inflight_credits;
+};
+
+/* DLB2 HW sets the 2bit qid_depth in rx QEs based on the programmable depth
+ * threshold. The global default value in config/common_base (or rte_config.h)
+ * can be overridden on a per-qid basis using a vdev command line parameter.
+ * 3: depth > threshold
+ * 2: threshold >= depth > 3/4 threshold
+ * 1: 3/4 threshold >= depth > 1/2 threshold
+ * 0: depth <= 1/2 threshold.
+ */
+#define DLB2_QID_DEPTH_LE50 0
+#define DLB2_QID_DEPTH_GT50_LE75 1
+#define DLB2_QID_DEPTH_GT75_LE100 2
+#define DLB2_QID_DEPTH_GT100 3
+#define DLB2_NUM_QID_DEPTH_STAT_VALS 4 /* 2 bits */
+
+struct dlb2_queue_stats {
+	uint64_t enq_ok;
+	uint64_t qid_depth[DLB2_NUM_QID_DEPTH_STAT_VALS];
+};
+
+struct dlb2_port_stats {
+	struct dlb2_traffic_stats traffic;
+	uint64_t tx_op_cnt[4]; /* indexed by rte_event.op */
+	uint64_t tx_implicit_rel;
+	uint64_t tx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t tx_invalid;
+	uint64_t rx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t rx_sched_invalid;
+	struct dlb2_queue_stats queue[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_eventdev_port {
+	struct dlb2_port qm_port; /* hw specific data structure */
+	struct rte_event_port_conf conf; /* user-supplied configuration */
+	uint16_t inflight_credits; /* num credits this port has right now */
+	uint16_t credit_update_quanta;
+	struct dlb2_eventdev *dlb2; /* backlink optimization */
+	struct dlb2_port_stats stats __rte_cache_aligned;
+	struct dlb2_event_queue_link link[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	int num_links;
+	uint32_t id; /* port id */
+	/* num releases yet to be completed on this port.
+	 * Only applies to load-balanced ports.
+	 */
+	uint16_t outstanding_releases;
+	uint16_t inflight_max; /* app requested max inflights for this port */
+	/* setup_done is set when the event port is setup */
+	bool setup_done;
+	/* enq_configured is set when the qm port is created */
+	bool enq_configured;
+	uint8_t implicit_release; /* release events before dequeueing */
+}  __rte_cache_aligned;
+
+struct dlb2_queue {
+	uint32_t num_qid_inflights; /* User config */
+	uint32_t num_atm_inflights; /* User config */
+	enum dlb2_configuration_state config_state;
+	int sched_type; /* LB queue only */
+	uint32_t id;
+	bool is_directed;
+};
+
+struct dlb2_eventdev_queue {
+	struct dlb2_queue qm_queue;
+	struct rte_event_queue_conf conf; /* User config */
+	int depth_threshold; /* use default if 0 */
+	uint32_t id;
+	bool setup_done;
+	uint8_t num_links;
+};
+
+enum dlb2_run_state {
+	DLB2_RUN_STATE_STOPPED = 0,
+	DLB2_RUN_STATE_STOPPING,
+	DLB2_RUN_STATE_STARTING,
+	DLB2_RUN_STATE_STARTED
+};
+
+struct dlb2_eventdev {
+	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
+	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each queue */
+	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
+	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each port */
+	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
+	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
+	struct dlb2_get_num_resources_args hw_rsrc_query_results;
+	uint32_t xstats_count_mode_queue;
+	struct dlb2_hw_dev qm_instance; /* strictly hw related */
+	uint64_t global_dequeue_wait_ticks;
+	struct dlb2_xstats_entry *xstats;
+	struct rte_eventdev *event_dev; /* backlink to dev */
+	uint32_t xstats_count_mode_dev;
+	uint32_t xstats_count_mode_port;
+	uint32_t xstats_count;
+	uint32_t inflights; /* use __atomic builtins */
+	uint32_t new_event_limit;
+	int max_num_events_override;
+	int num_dir_credits_override;
+	volatile enum dlb2_run_state run_state;
+	uint16_t num_dir_queues; /* total num of evdev dir queues requested */
+	uint16_t num_dir_credits;
+	uint16_t num_ldb_credits;
+	uint16_t num_queues; /* total queues */
+	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested */
+	uint16_t num_ports; /* total num of evdev ports requested */
+	uint16_t num_ldb_ports; /* total num of ldb ports requested */
+	uint16_t num_dir_ports; /* total num of dir ports requested */
+	bool umwait_allowed;
+	bool global_dequeue_wait; /* Not using per dequeue wait if true */
+	bool defer_sched;
+	enum dlb2_cq_poll_modes poll_mode;
+	uint8_t revision;
+	bool configured;
+	uint16_t max_ldb_credits;
+	uint16_t max_dir_credits;
+
+	/* force hw credit pool counters into exclusive cache lines */
+
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t ldb_credit_pool __rte_cache_aligned;
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t dir_credit_pool __rte_cache_aligned;
+};
+
+/* used for collecting and passing around the dev args */
+struct dlb2_qid_depth_thresholds {
+	int val[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_devargs {
+	int socket_id;
+	int max_num_events;
+	int num_dir_credits_override;
+	int dev_id;
+	int defer_sched;
+	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
+	enum dlb2_cos cos_id;
+};
+
+/* End Eventdev related defines and structs */
+
+/* Forwards for non-inlined functions */
+
+void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
+
+int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
+
+void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
+
+int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n);
+
+int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstat_names,
+		unsigned int *ids, unsigned int size);
+
+uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+					  const char *name, unsigned int *id);
+
+int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode,
+		int16_t queue_port_id,
+		const uint32_t ids[],
+		uint32_t nb_ids);
+
+int test_dlb2_eventdev(void);
+
+int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+				const char *name,
+				struct dlb2_devargs *dlb2_args);
+
+int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+				  const char *name);
+
+uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *queue);
+
+int dlb2_parse_params(const char *params,
+		      const char *name,
+		      struct dlb2_devargs *dlb2_args);
+
+/* Extern globals */
+extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
+
+#endif	/* _DLB2_PRIV_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v11 04/23] event/dlb2: add definitions shared with LKM or shared code

[Timothy McDaniel]

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_user.h | 679 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 679 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_user.h

diff --git a/drivers/event/dlb2/dlb2_user.h b/drivers/event/dlb2/dlb2_user.h
new file mode 100644
index 0000000..f4bda78
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_user.h
@@ -0,0 +1,679 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#define DLB2_MAX_NAME_LEN 64
+
+#include <linux/types.h>
+
+enum dlb2_error {
+	DLB2_ST_SUCCESS = 0,
+	DLB2_ST_NAME_EXISTS,
+	DLB2_ST_DOMAIN_UNAVAILABLE,
+	DLB2_ST_LDB_PORTS_UNAVAILABLE,
+	DLB2_ST_DIR_PORTS_UNAVAILABLE,
+	DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB2_ST_INVALID_DOMAIN_ID,
+	DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB2_ST_INVALID_LDB_QUEUE_ID,
+	DLB2_ST_INVALID_CQ_DEPTH,
+	DLB2_ST_INVALID_CQ_VIRT_ADDR,
+	DLB2_ST_INVALID_PORT_ID,
+	DLB2_ST_INVALID_QID,
+	DLB2_ST_INVALID_PRIORITY,
+	DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+	DLB2_ST_INVALID_DIR_QUEUE_ID,
+	DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB2_ST_DOMAIN_NOT_CONFIGURED,
+	DLB2_ST_INTERNAL_ERROR,
+	DLB2_ST_DOMAIN_IN_USE,
+	DLB2_ST_DOMAIN_NOT_FOUND,
+	DLB2_ST_QUEUE_NOT_FOUND,
+	DLB2_ST_DOMAIN_STARTED,
+	DLB2_ST_DOMAIN_NOT_STARTED,
+	DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB2_ST_DOMAIN_RESET_FAILED,
+	DLB2_ST_MBOX_ERROR,
+	DLB2_ST_INVALID_HIST_LIST_DEPTH,
+	DLB2_ST_NO_MEMORY,
+	DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB2_ST_INVALID_COS_ID,
+};
+
+static const char dlb2_error_strings[][128] = {
+	"DLB2_ST_SUCCESS",
+	"DLB2_ST_NAME_EXISTS",
+	"DLB2_ST_DOMAIN_UNAVAILABLE",
+	"DLB2_ST_LDB_PORTS_UNAVAILABLE",
+	"DLB2_ST_DIR_PORTS_UNAVAILABLE",
+	"DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+	"DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+	"DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+	"DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+	"DLB2_ST_INVALID_DOMAIN_ID",
+	"DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+	"DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+	"DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+	"DLB2_ST_INVALID_LDB_QUEUE_ID",
+	"DLB2_ST_INVALID_CQ_DEPTH",
+	"DLB2_ST_INVALID_CQ_VIRT_ADDR",
+	"DLB2_ST_INVALID_PORT_ID",
+	"DLB2_ST_INVALID_QID",
+	"DLB2_ST_INVALID_PRIORITY",
+	"DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+	"DLB2_ST_INVALID_DIR_QUEUE_ID",
+	"DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+	"DLB2_ST_DOMAIN_NOT_CONFIGURED",
+	"DLB2_ST_INTERNAL_ERROR",
+	"DLB2_ST_DOMAIN_IN_USE",
+	"DLB2_ST_DOMAIN_NOT_FOUND",
+	"DLB2_ST_QUEUE_NOT_FOUND",
+	"DLB2_ST_DOMAIN_STARTED",
+	"DLB2_ST_DOMAIN_NOT_STARTED",
+	"DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+	"DLB2_ST_DOMAIN_RESET_FAILED",
+	"DLB2_ST_MBOX_ERROR",
+	"DLB2_ST_INVALID_HIST_LIST_DEPTH",
+	"DLB2_ST_NO_MEMORY",
+	"DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+	"DLB2_ST_INVALID_COS_ID",
+};
+
+struct dlb2_cmd_response {
+	__u32 status; /* Interpret using enum dlb2_error */
+	__u32 id;
+};
+
+/*******************/
+/* 'dlb2' commands */
+/*******************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+	DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	This ioctl interface is the same in all driver versions and is always
+ *	the first ioctl.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+	/* Output parameters */
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ *	load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ *		percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ *	load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ *   ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+enum dlb2_cq_poll_modes {
+	DLB2_CQ_POLL_MODE_STD,
+	DLB2_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB2_CQ_POLL_MODE must be last */
+	NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/********************************/
+/* 'scheduling domain' commands */
+/********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_start_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
+#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
+
+
+#endif /* __DLB2_USER_H */
-- 
2.6.4

[dpdk-dev] [PATCH v11 05/23] event/dlb2: add inline functions

[Timothy McDaniel]

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_inline_fns.h | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h

diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
new file mode 100644
index 0000000..ac8d01a
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_inline_fns.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_INLINE_FNS_H_
+#define _DLB2_INLINE_FNS_H_
+
+/* Inline functions required in more than one source file. */
+
+static inline struct dlb2_eventdev *
+dlb2_pmd_priv(const struct rte_eventdev *eventdev)
+{
+	return eventdev->data->dev_private;
+}
+
+static inline void
+dlb2_movntdq_single(void *pp_addr, void *qe4)
+{
+	long long *_qe  = (long long *)qe4;
+	__m128i src_data0 = (__m128i){_qe[0], _qe[1]};
+
+	_mm_stream_si128(pp_addr, src_data0);
+}
+
+static inline void
+dlb2_movdir64b(void *dest, void *src)
+{
+	asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
+		:
+	: "a" (dest), "d" (src));
+}
+
+#endif /* _DLB2_INLINE_FNS_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v11 06/23] event/dlb2: add eventdev probe

[Timothy McDaniel]

Add the eventdev portion of probe, and parse command line
options, but do not initialize hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                      |  343 ++++
 drivers/event/dlb2/meson.build                 |    5 +-
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +++
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  621 ++++++
 drivers/event/dlb2/pf/dlb2_main.h              |   98 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  196 ++
 13 files changed, 7497 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2082dd6..1b11979 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2,6 +2,349 @@
  * Copyright(c) 2016-2020 Intel Corporation
  */
 
+#include <assert.h>
+#include <errno.h>
+#include <nmmintrin.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_cycles.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_io.h>
+#include <rte_kvargs.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_ring.h>
+#include <rte_string_fns.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+/*
+ * Resources exposed to eventdev. Some values overridden at runtime using
+ * values returned by the DLB kernel driver.
+ */
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
+#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
+#endif
+
+struct process_local_port_data
+dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
+
+#define DLB2_BASE_10 10
+
+static int
+dlb2_string_to_int(int *result, const char *str)
+{
+	long ret;
+	char *endptr;
+
+	if (str == NULL || result == NULL)
+		return -EINVAL;
+
+	errno = 0;
+	ret = strtol(str, &endptr, DLB2_BASE_10);
+	if (errno)
+		return -errno;
+
+	/* long int and int may be different width for some architectures */
+	if (ret < INT_MIN || ret > INT_MAX || endptr == str)
+		return -EINVAL;
+
+	*result = ret;
+	return 0;
+}
+
+static int
+set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
+{
+	int *socket_id = opaque;
+	int ret;
+
+	ret = dlb2_string_to_int(socket_id, value);
+	if (ret < 0)
+		return ret;
+
+	if (*socket_id > RTE_MAX_NUMA_NODES)
+		return -EINVAL;
+	return 0;
+}
+
+static int
+set_max_num_events(const char *key __rte_unused,
+		   const char *value,
+		   void *opaque)
+{
+	int *max_num_events = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(max_num_events, value);
+	if (ret < 0)
+		return ret;
+
+	if (*max_num_events < 0 || *max_num_events >
+			DLB2_MAX_NUM_LDB_CREDITS) {
+		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
+			     DLB2_MAX_NUM_LDB_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_num_dir_credits(const char *key __rte_unused,
+		    const char *value,
+		    void *opaque)
+{
+	int *num_dir_credits = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(num_dir_credits, value);
+	if (ret < 0)
+		return ret;
+
+	if (*num_dir_credits < 0 ||
+	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
+		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
+			     DLB2_MAX_NUM_DIR_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_dev_id(const char *key __rte_unused,
+	   const char *value,
+	   void *opaque)
+{
+	int *dev_id = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(dev_id, value);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+set_cos(const char *key __rte_unused,
+	const char *value,
+	void *opaque)
+{
+	enum dlb2_cos *cos_id = opaque;
+	int x = 0;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(&x, value);
+	if (ret < 0)
+		return ret;
+
+	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
+		DLB2_LOG_ERR(
+			"COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
+			x);
+		return -EINVAL;
+	}
+
+	*cos_id = x;
+
+	return 0;
+}
+
+
+static int
+set_qid_depth_thresh(const char *key __rte_unused,
+		     const char *value,
+		     void *opaque)
+{
+	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
+	int first, last, thresh, i;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	/* command line override may take one of the following 3 forms:
+	 * qid_depth_thresh=all:<threshold_value> ... all queues
+	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
+	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
+	 */
+	if (sscanf(value, "all:%d", &thresh) == 1) {
+		first = 0;
+		last = DLB2_MAX_NUM_QUEUES - 1;
+	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
+		/* we have everything we need */
+	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
+		last = first;
+	} else {
+		DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val, qid-qid:val, or qid:val\n");
+		return -EINVAL;
+	}
+
+	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
+		return -EINVAL;
+	}
+
+	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
+			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
+		return -EINVAL;
+	}
+
+	for (i = first; i <= last; i++)
+		qid_thresh->val[i] = thresh; /* indexed by qid */
+
+	return 0;
+}
+
+int
+dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+			    const char *name,
+			    struct dlb2_devargs *dlb2_args)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+	RTE_SET_USED(dlb2_args);
+
+	return 0;
+}
+
+int
+dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+			      const char *name)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+int
+dlb2_parse_params(const char *params,
+		  const char *name,
+		  struct dlb2_devargs *dlb2_args)
+{
+	int ret = 0;
+	static const char * const args[] = { NUMA_NODE_ARG,
+					     DLB2_MAX_NUM_EVENTS,
+					     DLB2_NUM_DIR_CREDITS,
+					     DEV_ID_ARG,
+					     DLB2_QID_DEPTH_THRESH_ARG,
+					     DLB2_COS_ARG,
+					     NULL };
+
+	if (params != NULL && params[0] != '\0') {
+		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
+
+		if (kvlist == NULL) {
+			RTE_LOG(INFO, PMD,
+				"Ignoring unsupported parameters when creating device '%s'\n",
+				name);
+		} else {
+			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
+						     set_numa_node,
+						     &dlb2_args->socket_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing numa node parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
+						 set_max_num_events,
+						 &dlb2_args->max_num_events);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist,
+					DLB2_NUM_DIR_CREDITS,
+					set_num_dir_credits,
+					&dlb2_args->num_dir_credits_override);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
+						 set_dev_id,
+						 &dlb2_args->dev_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(
+					kvlist,
+					DLB2_QID_DEPTH_THRESH_ARG,
+					set_qid_depth_thresh,
+					&dlb2_args->qid_depth_thresholds);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
+						 set_cos,
+						 &dlb2_args->cos_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing cos parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
 
+			rte_kvargs_free(kvlist);
+		}
+	}
+	return ret;
+}
 RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 6a569c9..2855dd5 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,7 +7,10 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files('dlb2.c')
+sources = files('dlb2.c',
+		'pf/dlb2_main.c',
+		'pf/dlb2_pf.c'
+)
 
 headers = files()
 
diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
new file mode 100644
index 0000000..1d99f1e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_osdep_list.h"
+#include "dlb2_osdep_types.h"
+
+#define DLB2_MAX_NUM_VDEVS			16
+#define DLB2_MAX_NUM_DOMAINS			32
+#define DLB2_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS			64
+#define DLB2_MAX_NUM_DIR_PORTS			64
+#define DLB2_MAX_NUM_LDB_CREDITS		(8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS		(2 * 1024)
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB2_MAX_NUM_AQED_ENTRIES		2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB2_QID_PRIORITIES			8
+#define DLB2_NUM_ARB_WEIGHTS			8
+#define DLB2_MAX_WEIGHT				255
+#define DLB2_NUM_COS_DOMAINS			4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#ifdef FPGA
+#define DLB2_HZ					2000000
+#else
+#define DLB2_HZ					800000000
+#endif
+
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
+
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+	DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+	DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID		3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID		4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW		5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ	1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
+#define DLB2_VF_BASE_CQ_VECTOR_ID	     0
+#define DLB2_VF_LAST_CQ_VECTOR_ID	     30
+#define DLB2_VF_MBOX_VECTOR_ID		     31
+#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
+
+#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
+					     DLB2_MAX_NUM_DIR_PORTS + 1)
+
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE   0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
+				DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE   0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
+				DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE       0x2100000
+#define DLB2_LDB_PP_STRIDE     0x1000
+#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
+				DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE       0x2000000
+#define DLB2_DIR_PP_STRIDE     0x1000
+#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
+				DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
+struct dlb2_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb2_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb2_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
+struct dlb2_ldb_queue {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB2_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB2_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB2_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB2_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+	enum dlb2_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb2_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+	u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+	u32 bound[6] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & (1 << i))) {
+			group->slot_use_bitmap |= 1 << i;
+			return i;
+		}
+	}
+
+	return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+	int i, cnt = 0;
+
+	for (i = 0; i < 32; i++)
+		cnt += !!(group->slot_use_bitmap & (1 << i));
+
+	return cnt;
+}
+
+struct dlb2_hw_domain {
+	struct dlb2_function_resources *parent_func;
+	struct dlb2_list_entry func_list;
+	struct dlb2_list_head used_ldb_queues;
+	struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head used_dir_pq_pairs;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb2_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb2_bitmap;
+
+struct dlb2_function_resources {
+	struct dlb2_list_head avail_domains;
+	struct dlb2_list_head used_domains;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	struct dlb2_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+	struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+	struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+	struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
+struct dlb2_mbox {
+	u32 *mbox;
+	u32 *isr_in_progress;
+};
+
+struct dlb2_sw_mbox {
+	struct dlb2_mbox vdev_to_pf;
+	struct dlb2_mbox pf_to_vdev;
+	void (*pf_to_vdev_inject)(void *arg);
+	void *pf_to_vdev_inject_arg;
+};
+
+struct dlb2_hw {
+	/* BAR 0 address */
+	void  *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void  *func_kva;
+	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb2_hw_resources rsrcs;
+	struct dlb2_function_resources pf;
+	struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+	struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+	/* Virtualization */
+	int virt_mode;
+	struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
+	unsigned int pasid[DLB2_MAX_NUM_VDEVS];
+};
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
new file mode 100644
index 0000000..ce462c0
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
@@ -0,0 +1,596 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_BASE_DLB2_MBOX_H
+#define __DLB2_BASE_DLB2_MBOX_H
+
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+
+#define DLB2_MBOX_INTERFACE_VERSION 1
+
+/*
+ * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
+ * to send requests of its own (e.g. notifying a VF of an impending FLR).
+ * To avoid communication race conditions, e.g. the PF sends a response and then
+ * sends a request before the VF reads the response, the PF->VF mailbox is
+ * divided into two sections:
+ * - Bytes 0-47: PF responses
+ * - Bytes 48-63: PF requests
+ *
+ * Partitioning the PF->VF mailbox allows responses and requests to occupy the
+ * mailbox simultaneously.
+ */
+#define DLB2_PF2VF_RESP_BYTES	  48
+#define DLB2_PF2VF_RESP_BASE	  0
+#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
+
+#define DLB2_PF2VF_REQ_BYTES	  16
+#define DLB2_PF2VF_REQ_BASE	  (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
+#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
+
+/*
+ * Similarly, the VF->PF mailbox is divided into two sections:
+ * - Bytes 0-239: VF requests
+ * -- (Bytes 0-3 are unused due to a hardware errata)
+ * - Bytes 240-255: VF responses
+ */
+#define DLB2_VF2PF_REQ_BYTES	 236
+#define DLB2_VF2PF_REQ_BASE	 4
+#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
+
+#define DLB2_VF2PF_RESP_BYTES	  16
+#define DLB2_VF2PF_RESP_BASE	  (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
+#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
+
+/* VF-initiated commands */
+enum dlb2_mbox_cmd_type {
+	DLB2_MBOX_CMD_REGISTER,
+	DLB2_MBOX_CMD_UNREGISTER,
+	DLB2_MBOX_CMD_GET_NUM_RESOURCES,
+	DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
+	DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
+	DLB2_MBOX_CMD_CREATE_LDB_PORT,
+	DLB2_MBOX_CMD_CREATE_DIR_PORT,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT,
+	DLB2_MBOX_CMD_DISABLE_LDB_PORT,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT,
+	DLB2_MBOX_CMD_DISABLE_DIR_PORT,
+	DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_MAP_QID,
+	DLB2_MBOX_CMD_UNMAP_QID,
+	DLB2_MBOX_CMD_START_DOMAIN,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
+	DLB2_MBOX_CMD_ARM_CQ_INTR,
+	DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
+	DLB2_MBOX_CMD_GET_SN_ALLOCATION,
+	DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
+	DLB2_MBOX_CMD_GET_COS_BW,
+	DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
+	DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_QE_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_CMD_TYPES,
+};
+
+static const char dlb2_mbox_cmd_type_strings[][128] = {
+	"DLB2_MBOX_CMD_REGISTER",
+	"DLB2_MBOX_CMD_UNREGISTER",
+	"DLB2_MBOX_CMD_GET_NUM_RESOURCES",
+	"DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_LDB_PORT",
+	"DLB2_MBOX_CMD_CREATE_DIR_PORT",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_DISABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_DISABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_MAP_QID",
+	"DLB2_MBOX_CMD_UNMAP_QID",
+	"DLB2_MBOX_CMD_START_DOMAIN",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
+	"DLB2_MBOX_CMD_ARM_CQ_INTR",
+	"DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
+	"DLB2_MBOX_CMD_GET_SN_ALLOCATION",
+	"DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
+	"DLB2_MBOX_CMD_GET_COS_BW",
+	"DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
+	"DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
+};
+
+/* PF-initiated commands */
+enum dlb2_mbox_vf_cmd_type {
+	DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
+	DLB2_MBOX_VF_CMD_NOTIFICATION,
+	DLB2_MBOX_VF_CMD_IN_USE,
+
+	/* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_VF_CMD_TYPES,
+};
+
+static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
+	"DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
+	"DLB2_MBOX_VF_CMD_NOTIFICATION",
+	"DLB2_MBOX_VF_CMD_IN_USE",
+};
+
+#define DLB2_MBOX_CMD_TYPE(hdr) \
+	(((struct dlb2_mbox_req_hdr *)hdr)->type)
+#define DLB2_MBOX_CMD_STRING(hdr) \
+	dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
+
+enum dlb2_mbox_status_type {
+	DLB2_MBOX_ST_SUCCESS,
+	DLB2_MBOX_ST_INVALID_CMD_TYPE,
+	DLB2_MBOX_ST_VERSION_MISMATCH,
+	DLB2_MBOX_ST_INVALID_OWNER_VF,
+};
+
+static const char dlb2_mbox_status_type_strings[][128] = {
+	"DLB2_MBOX_ST_SUCCESS",
+	"DLB2_MBOX_ST_INVALID_CMD_TYPE",
+	"DLB2_MBOX_ST_VERSION_MISMATCH",
+	"DLB2_MBOX_ST_INVALID_OWNER_VF",
+};
+
+#define DLB2_MBOX_ST_TYPE(hdr) \
+	(((struct dlb2_mbox_resp_hdr *)hdr)->status)
+#define DLB2_MBOX_ST_STRING(hdr) \
+	dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
+
+/* This structure is always the first field in a request structure */
+struct dlb2_mbox_req_hdr {
+	u32 type;
+};
+
+/* This structure is always the first field in a response structure */
+struct dlb2_mbox_resp_hdr {
+	u32 status;
+};
+
+struct dlb2_mbox_register_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 min_interface_version;
+	u16 max_interface_version;
+};
+
+struct dlb2_mbox_register_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 interface_version;
+	u8 pf_id;
+	u8 vf_id;
+	u8 is_auxiliary_vf;
+	u8 primary_vf_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u16 num_sched_domains;
+	u16 num_ldb_queues;
+	u16 num_ldb_ports;
+	u16 num_cos_ldb_ports[4];
+	u16 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 max_contiguous_hist_list_entries;
+	u16 num_ldb_credits;
+	u16 num_dir_credits;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 num_ldb_queues;
+	u32 num_ldb_ports;
+	u32 num_cos_ldb_ports[4];
+	u32 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u8 cos_strict;
+	u8 padding0[3];
+	u32 padding1;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 num_sequence_numbers;
+	u32 num_qid_inflights;
+	u32 num_atomic_inflights;
+	u32 lock_id_comp_level;
+	u32 depth_threshold;
+	u32 padding;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 depth_threshold;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u16 cq_depth;
+	u16 cq_history_list_size;
+	u8 cos_id;
+	u8 cos_strict;
+	u16 padding1;
+	u64 cq_base_address;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u64 cq_base_address;
+	u16 cq_depth;
+	u16 padding0;
+	s32 queue_id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_map_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+	u32 priority;
+	u32 padding0;
+};
+
+struct dlb2_mbox_map_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_start_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+};
+
+struct dlb2_mbox_start_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 is_ldb;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding0;
+};
+
+/*
+ * The alert_id and aux_alert_data follows the format of the alerts defined in
+ * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
+ * the aux_alert_data value varies depending on the alert.
+ */
+struct dlb2_mbox_vf_alert_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 alert_id;
+	u32 aux_alert_data;
+};
+
+enum dlb2_mbox_vf_notification_type {
+	DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
+	DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
+
+	/* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
+	NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
+};
+
+struct dlb2_mbox_vf_notification_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 notification;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 in_use;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 num;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 cos_id;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 mode;
+};
+
+#endif /* __DLB2_BASE_DLB2_MBOX_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
new file mode 100644
index 0000000..c4c34eb
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
@@ -0,0 +1,230 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_H
+#define __DLB2_OSDEP_H
+
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <rte_string_fns.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include "../dlb2_main.h"
+#include "dlb2_resource.h"
+#include "../../dlb2_log.h"
+#include "../../dlb2_user.h"
+
+
+#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
+#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
+#define DLB2_CSR_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
+#define DLB2_FUNC_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
+/* Map to PMDs logging interface */
+#define DLB2_ERR(dev, fmt, args...) \
+	DLB2_LOG_ERR(fmt, ## args)
+
+#define DLB2_INFO(dev, fmt, args...) \
+	DLB2_LOG_INFO(fmt, ## args)
+
+#define DLB2_DEBUG(dev, fmt, args...) \
+	DLB2_LOG_DBG(fmt, ## args)
+
+/**
+ * os_udelay() - busy-wait for a number of microseconds
+ * @usecs: delay duration.
+ */
+static inline void os_udelay(int usecs)
+{
+	rte_delay_us(usecs);
+}
+
+/**
+ * os_msleep() - sleep for a number of milliseconds
+ * @usecs: delay duration.
+ */
+static inline void os_msleep(int msecs)
+{
+	rte_delay_ms(msecs);
+}
+
+#define DLB2_PP_BASE(__is_ldb) \
+	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
+
+/**
+ * os_map_producer_port() - map a producer port into the caller's address space
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ *
+ * This function maps the requested producer port memory into the caller's
+ * address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void *os_map_producer_port(struct dlb2_hw *hw,
+					 u8 port_id,
+					 bool is_ldb)
+{
+	uint64_t addr;
+	uint64_t pp_dma_base;
+
+	pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
+	addr = (pp_dma_base + (PAGE_SIZE * port_id));
+
+	return (void *)(uintptr_t)addr;
+}
+
+/**
+ * os_unmap_producer_port() - unmap a producer port
+ * @addr: mapped producer port address
+ *
+ * This function undoes os_map_producer_port() by unmapping the producer port
+ * memory from the caller's address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(addr);
+}
+
+/**
+ * os_fence_hcw() - fence an HCW to ensure it arrives at the device
+ * @hw: dlb2_hw handle for a particular device.
+ * @pp_addr: producer port address
+ */
+static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
+{
+	RTE_SET_USED(hw);
+
+	/* To ensure outstanding HCWs reach the device, read the PP address. IA
+	 * memory ordering prevents reads from passing older writes, and the
+	 * mfence also ensures this.
+	 */
+	rte_mb();
+
+	*(volatile u64 *)pp_addr;
+}
+
+/**
+ * DLB2_HW_ERR() - log an error message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_ERR(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_ERR(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/**
+ * DLB2_HW_DBG() - log an info message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_DBG(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_DEBUG(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/* The callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void *dlb2_complete_queue_map_unmap(void *__args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
+	int ret;
+
+	while (1) {
+		rte_spinlock_lock(&dlb2_dev->resource_mutex);
+
+		ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+		ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+		if (ret != 0) {
+			rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+			/* Relinquish the CPU so the application can process
+			 * its CQs, so this function doesn't deadlock.
+			 */
+			sched_yield();
+		} else {
+			break;
+		}
+	}
+
+	dlb2_dev->worker_launched = false;
+
+	rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+
+	return NULL;
+}
+
+
+/**
+ * os_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static inline void os_schedule_work(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+	pthread_t complete_queue_map_unmap_thread;
+	int ret;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
+				     "dlb_queue_unmap_waiter",
+				     NULL,
+				     dlb2_complete_queue_map_unmap,
+				     dlb2_dev);
+	if (ret)
+		DLB2_ERR(dlb2_dev,
+			 "Could not create queue complete map/unmap thread, err=%d\n",
+			 ret);
+	else
+		dlb2_dev->worker_launched = true;
+}
+
+/**
+ * os_worker_active() - query whether the map/unmap worker thread is active
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether a thread (launched by
+ * os_schedule_work()) is active. This function is used to determine
+ * whether or not to launch a worker thread.
+ */
+static inline bool os_worker_active(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	return dlb2_dev->worker_launched;
+}
+
+#endif /*  __DLB2_OSDEP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
new file mode 100644
index 0000000..d47d1ed
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <rte_bitmap.h>
+#include <rte_string_fns.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include "../dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+	struct rte_bitmap *map;
+	unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+				    unsigned int len)
+{
+	struct dlb2_bitmap *bm;
+	void *mem;
+	uint32_t alloc_size;
+	uint32_t nbits = (uint32_t)len;
+
+	if (bitmap == NULL || nbits == 0)
+		return -EINVAL;
+
+	/* Allocate DLB2 bitmap control struct */
+	bm = rte_malloc("DLB2_PF",
+			sizeof(struct dlb2_bitmap),
+			RTE_CACHE_LINE_SIZE);
+
+	if (bm == NULL)
+		return -ENOMEM;
+
+	/* Allocate bitmap memory */
+	alloc_size = rte_bitmap_get_memory_footprint(nbits);
+	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
+	if (mem == NULL) {
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->map = rte_bitmap_init(len, mem, alloc_size);
+	if (bm->map == NULL) {
+		rte_free(mem);
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap == NULL)
+		return;
+
+	rte_free(bitmap->map);
+	rte_free(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++)
+		rte_bitmap_set(bitmap->map, i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	rte_bitmap_reset(bitmap->map);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set() - set a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
+				  unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_set(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+					unsigned int bit,
+					unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_set(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear() - clear a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
+				    unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_clear(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+					  unsigned int bit,
+					  unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_clear(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+						 unsigned int len)
+{
+	unsigned int i, j = 0;
+
+	if (bitmap  == NULL || bitmap->map == NULL || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			if (++j == len)
+				return i - j + 1;
+		} else {
+			j = 0;
+		}
+	}
+
+	/* No set bit range of length len? */
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			return i;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+	int weight = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			weight++;
+	}
+	return weight;
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				      bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+	int max_len = 0, len = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			len++;
+		} else {
+			if (len > max_len)
+				max_len = len;
+			len = 0;
+		}
+	}
+
+	if (len > max_len)
+		max_len = len;
+
+	return max_len;
+}
+
+/**
+ * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
+ * @dest: pointer to dlb2_bitmap structure, which will contain the results of
+ *	  the 'or' of src1 and src2.
+ * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
+ * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
+ *
+ * This function 'or's two bitmaps together and stores the result in a third
+ * bitmap. The source and destination bitmaps can be the same.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - One of the bitmaps is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
+				 struct dlb2_bitmap *src1,
+				 struct dlb2_bitmap *src2)
+{
+	unsigned int i, min;
+	int numset = 0;
+
+	if (dest  == NULL || dest->map == NULL ||
+	    src1  == NULL || src1->map == NULL ||
+	    src2  == NULL || src2->map == NULL)
+		return -EINVAL;
+
+	min = dest->len;
+	min = (min > src1->len) ? src1->len : min;
+	min = (min > src2->len) ? src2->len : min;
+
+	for (i = 0; i != min; i++) {
+		if  (rte_bitmap_get(src1->map, i) ||
+		     rte_bitmap_get(src2->map, i)) {
+			rte_bitmap_set(dest->map, i);
+			numset++;
+		} else {
+			rte_bitmap_clear(dest->map, i);
+		}
+	}
+
+	return numset;
+}
+
+#endif /*  __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
new file mode 100644
index 0000000..5531739
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_LIST_H
+#define __DLB2_OSDEP_LIST_H
+
+#include <rte_tailq.h>
+
+struct dlb2_list_entry {
+	TAILQ_ENTRY(dlb2_list_entry) node;
+};
+
+/* Dummy - just a struct definition */
+TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
+
+/* =================
+ * TAILQ Supplements
+ * =================
+ */
+
+#ifndef TAILQ_FOREACH_ENTRY
+#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)		\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter)						\
+		&& (ptr = container_of(iter, typeof(*(ptr)), name)); \
+	    (iter) = TAILQ_NEXT((iter), node))
+#endif
+
+#ifndef TAILQ_FOREACH_ENTRY_SAFE
+#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)	\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter) &&						\
+		(ptr = container_of(iter, typeof(*(ptr)), name)) &&\
+		((tvar) = TAILQ_NEXT((iter), node), 1);	\
+	    (iter) = (tvar))
+#endif
+
+/***********************/
+/*** List operations ***/
+/***********************/
+
+/**
+ * dlb2_list_init_head() - initialize the head of a list
+ * @head: list head
+ */
+static inline void dlb2_list_init_head(struct dlb2_list_head *head)
+{
+	TAILQ_INIT(head);
+}
+
+/**
+ * dlb2_list_add() - add an entry to a list
+ * @head: list head
+ * @entry: new list entry
+ */
+static inline void
+dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
+{
+	TAILQ_INSERT_TAIL(head, entry, node);
+}
+
+/**
+ * dlb2_list_del() - delete an entry from a list
+ * @entry: list entry
+ * @head: list head
+ */
+static inline void dlb2_list_del(struct dlb2_list_head *head,
+				 struct dlb2_list_entry *entry)
+{
+	TAILQ_REMOVE(head, entry, node);
+}
+
+/**
+ * dlb2_list_empty() - check if a list is empty
+ * @head: list head
+ *
+ * Return:
+ * Returns 1 if empty, 0 if not.
+ */
+static inline int dlb2_list_empty(struct dlb2_list_head *head)
+{
+	return TAILQ_EMPTY(head);
+}
+
+/**
+ * dlb2_list_splice() - splice a list
+ * @src_head: list to be added
+ * @ head: where src_head will be inserted
+ */
+static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
+				    struct dlb2_list_head *head)
+{
+	TAILQ_CONCAT(head, src_head, node);
+}
+
+/**
+ * DLB2_LIST_HEAD() - retrieve the head of the list
+ * @head: list head
+ * @type: type of the list variable
+ * @name: name of the list field within the containing struct
+ */
+#define DLB2_LIST_HEAD(head, type, name)                       \
+	(TAILQ_FIRST(&head) ?					\
+		container_of(TAILQ_FIRST(&head), type, name) :	\
+		NULL)
+
+/**
+ * DLB2_LIST_FOR_EACH() - iterate over a list
+ * @head: list head
+ * @ptr: pointer to struct containing a struct list
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ */
+#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
+	TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
+
+/**
+ * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
+ * an element is removed from the list while processing it.
+ * @ptr: pointer to struct containing a struct list
+ * @ptr_tmp: pointer to struct containing a struct list (temporary)
+ * @head: list head
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ * @iter_tmp: iterator variable (temporary)
+ */
+#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
+	TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
+
+#endif /*  __DLB2_OSDEP_LIST_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
new file mode 100644
index 0000000..0a48f7e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_TYPES_H
+#define __DLB2_OSDEP_TYPES_H
+
+#include <linux/types.h>
+
+#include <inttypes.h>
+#include <ctype.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Types for user mode PF PMD */
+typedef uint8_t         u8;
+typedef int8_t          s8;
+typedef uint16_t        u16;
+typedef int16_t         s16;
+typedef uint32_t        u32;
+typedef int32_t         s32;
+typedef uint64_t        u64;
+
+#define __iomem
+
+/* END types for user mode PF PMD */
+
+#endif /* __DLB2_OSDEP_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
new file mode 100644
index 0000000..43ecad4
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
@@ -0,0 +1,2527 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "dlb2_osdep_types.h"
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+	struct {
+		u32 isr_pend : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+	struct {
+		u32 vf0_reset_in_progress : 1;
+		u32 vf1_reset_in_progress : 1;
+		u32 vf2_reset_in_progress : 1;
+		u32 vf3_reset_in_progress : 1;
+		u32 vf4_reset_in_progress : 1;
+		u32 vf5_reset_in_progress : 1;
+		u32 vf6_reset_in_progress : 1;
+		u32 vf7_reset_in_progress : 1;
+		u32 vf8_reset_in_progress : 1;
+		u32 vf9_reset_in_progress : 1;
+		u32 vf10_reset_in_progress : 1;
+		u32 vf11_reset_in_progress : 1;
+		u32 vf12_reset_in_progress : 1;
+		u32 vf13_reset_in_progress : 1;
+		u32 vf14_reset_in_progress : 1;
+		u32 vf15_reset_in_progress : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+	struct {
+		u32 vec_mask : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+	struct {
+		u32 func_vf_bar_dis : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+	struct {
+		u32 total_vas : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+	struct {
+		u32 total_dir_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+	struct {
+		u32 total_ldb_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+	struct {
+		u32 total_dir_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+	struct {
+		u32 total_ldb_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+	struct {
+		u32 total_dir_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+	struct {
+		u32 total_ldb_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 meas : 1;
+		u32 debug : 7;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 cq_int_rearm : 1;
+		u32 dsi_error : 1;
+		u32 rsvd0 : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 rsvd0 : 3;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+	struct {
+		u32 qid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+	struct {
+		u32 vqid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+	struct {
+		u32 qid : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 debug : 8;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 isz : 1;
+		u32 dsi_error : 1;
+		u32 dlbrsvd : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 vf_synd0_parity : 1;
+		u32 vf_synd1_parity : 1;
+		u32 vf_synd2_parity : 1;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+	struct {
+		u32 sn_cfg_v : 1;
+		u32 fid_cfg_v : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS  0
+#define DLB2_CQ_ISR_MODE_MSI  1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI  3
+union dlb2_sys_ldb_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+	struct {
+		u32 keep_pf_ppid : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+	struct {
+		u32 illegal_hcw : 1;
+		u32 illegal_pp : 1;
+		u32 illegal_pasid : 1;
+		u32 illegal_qid : 1;
+		u32 disabled_qid : 1;
+		u32 illegal_ldb_qid_cfg : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+	struct {
+		u32 msix_0_ack : 1;
+		u32 msix_1_ack : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+	struct {
+		u32 msix_0_passthru : 1;
+		u32 msix_1_passthru : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED     0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+	struct {
+		u32 mode : 1;
+		u32 poll_mode : 1;
+		u32 poll_mask : 1;
+		u32 poll_lock : 1;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+	struct {
+		u32 cq : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 alarm : 1;
+		u32 cwd : 1;
+		u32 vf_pf_mb : 1;
+		u32 rsvd0 : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+	struct {
+		u32 qid_fid_limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+	struct {
+		u32 compress_code : 3;
+		u32 rsvd0 : 29;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+	(DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+	struct {
+		u32 count : 15;
+		u32 rsvd0 : 17;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+	struct {
+		u32 sn : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+	struct {
+		u32 mode : 3;
+		u32 slot : 4;
+		u32 rsvz0 : 1;
+		u32 grp : 1;
+		u32 rsvz1 : 1;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+	struct {
+		u32 en : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+	struct {
+		u32 write_pointer : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+	struct {
+		u32 base : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+	struct {
+		u32 limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+	struct {
+		u32 pop_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+	struct {
+		u32 push_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+	struct {
+		u32 depth : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+	struct {
+		u32 write_pointer : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 dlb_cor_alarm_enable : 1;
+		u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+		u32 cfg_64bytes_qe_dir_cq_mode : 1;
+		u32 pad_write_ldb : 1;
+		u32 pad_write_dir : 1;
+		u32 pad_first_write_ldb : 1;
+		u32 pad_first_write_dir : 1;
+		u32 rsvz0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+	struct {
+		u32 egress_credit_status_empty : 1;
+		u32 egress_credit_status_afull : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+		u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+		u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+		u32 chp_rop_pipe_credit_status_empty : 1;
+		u32 chp_rop_pipe_credit_status_afull : 1;
+		u32 qed_to_cq_pipe_credit_status_empty : 1;
+		u32 qed_to_cq_pipe_credit_status_afull : 1;
+		u32 egress_lsp_token_credit_status_empty : 1;
+		u32 egress_lsp_token_credit_status_afull : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 rsvz0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+	struct {
+		u32 sn_mode_0 : 3;
+		u32 rszv0 : 5;
+		u32 sn_mode_1 : 3;
+		u32 rszv1 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+	struct {
+		u32 unit_single_step_mode : 1;
+		u32 rr_en : 1;
+		u32 rszv0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+	struct {
+		u32 prio : 24;
+		u32 v : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+	struct {
+		u32 qid_p0 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p1 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p2 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p3 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+	struct {
+		u32 qid_p4 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p5 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p6 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p7 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+	struct {
+		u32 token_depth_select : 4;
+		u32 disable_wb_opt : 1;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+	struct {
+		u32 token_count : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+	struct {
+		u32 thresh : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+	(DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+	(DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+	struct {
+		u32 depth : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+	struct {
+		u32 cq : 8;
+		u32 qidix : 3;
+		u32 value : 1;
+		u32 nalb_haswork_v : 1;
+		u32 rlist_haswork_v : 1;
+		u32 slist_haswork_v : 1;
+		u32 inflight_ok_v : 1;
+		u32 aqed_nfull_v : 1;
+		u32 rsvz0 : 15;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+	struct {
+		u32 transfer : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+	struct {
+		u32 bw_range : 9;
+		u32 rsvz0 : 22;
+		u32 no_extra_credit : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+	struct {
+		u32 disab_atq_empty_arb : 1;
+		u32 inc_tok_unit_idle : 1;
+		u32 disab_rlist_pri : 1;
+		u32 inc_cmp_unit_idle : 1;
+		u32 rsvz0 : 2;
+		u32 dir_single_op : 1;
+		u32 dir_half_bw : 1;
+		u32 dir_single_out : 1;
+		u32 dir_disab_multi : 1;
+		u32 atq_single_op : 1;
+		u32 atq_half_bw : 1;
+		u32 atq_single_out : 1;
+		u32 atq_disab_multi : 1;
+		u32 dirrpl_single_op : 1;
+		u32 dirrpl_half_bw : 1;
+		u32 dirrpl_single_out : 1;
+		u32 lbrpl_single_op : 1;
+		u32 lbrpl_half_bw : 1;
+		u32 lbrpl_single_out : 1;
+		u32 ldb_single_op : 1;
+		u32 ldb_half_bw : 1;
+		u32 ldb_disab_multi : 1;
+		u32 atm_single_sch : 1;
+		u32 atm_single_cmp : 1;
+		u32 ldb_ce_tog_arb : 1;
+		u32 rsvz1 : 1;
+		u32 smon0_valid_sel : 2;
+		u32 smon0_value_sel : 1;
+		u32 smon0_compare_sel : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+	struct {
+		u32 chp_pf_reset_done : 1;
+		u32 rop_pf_reset_done : 1;
+		u32 lsp_pf_reset_done : 1;
+		u32 nalb_pf_reset_done : 1;
+		u32 ap_pf_reset_done : 1;
+		u32 dp_pf_reset_done : 1;
+		u32 qed_pf_reset_done : 1;
+		u32 dqed_pf_reset_done : 1;
+		u32 aqed_pf_reset_done : 1;
+		u32 sys_pf_reset_done : 1;
+		u32 pf_reset_active : 1;
+		u32 flrsm_state : 7;
+		u32 rsvd0 : 13;
+		u32 dlb_proc_reset_done : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+	struct {
+		u32 chp_pipeidle : 1;
+		u32 rop_pipeidle : 1;
+		u32 lsp_pipeidle : 1;
+		u32 nalb_pipeidle : 1;
+		u32 ap_pipeidle : 1;
+		u32 dp_pipeidle : 1;
+		u32 qed_pipeidle : 1;
+		u32 dqed_pipeidle : 1;
+		u32 aqed_pipeidle : 1;
+		u32 sys_pipeidle : 1;
+		u32 chp_unit_idle : 1;
+		u32 rop_unit_idle : 1;
+		u32 lsp_unit_idle : 1;
+		u32 nalb_unit_idle : 1;
+		u32 ap_unit_idle : 1;
+		u32 dp_unit_idle : 1;
+		u32 qed_unit_idle : 1;
+		u32 dqed_unit_idle : 1;
+		u32 aqed_unit_idle : 1;
+		u32 sys_unit_idle : 1;
+		u32 rsvd1 : 4;
+		u32 mstr_cfg_ring_idle : 1;
+		u32 mstr_cfg_mstr_idle : 1;
+		u32 mstr_flr_clkreq_b : 1;
+		u32 mstr_proc_idle : 1;
+		u32 mstr_proc_idle_masked : 1;
+		u32 rsvd0 : 2;
+		u32 dlb_func_idle : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+	struct {
+		u32 prochot : 1;
+		u32 pgcb_dlb_idle : 1;
+		u32 pgcb_dlb_pg_rdy_ack_b : 1;
+		u32 pmsm_pgcb_req_b : 1;
+		u32 pgbc_pmc_pg_req_b : 1;
+		u32 pmc_pgcb_pg_ack_b : 1;
+		u32 pmc_pgcb_fet_en_b : 1;
+		u32 pgcb_fet_en_b : 1;
+		u32 rsvz0 : 1;
+		u32 rsvz1 : 1;
+		u32 fuse_force_on : 1;
+		u32 fuse_proc_disable : 1;
+		u32 rsvz2 : 1;
+		u32 rsvz3 : 1;
+		u32 pm_fsm_d0tod3_ok : 1;
+		u32 pm_fsm_d3tod0_ok : 1;
+		u32 dlb_in_d3 : 1;
+		u32 rsvz4 : 7;
+		u32 pmsm : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+	struct {
+		u32 disable : 1;
+		u32 rsvz0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+	struct {
+		u32 isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+	struct {
+		u32 pf_isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+	struct {
+		u32 isr_pend : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+	struct {
+		u32 reset_in_progress : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+	struct {
+		u32 vf_msi_isr : 32;
+	} field;
+	u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
new file mode 100644
index 0000000..503fdf3
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
@@ -0,0 +1,1913 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_osdep_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_start_domain() - start a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: start domain arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int dlb2_hw_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp,
+			 bool vdev_request,
+			 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_request,
+		    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding unmap procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_finish_map_qid_procedures() - finish any pending map procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding map procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ *					interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+			  int port_id,
+			  bool is_ldb,
+			  bool vdev_request,
+			  unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+					 u32 *ldb_interrupts,
+					 u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+				 u32 *ldb_interrupts,
+				 u32 *dir_interrupts);
+
+/**
+ * dlb2_read_vf_intr_status() - read the VF interrupt status register
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function can be called from a VF's interrupt handler to determine
+ * which interrupts have fired. The first 31 bits correspond to CQ interrupt
+ * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
+ *
+ * Return:
+ * Returns a bit vector indicating which interrupt vectors are active.
+ */
+u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_intr_status() - ack VF interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function ACKs a VF's interrupts. Its interrupts argument should be the
+ * value returned by dlb2_read_vf_intr_status().
+ */
+void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function clears the VF's MSI interrupt pending register. Its interrupts
+ * argument should be contain the MSI vectors to ACK. For example, if MSI MME
+ * is in mode 0, then one bit 0 should ever be set.
+ */
+void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When done processing the PF mailbox request, this function unsets
+ * the PF's mailbox ISR register.
+ */
+void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
+
+/**
+ * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
+ *					vdevs
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the vdev->PF ISR fires, this function can be called to determine which
+ * vdev(s) are requesting service. This bitvector must be passed to
+ * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
+ * vdevs.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested service.
+ */
+u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ *
+ * When done processing all VF mailbox requests, this function unsets the VF's
+ * mailbox ISR register.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
+ *				    FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the VF FLR ISR fires, this function can be called to determine which
+ * VF(s) are requesting FLRs. This bitvector must passed to
+ * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
+ */
+u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF FLR requests, this function unsets the VF's FLR
+ * ISR register.
+ */
+void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF requests, this function communicates to the
+ * hardware that processing is complete.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
+			     u32 mbox_bitvec,
+			     u32 flr_bitvec);
+
+/**
+ * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the watchdog timer interrupt cause registers to
+ * determine which port(s) had a watchdog timeout, and notifies the
+ * application(s) that own the port(s).
+ */
+void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads and logs the alarm syndrome, then acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_request,
+			      unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of resources in use by the PF or a VF. It
+ * fills in the fields that args points to, except the following:
+ * - max_contiguous_atomic_inflights
+ * - max_contiguous_hist_list_entries
+ * - max_contiguous_ldb_credits
+ * - max_contiguous_dir_credits
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
+				   struct dlb2_get_num_resources_args *arg,
+				   bool vdev_request,
+				   unsigned int vdev_id);
+
+/**
+ * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
+ *				       the PF
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function sends a VF->PF mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the PF has processed
+ * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
+ * when the PF has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
+ *				 request
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * processing a VF->PF mailbox request. It should only be called after sending
+ * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vf_flr_complete() - check the status of a VF FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * executing the VF FLR. It should only be called after setting the VF's FLR
+ * bit.
+ */
+bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
+ *					vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function sends a PF->vdev mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the vdev has
+ * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
+ * determine when the vdev has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
+ *			       request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function returns a boolean indicating whether the vdev has finished
+ * processing a PF->vdev mailbox request. It should only be called after
+ * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
+			     unsigned int vf_id,
+			     void *data,
+			     int len);
+
+/**
+ * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
+			       unsigned int vf_id,
+			       void *data,
+			       int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_reset_vdev() - reset the hardware owned by a virtual device
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function resets the hardware owned by a vdev, by resetting the vdev's
+ * domains one by one.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function returns whether or not the vdev's resource assignments are
+ * locked. If locked, no resources can be added to or subtracted from the
+ * group.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_lock_vdev() - lock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sets a flag indicating that the vdev is using its resources.
+ * When the vdev is locked, its resource assignment cannot be changed.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_unlock_vdev() - unlock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function unlocks the vdev's resource assignment, allowing it to be
+ * modified.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of scheduling domains to assign to this vdev
+ *
+ * This function assigns num scheduling domains to the specified vdev. If the
+ * vdev already has domains assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB queues to assign to this vdev
+ *
+ * This function assigns num LDB queues to the specified vdev. If the vdev
+ * already has LDB queues assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports to the specified vdev. If the vdev
+ * already has LDB ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @cos: class-of-service ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports from class-of-service cos to the
+ * specified vdev. If the vdev already has LDB ports from this class-of-service
+ * assigned, this existing assignment is adjusted accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
+				   u32 id,
+				   u32 cos,
+				   u32 num);
+
+/**
+ * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR ports to assign to this vdev
+ *
+ * This function assigns num DIR ports to the specified vdev. If the vdev
+ * already has DIR ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB credit credits to assign to this vdev
+ *
+ * This function assigns num LDB credit to the specified vdev. If the vdev
+ * already has LDB credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR credits to assign to this vdev
+ *
+ * This function assigns num DIR credit to the specified vdev. If the vdev
+ * already has DIR credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of history list entries to assign to this vdev
+ *
+ * This function assigns num history list entries to the specified vdev. If the
+ * vdev already has history list entries assigned, this existing assignment is
+ * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
+ * this function may fail if a sufficiently large contiguous chunk is not
+ * available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of atomic inflights to assign to this vdev
+ *
+ * This function assigns num atomic inflights to the specified vdev. If the vdev
+ * already has atomic inflights assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of entries, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function takes any resources currently assigned to the vdev and
+ * reassigns them to the PF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_notify_vf() - send an alarm to a VF
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID
+ * @notification: notification
+ *
+ * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
+ *
+ * Return:
+ * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
+ */
+int dlb2_notify_vf(struct dlb2_hw *hw,
+		   unsigned int vf_id,
+		   u32 notification);
+
+/**
+ * dlb2_vdev_in_use() - query whether a virtual device is in use
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sends a mailbox request to the vdev to query whether the vdev
+ * is in use.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
+ * an internal error occurs.
+ */
+int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+enum dlb2_virt_mode {
+	DLB2_VIRT_NONE,
+	DLB2_VIRT_SRIOV,
+	DLB2_VIRT_SIOV,
+
+	/* NUM_DLB2_VIRT_MODES must be last */
+	NUM_DLB2_VIRT_MODES,
+};
+
+/**
+ * dlb2_hw_set_virt_mode() - set the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: either none, SR-IOV, or Scalable IOV.
+ *
+ * This function sets the virtualization mode of the device. This controls
+ * whether the device uses a software or hardware mailbox.
+ *
+ * This should be called by the PF driver when either SR-IOV or Scalable IOV is
+ * selected as the virtualization mechanism, and by the VF/VDEV driver during
+ * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
+ *
+ * Errors:
+ * EINVAL - Invalid mode.
+ */
+int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
+
+/**
+ * dlb2_hw_get_virt_mode() - get the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function gets the virtualization mode of the device.
+ */
+enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_register_sw_mbox() - register a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
+ * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
+ * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
+ * @inject_arg: user argument for pf2vdev_inject callback.
+ *
+ * When Scalable IOV is enabled, the VDCM must register a software mailbox for
+ * every virtual device during vdev creation.
+ *
+ * This function notifies the driver to use a software mailbox using the
+ * provided pointers, instead of the device's hardware mailbox. When the driver
+ * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
+ * go to the software mailbox instead of the hardware one. This is used in
+ * Scalable IOV virtualization.
+ */
+void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
+			      unsigned int vdev_id,
+			      u32 *vdev2pf_mbox,
+			      u32 *pf2vdev_mbox,
+			      void (*pf2vdev_inject)(void *),
+			      void *inject_arg);
+
+/**
+ * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function notifies the driver to stop using a previously registered
+ * software mailbox.
+ */
+void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ * @addr_lo: least-significant 32 bits of address.
+ * @data: 32 data bits.
+ *
+ * This sets up the CQ's IMS entry with the provided address and data values.
+ * This function should only be called if the device is configured for Scalable
+ * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
+ * not needed.
+ */
+void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb,
+				u32 addr_lo,
+				u32 data);
+
+/**
+ * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ *
+ * This clears the CQ's IMS entry, reverting it to its reset state.
+ */
+void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb);
+
+/**
+ * dlb2_hw_register_pasid() - register a vdev's PASID
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @pasid: the vdev's PASID.
+ *
+ * This function stores the user-supplied PASID, and uses it when configuring
+ * the vdev's CQs.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+int dlb2_hw_register_pasid(struct dlb2_hw *hw,
+			   unsigned int vdev_id,
+			   unsigned int pasid);
+
+/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vf_request,
+				unsigned int vf_id);
+
+/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ *	to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ *	port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ *	    cause the total bandwidth across all classes of service to exceed
+ *	    100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+enum dlb2_wd_tmo {
+	/* 40s watchdog timeout */
+	DLB2_WD_TMO_40S,
+	/* 10s watchdog timeout */
+	DLB2_WD_TMO_10S,
+	/* 1s watchdog timeout */
+	DLB2_WD_TMO_1S,
+
+	/* Must be last */
+	NUM_DLB2_WD_TMOS,
+};
+
+/**
+ * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
+ *	caller-specified timeout.
+ * @hw: dlb2_hw handle for a particular device.
+ * @tmo: watchdog timeout.
+ *
+ * This function should be called during device initialization and after reset.
+ * The watchdog timer interrupt must also be enabled per-CQ, using either
+ * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid timeout.
+ */
+int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
+
+/**
+ * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid directed port ID.
+ */
+int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid load-balanced port ID.
+ */
+int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *	ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * load-balanced CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_ldb_cq_interrupt().
+ */
+void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical directed port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * directed CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_dir_cq_interrupt().
+ */
+void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+#endif /* __DLB2_RESOURCE_H */
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
new file mode 100644
index 0000000..6cd0ddf
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -0,0 +1,621 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "base/dlb2_resource.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_regs.h"
+#include "dlb2_main.h"
+#include "../dlb2_user.h"
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+
+#define PF_ID_ZERO 0	/* PF ONLY! */
+#define NO_OWNER_VF 0	/* PF ONLY! */
+#define NOT_VF_REQ false /* PF ONLY! */
+
+#define DLB2_PCI_CFG_SPACE_SIZE 256
+#define DLB2_PCI_CAP_POINTER 0x34
+#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
+#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
+#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
+#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
+#define DLB2_PCI_EXT_CAP_ID_ERR 1
+#define DLB2_PCI_ERR_UNCOR_MASK 8
+#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
+
+#define DLB2_PCI_EXP_DEVCTL 8
+#define DLB2_PCI_LNKCTL 16
+#define DLB2_PCI_SLTCTL 24
+#define DLB2_PCI_RTCTL 28
+#define DLB2_PCI_EXP_DEVCTL2 40
+#define DLB2_PCI_LNKCTL2 48
+#define DLB2_PCI_SLTCTL2 56
+#define DLB2_PCI_CMD 4
+#define DLB2_PCI_X_CMD 2
+#define DLB2_PCI_EXP_DEVSTA 10
+#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
+#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
+
+#define DLB2_PCI_CAP_ID_EXP       0x10
+#define DLB2_PCI_CAP_ID_MSIX      0x11
+#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
+#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
+#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
+
+#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
+#define DLB2_PCI_PRI_ALLOC_REQ           0xC
+#define DLB2_PCI_PRI_CTRL                0x4
+#define DLB2_PCI_MSIX_FLAGS              0x2
+#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
+#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
+#define DLB2_PCI_ERR_ROOT_STATUS         0x30
+#define DLB2_PCI_ERR_COR_STATUS          0x10
+#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
+#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
+#define DLB2_PCI_ACS_CAP                 0x4
+#define DLB2_PCI_ACS_CTRL                0x6
+#define DLB2_PCI_ACS_SV                  0x1
+#define DLB2_PCI_ACS_RR                  0x4
+#define DLB2_PCI_ACS_CR                  0x8
+#define DLB2_PCI_ACS_UF                  0x10
+#define DLB2_PCI_ACS_EC                  0x20
+
+/* Stubs: Allow building partial probe patch */
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	int ret = 0;
+	RTE_SET_USED(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+/* End stubs */
+
+static int
+dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint32_t hdr;
+	size_t sz;
+	int pos;
+
+	pos = DLB2_PCI_CFG_SPACE_SIZE;
+	sz = sizeof(hdr);
+
+	while (pos > 0xFF) {
+		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
+			return -1;
+
+		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
+			return pos;
+
+		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint8_t pos;
+	int ret;
+	uint16_t hdr;
+
+	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
+	pos &= 0xFC;
+
+	if (ret != 1)
+		return -1;
+
+	while (pos > 0x3F) {
+		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
+		if (ret != 2)
+			return -1;
+
+		if (DLB2_PCI_CAP_ID(hdr) == id)
+			return pos;
+
+		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
+			return -1;
+
+		pos = DLB2_PCI_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+	rte_spinlock_init(&dlb2_dev->resource_mutex);
+	rte_spinlock_init(&dlb2_dev->measurement_lock);
+
+	return 0;
+}
+
+static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
+{
+	u32 retries = 0;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
+		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+		union dlb2_cfg_mstr_cfg_pm_status pm_st;
+		u32 addr;
+
+		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+			break;
+
+		rte_delay_ms(1);
+	};
+
+	if (retries == DLB2_READY_RETRY_LIMIT) {
+		DLB2_LOG_ERR("[%s()] wait for device ready timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+struct dlb2_dev *
+dlb2_probe(struct rte_pci_device *pdev)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret = 0;
+
+	DLB2_INFO(dlb2_dev, "probe\n");
+
+	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
+			      RTE_CACHE_LINE_SIZE);
+
+	if (dlb2_dev == NULL) {
+		ret = -ENOMEM;
+		goto dlb2_dev_malloc_fail;
+	}
+
+	/* PCI Bus driver has already mapped bar space into process.
+	 * Save off our IO register and FUNC addresses.
+	 */
+
+	/* BAR 0 */
+	if (pdev->mem_resource[0].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
+	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.func_kva,
+		  (void *)dlb2_dev->hw.func_phys_addr,
+		  (void *)(pdev->mem_resource[0].len));
+
+	/* BAR 2 */
+	if (pdev->mem_resource[2].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
+	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.csr_kva,
+		  (void *)dlb2_dev->hw.csr_phys_addr,
+		  (void *)(pdev->mem_resource[2].len));
+
+	dlb2_dev->pdev = pdev;
+
+	/* PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb2_pf_enable_pm(dlb2_dev);
+
+	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb2_pf_reset(dlb2_dev);
+	if (ret)
+		goto dlb2_reset_fail;
+
+	ret = dlb2_pf_init_driver_state(dlb2_dev);
+	if (ret)
+		goto init_driver_state_fail;
+
+	ret = dlb2_resource_init(&dlb2_dev->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	return dlb2_dev;
+
+resource_init_fail:
+	dlb2_resource_free(&dlb2_dev->hw);
+init_driver_state_fail:
+dlb2_reset_fail:
+pci_mmap_bad_addr:
+wait_for_device_ready_fail:
+	rte_free(dlb2_dev);
+dlb2_dev_malloc_fail:
+	rte_errno = ret;
+	return NULL;
+}
+
+int
+dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
+{
+	int ret = 0;
+	int i = 0;
+	uint32_t dword[16];
+	uint16_t cmd;
+	off_t off;
+
+	uint16_t dev_ctl_word;
+	uint16_t dev_ctl2_word;
+	uint16_t lnk_word;
+	uint16_t lnk_word2;
+	uint16_t slt_word;
+	uint16_t slt_word2;
+	uint16_t rt_ctl_word;
+	uint32_t pri_reqs_dword;
+	uint16_t pri_ctrl_word;
+
+	int pcie_cap_offset;
+	int pri_cap_offset;
+	int msix_cap_offset;
+	int err_cap_offset;
+	int acs_cap_offset;
+	int wait_count;
+
+	uint16_t devsta_busy_word;
+	uint16_t devctl_word;
+
+	struct rte_pci_device *pdev = dlb2_dev->pdev;
+
+	/* Save PCI config state */
+
+	for (i = 0; i < 16; i++) {
+		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
+			return ret;
+	}
+
+	pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
+
+	if (pcie_cap_offset < 0) {
+		DLB2_LOG_ERR("[%s()] failed to find the pcie capability\n",
+		       __func__);
+		return pcie_cap_offset;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
+		dev_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
+		lnk_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
+		slt_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_RTCTL;
+	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
+		rt_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
+		dev_ctl2_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
+		lnk_word2 = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
+		slt_word2 = 0;
+
+	off = DLB2_PCI_EXT_CAP_ID_PRI;
+	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (pri_cap_offset >= 0) {
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
+			pri_reqs_dword = 0;
+	}
+
+	/* clear the PCI command register before issuing the FLR */
+
+	off = DLB2_PCI_CMD;
+	cmd = 0;
+	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+		DLB2_LOG_ERR("[%s()] failed to write the pci command\n",
+		       __func__);
+		return ret;
+	}
+
+	/* issue the FLR */
+	for (wait_count = 0; wait_count < 4; wait_count++) {
+		int sleep_time;
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
+		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
+		if (ret != 2) {
+			DLB2_LOG_ERR("[%s()] failed to read the pci device status\n",
+			       __func__);
+			return ret;
+		}
+
+		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
+			break;
+
+		sleep_time = (1 << (wait_count)) * 100;
+		rte_delay_ms(sleep_time);
+	}
+
+	if (wait_count == 4) {
+		DLB2_LOG_ERR("[%s()] wait for pci pending transactions timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		DLB2_LOG_ERR("[%s()] failed to read the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
+
+	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		DLB2_LOG_ERR("[%s()] failed to write the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	rte_delay_ms(100);
+
+	/* Restore PCI config state */
+
+	if (pcie_cap_offset >= 0) {
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
+		if (ret != 2) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie device control at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
+		if (ret != 2) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
+		if (ret != 2) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_RTCTL;
+		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
+		if (ret != 2) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
+		if (ret != 2) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
+		if (ret != 2) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
+		if (ret != 2) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	if (pri_cap_offset >= 0) {
+		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
+
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
+		if (ret != 4) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
+		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
+		if (ret != 2) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ERR;
+	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (err_cap_offset >= 0) {
+		uint32_t tmp;
+
+		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	for (i = 16; i > 0; i--) {
+		off = (i - 1) * 4;
+		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
+		if (ret != 4) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_CMD;
+	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
+		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+			DLB2_LOG_ERR("[%s()] failed to write the pci command\n",
+			       __func__);
+			return ret;
+		}
+	}
+
+	msix_cap_offset = dlb2_pci_find_capability(pdev,
+						   DLB2_PCI_CAP_ID_MSIX);
+	if (msix_cap_offset >= 0) {
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
+			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				DLB2_LOG_ERR("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				DLB2_LOG_ERR("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ACS;
+	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (acs_cap_offset >= 0) {
+		uint16_t acs_cap, acs_ctrl, acs_mask;
+		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
+		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
+			acs_cap = 0;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
+		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
+		acs_ctrl |= (acs_cap & acs_mask);
+
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
+		acs_mask |= DLB2_PCI_ACS_EC;
+		acs_ctrl &= ~acs_mask;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
new file mode 100644
index 0000000..f082a94
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -0,0 +1,98 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+#endif
+
+#include "base/dlb2_hw_types.h"
+#include "../dlb2_user.h"
+
+#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
+
+struct dlb2_dev;
+
+struct dlb2_port_memory {
+	struct dlb2_list_head list;
+	void *cq_base;
+	bool valid;
+};
+
+struct dlb2_dev {
+	struct rte_pci_device *pdev;
+	struct dlb2_hw hw;
+	/* struct list_head list; */
+	struct device *dlb2_device;
+	bool domain_reset_failed;
+	/* The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	rte_spinlock_t resource_mutex;
+	rte_spinlock_t measurement_lock;
+	bool worker_launched;
+	u8 revision;
+};
+
+struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
+int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
+int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+
+#endif /* __DLB2_MAIN_H */
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
new file mode 100644
index 0000000..ae4d785
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -0,0 +1,196 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_devargs.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_pci.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+#include "dlb2_main.h"
+#include "base/dlb2_hw_types.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_resource.h"
+
+static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
+
+/* Stubs: Allow building partial probe patch */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(arg);
+	RTE_SET_USED(vdev_req);
+	RTE_SET_USED(vdev_id);
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+/* End stubs */
+
+static void
+dlb2_pf_iface_fn_ptrs_init(void)
+{
+/* flexible iface fn ptr assignments will go here */
+}
+
+/* PCI DEV HOOKS */
+static int
+dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
+{
+	int ret = 0;
+	struct rte_pci_device *pci_dev;
+	struct dlb2_devargs dlb2_args = {
+		.socket_id = rte_socket_id(),
+		.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+		.num_dir_credits_override = -1,
+		.qid_depth_thresholds = { {0} },
+		.cos_id = DLB2_COS_DEFAULT
+	};
+	struct dlb2_eventdev *dlb2;
+
+	DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
+		     eventdev->data->dev_id, eventdev->data->socket_id);
+
+	dlb2_pf_iface_fn_ptrs_init();
+
+	pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
+
+		/* Probe the DLB2 PF layer */
+		dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
+
+		if (dlb2->qm_instance.pf_dev == NULL) {
+			DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
+				     rte_errno);
+			ret = -rte_errno;
+			goto dlb2_probe_failed;
+		}
+
+		/* Were we invoked with runtime parameters? */
+		if (pci_dev->device.devargs) {
+			ret = dlb2_parse_params(pci_dev->device.devargs->args,
+						pci_dev->device.devargs->name,
+						&dlb2_args);
+			if (ret) {
+				DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
+					     ret, rte_errno);
+				goto dlb2_probe_failed;
+			}
+		}
+
+		ret = dlb2_primary_eventdev_probe(eventdev,
+						  event_dlb2_pf_name,
+						  &dlb2_args);
+	} else {
+		ret = dlb2_secondary_eventdev_probe(eventdev,
+						    event_dlb2_pf_name);
+	}
+	if (ret)
+		goto dlb2_probe_failed;
+
+	DLB2_LOG_INFO("DLB2 PF Probe success\n");
+
+	return 0;
+
+dlb2_probe_failed:
+
+	DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
+
+	return ret;
+}
+
+#define EVENTDEV_INTEL_VENDOR_ID 0x8086
+
+static const struct rte_pci_id pci_id_dlb2_map[] = {
+	{
+		RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
+			       PCI_DEVICE_ID_INTEL_DLB2_PF)
+	},
+	{
+		.vendor_id = 0,
+	},
+};
+
+static int
+event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
+		     struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
+					     sizeof(struct dlb2_eventdev),
+					     dlb2_eventdev_pci_init,
+					     event_dlb2_pf_name);
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+}
+
+static int
+event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_remove(pci_dev, NULL);
+
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+
+}
+
+static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
+	.id_table = pci_id_dlb2_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe = event_dlb2_pci_probe,
+	.remove = event_dlb2_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
-- 
2.6.4

[dpdk-dev] [PATCH v11 07/23] event/dlb2: add flexible interface

[Timothy McDaniel]

This commit introduces the flexible interface. This
interface allows the core code to operate in PF mode (direct
hardware access) or bifurcated mode (hardware configured via
kernel driver). This driver currently only supports PF mode
but bifurcated mode will be added in a future DPDK patch-set.
Note that the flexible interface is not used for data path
operations, and thus there are no performance concerns
related to the use of function pointers.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_iface.c | 28 ++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.h | 29 +++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build  |  1 +
 3 files changed, 58 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h

diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
new file mode 100644
index 0000000..0fc9991
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+
+#include "dlb2_priv.h"
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+void (*dlb2_iface_low_level_io_init)(void);
+
+int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+				     uint8_t *revision);
+
+void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+				   enum dlb2_cq_poll_modes *mode);
+
+int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
new file mode 100644
index 0000000..4fb416e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_IFACE_H_
+#define _DLB2_IFACE_H_
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+extern void (*dlb2_iface_low_level_io_init)(void);
+
+extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+					    uint8_t *revision);
+
+extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+					  enum dlb2_cq_poll_modes *mode);
+
+extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
+
+#endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 2855dd5..c2fb347 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -8,6 +8,7 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
 endif
 
 sources = files('dlb2.c',
+		'dlb2_iface.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c'
 )
-- 
2.6.4

[dpdk-dev] [PATCH v11 08/23] event/dlb2: add probe-time hardware init

[Timothy McDaniel]

This commit adds probe-time low level hardware
initialization.  It also adds probe-time init for both
primary and secondary DPDK processes.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 188 +++++++++++++++++++-
 drivers/event/dlb2/meson.build             |   3 +-
 drivers/event/dlb2/pf/base/dlb2_resource.c | 274 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  23 +--
 drivers/event/dlb2/pf/dlb2_main.h          |   1 -
 drivers/event/dlb2/pf/dlb2_pf.c            |  78 ++++++--
 6 files changed, 523 insertions(+), 44 deletions(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 1b11979..16653db 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -31,6 +31,7 @@
 #include <rte_string_fns.h>
 
 #include "dlb2_priv.h"
+#include "dlb2_iface.h"
 #include "dlb2_inline_fns.h"
 
 /*
@@ -40,10 +41,108 @@
 #if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
 #error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
 #endif
+static struct rte_event_dev_info evdev_dlb2_default_info = {
+	.driver_name = "", /* probe will set */
+	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
+	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
+	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
+#else
+	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
+#endif
+	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
+	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
+	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
+	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
+	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
+	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+	.max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
+	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
+			  RTE_EVENT_DEV_CAP_EVENT_QOS |
+			  RTE_EVENT_DEV_CAP_BURST_MODE |
+			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
+			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
+			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
+};
 
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/* override defaults with value(s) provided on command line */
+static void
+dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
+				 int *qid_depth_thresholds)
+{
+	int q;
+
+	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
+		if (qid_depth_thresholds[q] != 0)
+			dlb2->ev_queues[q].depth_threshold =
+				qid_depth_thresholds[q];
+	}
+}
+
+static int
+dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
+	int ret;
+
+	/* Query driver resources provisioned for this device */
+
+	ret = dlb2_iface_get_num_resources(handle,
+					   &dlb2->hw_rsrc_query_results);
+	if (ret) {
+		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
+		return ret;
+	}
+
+	/* Complete filling in device resource info returned to evdev app,
+	 * overriding any default values.
+	 * The capabilities (CAPs) were set at compile time.
+	 */
+
+	evdev_dlb2_default_info.max_event_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	evdev_dlb2_default_info.max_event_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	evdev_dlb2_default_info.max_num_events =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	/* Save off values used when creating the scheduling domain. */
+
+	handle->info.num_sched_domains =
+		dlb2->hw_rsrc_query_results.num_sched_domains;
+
+	handle->info.hw_rsrc_max.nb_events_limit =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	handle->info.hw_rsrc_max.num_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues +
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.num_ldb_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	handle->info.hw_rsrc_max.num_ldb_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	handle->info.hw_rsrc_max.num_dir_ports =
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.reorder_window_size =
+		dlb2->hw_rsrc_query_results.num_hist_list_entries;
+
+	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
+
+	return 0;
+}
+
 #define DLB2_BASE_10 10
 
 static int
@@ -235,14 +334,71 @@ set_qid_depth_thresh(const char *key __rte_unused,
 	return 0;
 }
 
+static void
+dlb2_entry_points_init(struct rte_eventdev *dev)
+{
+	RTE_SET_USED(dev);
+
+	/* Eventdev PMD entry points */
+}
+
 int
 dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    const char *name,
 			    struct dlb2_devargs *dlb2_args)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
-	RTE_SET_USED(dlb2_args);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	dlb2->event_dev = dev; /* backlink */
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	dlb2->max_num_events_override = dlb2_args->max_num_events;
+	dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
+	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
+					    &dlb2->revision);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_hardware_init(&dlb2->qm_instance);
+
+	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
+			     err);
+		return err;
+	}
+
+	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	dlb2_init_queue_depth_thresholds(dlb2,
+					 dlb2_args->qid_depth_thresholds.val);
 
 	return 0;
 }
@@ -251,8 +407,30 @@ int
 dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
 			      const char *name)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index c2fb347..d7390ec 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -10,7 +10,8 @@ endif
 sources = files('dlb2.c',
 		'dlb2_iface.c',
 		'pf/dlb2_main.c',
-		'pf/dlb2_pf.c'
+		'pf/dlb2_pf.c',
+		'pf/base/dlb2_resource.c'
 )
 
 headers = files()
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
new file mode 100644
index 0000000..6de8b95
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_mbox.h"
+#include "dlb2_osdep.h"
+#include "dlb2_osdep_bitmap.h"
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+	int i;
+
+	dlb2_list_init_head(&domain->used_ldb_queues);
+	dlb2_list_init_head(&domain->used_dir_pq_pairs);
+	dlb2_list_init_head(&domain->avail_ldb_queues);
+	dlb2_list_init_head(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+	int i;
+
+	dlb2_list_init_head(&rsrc->avail_domains);
+	dlb2_list_init_head(&rsrc->used_domains);
+	dlb2_list_init_head(&rsrc->avail_ldb_queues);
+	dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
+	arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
+	arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
+	arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb2_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	struct dlb2_list_entry *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. This is application
+	 * dependent, but the driver interleaves port IDs as much as possible
+	 * to reduce the likelihood of this. This initial allocation maximizes
+	 * the average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	/* Zero-out resource tracking data structures */
+	memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+	memset(&hw->pf, 0, sizeof(hw->pf));
+
+	dlb2_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+		dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+		dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_domains, list);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_ldb_queues, list);
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+		struct dlb2_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
+			      &port->func_list);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+				DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+	if (ret)
+		goto unwind;
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+					DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+		if (ret)
+			goto unwind;
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb2_resource_free(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+	r0.field.disable = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 6cd0ddf..7bed67f 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -19,6 +19,7 @@
 #include "dlb2_main.h"
 #include "../dlb2_user.h"
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 
 #define PF_ID_ZERO 0	/* PF ONLY! */
@@ -72,27 +73,6 @@
 #define DLB2_PCI_ACS_UF                  0x10
 #define DLB2_PCI_ACS_EC                  0x20
 
-/* Stubs: Allow building partial probe patch */
-void dlb2_resource_free(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-int dlb2_resource_init(struct dlb2_hw *hw)
-{
-	int ret = 0;
-	RTE_SET_USED(hw);
-
-	return ret;
-}
-
-void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-/* End stubs */
-
 static int
 dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
 {
@@ -149,7 +129,6 @@ static int
 dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
 {
 	rte_spinlock_init(&dlb2_dev->resource_mutex);
-	rte_spinlock_init(&dlb2_dev->measurement_lock);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index f082a94..f3bee71 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -38,7 +38,6 @@ struct dlb2_dev {
 	 * hardware registers.
 	 */
 	rte_spinlock_t resource_mutex;
-	rte_spinlock_t measurement_lock;
 	bool worker_launched;
 	u8 revision;
 };
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index ae4d785..7d64309 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -32,6 +32,7 @@
 #include <rte_string_fns.h>
 
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 #include "dlb2_main.h"
 #include "base/dlb2_hw_types.h"
@@ -40,35 +41,82 @@
 
 static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
 
-/* Stubs: Allow building partial probe patch */
-int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
-			      struct dlb2_get_num_resources_args *arg,
-			      bool vdev_req,
-			      unsigned int vdev_id)
+static void
+dlb2_pf_low_level_io_init(void)
+{
+	int i;
+	/* Addresses will be initialized at port create */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		/* First directed ports */
+		dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
+
+		/* Now load balanced ports */
+		dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
+	}
+}
+
+static int
+dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
+{
+	RTE_SET_USED(handle);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+static int
+dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
+			   uint8_t *revision)
 {
-	RTE_SET_USED(hw);
-	RTE_SET_USED(arg);
-	RTE_SET_USED(vdev_req);
-	RTE_SET_USED(vdev_id);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	*revision = dlb2_dev->revision;
 
 	return 0;
 }
 
-void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+static void
+dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+	dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_num_resources_args *rsrcs)
 {
-	RTE_SET_USED(hw);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
 }
 
-void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+static int
+dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
+			 enum dlb2_cq_poll_modes *mode)
 {
-	RTE_SET_USED(hw);
+	RTE_SET_USED(handle);
+
+	*mode = DLB2_CQ_POLL_MODE_SPARSE;
+
+	return 0;
 }
-/* End stubs */
 
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
-/* flexible iface fn ptr assignments will go here */
+	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
+	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
+	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
+	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
+	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v11 09/23] event/dlb2: add xstats

[Timothy McDaniel]

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
Reviewed-by: Chen, Mike Ximing <mike.ximing.chen@intel.com>
---
 drivers/event/dlb2/dlb2.c        |   35 +-
 drivers/event/dlb2/dlb2_xstats.c | 1235 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build   |    1 +
 3 files changed, 1268 insertions(+), 3 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 16653db..925824e 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,6 +70,21 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/*
+ * DUMMY - added so that xstats path will compile/link.
+ * Will be replaced by real version in a subsequent
+ * patch.
+ */
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	RTE_SET_USED(dlb2);
+	RTE_SET_USED(queue);
+
+	return 0;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -337,9 +352,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
-	RTE_SET_USED(dev);
-
-	/* Eventdev PMD entry points */
+	/* Expose PMD's eventdev interface */
+	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dump             = dlb2_eventdev_dump,
+		.xstats_get       = dlb2_eventdev_xstats_get,
+		.xstats_get_names = dlb2_eventdev_xstats_get_names,
+		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
+		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+	};
+
+	dev->dev_ops = &dlb2_eventdev_entry_ops;
 }
 
 int
@@ -391,6 +413,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Complete xtstats runtime initialization */
+	err = dlb2_xstats_init(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
+		return err;
+	}
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
new file mode 100644
index 0000000..a274779
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_xstats.c
@@ -0,0 +1,1235 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eventdev.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+enum dlb2_xstats_type {
+	/* common to device and port */
+	rx_ok,				/**< Receive an event */
+	rx_drop,                        /**< Error bit set in received QE */
+	rx_interrupt_wait,		/**< Wait on an interrupt */
+	rx_umonitor_umwait,		/**< Block using umwait */
+	tx_ok,				/**< Transmit an event */
+	total_polls,			/**< Call dequeue_burst */
+	zero_polls,			/**< Call dequeue burst and return 0 */
+	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
+	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
+	tx_nospc_inflight_max,		/**< Reach the new_event_threshold */
+	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
+	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
+	/* device specific */
+	nb_events_limit,
+	inflight_events,
+	ldb_pool_size,
+	dir_pool_size,
+	/* port specific */
+	tx_new,				/**< Send an OP_NEW event */
+	tx_fwd,				/**< Send an OP_FORWARD event */
+	tx_rel,				/**< Send an OP_RELEASE event */
+	tx_implicit_rel,		/**< Issue an implicit event release */
+	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED event */
+	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL event */
+	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event */
+	tx_sched_directed,		/**< Send a directed event */
+	tx_invalid,                     /**< Send an event with an invalid op */
+	outstanding_releases,		/**< # of releases a port owes */
+	max_outstanding_releases,	/**< max # of releases a port can owe */
+	rx_sched_ordered,		/**< Dequeue an ordered event */
+	rx_sched_unordered,		/**< Dequeue an unordered event */
+	rx_sched_atomic,		/**< Dequeue an atomic event */
+	rx_sched_directed,		/**< Dequeue an directed event */
+	rx_sched_invalid,               /**< Dequeue event sched type invalid */
+	/* common to port and queue */
+	is_configured,			/**< Port is configured */
+	is_load_balanced,		/**< Port is LDB */
+	hw_id,				/**< Hardware ID */
+	/* queue specific */
+	num_links,			/**< Number of ports linked */
+	sched_type,			/**< Queue sched type */
+	enq_ok,				/**< # events enqueued to the queue */
+	current_depth,			/**< Current queue depth */
+	depth_threshold,		/**< Programmed depth threshold */
+	depth_le50_threshold,
+	/**< Depth LE to 50% of the configured hardware threshold */
+	depth_gt50_le75_threshold,
+	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold */
+	depth_gt75_le100_threshold,
+	/**< Depth GT 75%. but LE to the configured hardware threshold */
+	depth_gt100_threshold
+	/**< Depth GT 100% of the configured hw threshold */
+};
+
+typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
+		uint16_t obj_idx, /* port or queue id */
+		enum dlb2_xstats_type stat, int extra_arg);
+
+enum dlb2_xstats_fn_type {
+	DLB2_XSTATS_FN_DEV,
+	DLB2_XSTATS_FN_PORT,
+	DLB2_XSTATS_FN_QUEUE
+};
+
+struct dlb2_xstats_entry {
+	struct rte_event_dev_xstats_name name;
+	uint64_t reset_value; /* an offset to be taken away to emulate resets */
+	enum dlb2_xstats_fn_type fn_id;
+	enum dlb2_xstats_type stat;
+	enum rte_event_dev_xstats_mode mode;
+	int extra_arg;
+	uint16_t obj_idx;
+	uint8_t reset_allowed; /* when set, this value can be reset */
+};
+
+/* Some device stats are simply a summation of the corresponding port values */
+static uint64_t
+dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
+			     int which_stat)
+{
+	int i;
+	uint64_t val = 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
+
+		if (!port->setup_done)
+			continue;
+
+		switch (which_stat) {
+		case rx_ok:
+			val += port->stats.traffic.rx_ok;
+			break;
+		case rx_drop:
+			val += port->stats.traffic.rx_drop;
+			break;
+		case rx_interrupt_wait:
+			val += port->stats.traffic.rx_interrupt_wait;
+			break;
+		case rx_umonitor_umwait:
+			val += port->stats.traffic.rx_umonitor_umwait;
+			break;
+		case tx_ok:
+			val += port->stats.traffic.tx_ok;
+			break;
+		case total_polls:
+			val += port->stats.traffic.total_polls;
+			break;
+		case zero_polls:
+			val += port->stats.traffic.zero_polls;
+			break;
+		case tx_nospc_ldb_hw_credits:
+			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
+			break;
+		case tx_nospc_dir_hw_credits:
+			val += port->stats.traffic.tx_nospc_dir_hw_credits;
+			break;
+		case tx_nospc_inflight_max:
+			val += port->stats.traffic.tx_nospc_inflight_max;
+			break;
+		case tx_nospc_new_event_limit:
+			val += port->stats.traffic.tx_nospc_new_event_limit;
+			break;
+		case tx_nospc_inflight_credits:
+			val += port->stats.traffic.tx_nospc_inflight_credits;
+			break;
+		default:
+			return -1;
+		}
+	}
+	return val;
+}
+
+static uint64_t
+get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
+	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	switch (type) {
+	case rx_ok:
+	case rx_drop:
+	case rx_interrupt_wait:
+	case rx_umonitor_umwait:
+	case tx_ok:
+	case total_polls:
+	case zero_polls:
+	case tx_nospc_ldb_hw_credits:
+	case tx_nospc_dir_hw_credits:
+	case tx_nospc_inflight_max:
+	case tx_nospc_new_event_limit:
+	case tx_nospc_inflight_credits:
+		return dlb2_device_traffic_stat_get(dlb2, type);
+	case nb_events_limit:
+		return dlb2->new_event_limit;
+	case inflight_events:
+		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
+	case ldb_pool_size:
+		return dlb2->num_ldb_credits;
+	case dir_pool_size:
+		return dlb2->num_dir_credits;
+	default: return -1;
+	}
+}
+
+static uint64_t
+get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
+
+	switch (type) {
+	case rx_ok: return ev_port->stats.traffic.rx_ok;
+
+	case rx_drop: return ev_port->stats.traffic.rx_drop;
+
+	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
+
+	case rx_umonitor_umwait:
+		return ev_port->stats.traffic.rx_umonitor_umwait;
+
+	case tx_ok: return ev_port->stats.traffic.tx_ok;
+
+	case total_polls: return ev_port->stats.traffic.total_polls;
+
+	case zero_polls: return ev_port->stats.traffic.zero_polls;
+
+	case tx_nospc_ldb_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
+
+	case tx_nospc_dir_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
+
+	case tx_nospc_inflight_max:
+		return ev_port->stats.traffic.tx_nospc_inflight_max;
+
+	case tx_nospc_new_event_limit:
+		return ev_port->stats.traffic.tx_nospc_new_event_limit;
+
+	case tx_nospc_inflight_credits:
+		return ev_port->stats.traffic.tx_nospc_inflight_credits;
+
+	case is_configured: return ev_port->setup_done;
+
+	case is_load_balanced: return !ev_port->qm_port.is_directed;
+
+	case hw_id: return ev_port->qm_port.id;
+
+	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
+
+	case tx_fwd: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
+
+	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
+
+	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
+
+	case tx_sched_ordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case tx_sched_unordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case tx_sched_atomic:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case tx_sched_directed:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case tx_invalid: return ev_port->stats.tx_invalid;
+
+	case outstanding_releases: return ev_port->outstanding_releases;
+
+	case max_outstanding_releases:
+		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	case rx_sched_ordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case rx_sched_unordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case rx_sched_atomic:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case rx_sched_directed:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
+
+	default: return -1;
+	}
+}
+
+static uint64_t
+dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
+{
+	int port = 0;
+	uint64_t tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
+
+	return tally;
+}
+
+static uint64_t
+dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
+{
+	int port = 0;
+	uint64_t enq_ok_tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
+
+	return enq_ok_tally;
+}
+
+static uint64_t
+get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_queue *ev_queue =
+		&dlb2->ev_queues[obj_idx];
+
+	switch (type) {
+	case is_configured: return ev_queue->setup_done;
+
+	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
+
+	case hw_id: return ev_queue->qm_queue.id;
+
+	case num_links: return ev_queue->num_links;
+
+	case sched_type: return ev_queue->qm_queue.sched_type;
+
+	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
+
+	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
+
+	case depth_threshold: return ev_queue->depth_threshold;
+
+	case depth_le50_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_LE50);
+
+	case depth_gt50_le75_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT50_LE75);
+
+	case depth_gt75_le100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT75_LE100);
+
+	case depth_gt100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT100);
+
+	default: return -1;
+	}
+}
+
+int
+dlb2_xstats_init(struct dlb2_eventdev *dlb2)
+{
+	/*
+	 * define the stats names and types. Used to build up the device
+	 * xstats array
+	 * There are multiple set of stats:
+	 *   - device-level,
+	 *   - per-port,
+	 *   - per-qid,
+	 *
+	 * For each of these sets, we have three parallel arrays, one for the
+	 * names, the other for the stat type parameter to be passed in the fn
+	 * call to get that stat. The third array allows resetting or not.
+	 * All these arrays must be kept in sync
+	 */
+	static const char * const dev_stats[] = {
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"nb_events_limit",
+		"inflight_events",
+		"ldb_pool_size",
+		"dir_pool_size",
+	};
+	static const enum dlb2_xstats_type dev_types[] = {
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		nb_events_limit,
+		inflight_events,
+		ldb_pool_size,
+		dir_pool_size,
+	};
+	/* Note: generated device stats are not allowed to be reset. */
+	static const uint8_t dev_reset_allowed[] = {
+		0, /* rx_ok */
+		0, /* rx_drop */
+		0, /* rx_interrupt_wait */
+		0, /* rx_umonitor_umwait */
+		0, /* tx_ok */
+		0, /* total_polls */
+		0, /* zero_polls */
+		0, /* tx_nospc_ldb_hw_credits */
+		0, /* tx_nospc_dir_hw_credits */
+		0, /* tx_nospc_inflight_max */
+		0, /* tx_nospc_new_event_limit */
+		0, /* tx_nospc_inflight_credits */
+		0, /* nb_events_limit */
+		0, /* inflight_events */
+		0, /* ldb_pool_size */
+		0, /* dir_pool_size */
+	};
+	static const char * const port_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"tx_new",
+		"tx_fwd",
+		"tx_rel",
+		"tx_implicit_rel",
+		"tx_sched_ordered",
+		"tx_sched_unordered",
+		"tx_sched_atomic",
+		"tx_sched_directed",
+		"tx_invalid",
+		"outstanding_releases",
+		"max_outstanding_releases",
+		"rx_sched_ordered",
+		"rx_sched_unordered",
+		"rx_sched_atomic",
+		"rx_sched_directed",
+		"rx_sched_invalid"
+	};
+	static const enum dlb2_xstats_type port_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		tx_new,
+		tx_fwd,
+		tx_rel,
+		tx_implicit_rel,
+		tx_sched_ordered,
+		tx_sched_unordered,
+		tx_sched_atomic,
+		tx_sched_directed,
+		tx_invalid,
+		outstanding_releases,
+		max_outstanding_releases,
+		rx_sched_ordered,
+		rx_sched_unordered,
+		rx_sched_atomic,
+		rx_sched_directed,
+		rx_sched_invalid
+	};
+	static const uint8_t port_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		1, /* rx_ok */
+		1, /* rx_drop */
+		1, /* rx_interrupt_wait */
+		1, /* rx_umonitor_umwait */
+		1, /* tx_ok */
+		1, /* total_polls */
+		1, /* zero_polls */
+		1, /* tx_nospc_ldb_hw_credits */
+		1, /* tx_nospc_dir_hw_credits */
+		1, /* tx_nospc_inflight_max */
+		1, /* tx_nospc_new_event_limit */
+		1, /* tx_nospc_inflight_credits */
+		1, /* tx_new */
+		1, /* tx_fwd */
+		1, /* tx_rel */
+		1, /* tx_implicit_rel */
+		1, /* tx_sched_ordered */
+		1, /* tx_sched_unordered */
+		1, /* tx_sched_atomic */
+		1, /* tx_sched_directed */
+		1, /* tx_invalid */
+		0, /* outstanding_releases */
+		0, /* max_outstanding_releases */
+		1, /* rx_sched_ordered */
+		1, /* rx_sched_unordered */
+		1, /* rx_sched_atomic */
+		1, /* rx_sched_directed */
+		1  /* rx_sched_invalid */
+	};
+
+	/* QID specific stats */
+	static const char * const qid_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"num_links",
+		"sched_type",
+		"enq_ok",
+		"current_depth",
+		"depth_threshold",
+		"depth_le50_threshold",
+		"depth_gt50_le75_threshold",
+		"depth_gt75_le100_threshold",
+		"depth_gt100_threshold",
+	};
+	static const enum dlb2_xstats_type qid_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		num_links,
+		sched_type,
+		enq_ok,
+		current_depth,
+		depth_threshold,
+		depth_le50_threshold,
+		depth_gt50_le75_threshold,
+		depth_gt75_le100_threshold,
+		depth_gt100_threshold,
+	};
+	static const uint8_t qid_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		0, /* num_links */
+		0, /* sched_type */
+		1, /* enq_ok */
+		0, /* current_depth */
+		0, /* depth_threshold */
+		1, /* depth_le50_threshold */
+		1, /* depth_gt50_le75_threshold */
+		1, /* depth_gt75_le100_threshold */
+		1, /* depth_gt100_threshold */
+	};
+
+	/* ---- end of stat definitions ---- */
+
+	/* check sizes, since a missed comma can lead to strings being
+	 * joined by the compiler.
+	 */
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
+
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_reset_allowed));
+
+	/* other vars */
+	const unsigned int count = RTE_DIM(dev_stats) +
+			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
+			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
+	unsigned int i, port, qid, stat_id = 0;
+
+	dlb2->xstats = rte_zmalloc_socket(NULL,
+			sizeof(dlb2->xstats[0]) * count, 0,
+			dlb2->qm_instance.info.socket_id);
+	if (dlb2->xstats == NULL)
+		return -ENOMEM;
+
+#define sname dlb2->xstats[stat_id].name.name
+	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
+		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
+			.fn_id = DLB2_XSTATS_FN_DEV,
+			.stat = dev_types[i],
+			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
+			.reset_allowed = dev_reset_allowed[i],
+		};
+		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
+	}
+	dlb2->xstats_count_mode_dev = stat_id;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
+		dlb2->xstats_offset_for_port[port] = stat_id;
+
+		uint32_t count_offset = stat_id;
+
+		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_PORT,
+				.obj_idx = port,
+				.stat = port_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_PORT,
+				.reset_allowed = port_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "port_%u_%s",
+				 port, port_stats[i]);
+		}
+
+		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_port = stat_id - dlb2->xstats_count_mode_dev;
+
+	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
+		uint32_t count_offset = stat_id;
+
+		dlb2->xstats_offset_for_qid[qid] = stat_id;
+
+		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_QUEUE,
+				.obj_idx = qid,
+				.stat = qid_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
+				.reset_allowed = qid_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "qid_%u_%s",
+				 qid, qid_stats[i]);
+		}
+
+		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_queue = stat_id -
+		(dlb2->xstats_count_mode_dev + dlb2->xstats_count_mode_port);
+#undef sname
+
+	dlb2->xstats_count = stat_id;
+
+	return 0;
+}
+
+void
+dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
+{
+	rte_free(dlb2->xstats);
+	dlb2->xstats_count = 0;
+}
+
+int
+dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstats_names,
+		unsigned int *ids, unsigned int size)
+{
+	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+	uint32_t start_offset = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			break;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			break;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
+		break;
+	default:
+		return -EINVAL;
+	};
+
+	if (xstats_mode_count > size || ids == NULL || xstats_names == NULL)
+		return xstats_mode_count;
+
+	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
+		if (dlb2->xstats[i].mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != dlb2->xstats[i].obj_idx)
+			continue;
+
+		xstats_names[xidx] = dlb2->xstats[i].name;
+		if (ids)
+			ids[xidx] = start_offset + xidx;
+		xidx++;
+	}
+	return xidx;
+}
+
+static int
+dlb2_xstats_update(struct dlb2_eventdev *dlb2,
+		enum rte_event_dev_xstats_mode mode,
+		uint8_t queue_port_id, const unsigned int ids[],
+		uint64_t values[], unsigned int n, const uint32_t reset)
+{
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			goto invalid_value;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			goto invalid_value;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		break;
+	default:
+		goto invalid_value;
+	};
+
+	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
+		dlb2_xstats_fn fn;
+
+		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != xs->obj_idx)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			goto invalid_value;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+
+		if (values)
+			values[xidx] = val;
+
+		if (xs->reset_allowed && reset)
+			xs->reset_value += val;
+
+		xidx++;
+	}
+
+	return xidx;
+
+invalid_value:
+	return -EINVAL;
+}
+
+int
+dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	const uint32_t reset = 0;
+
+	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
+				  reset);
+}
+
+uint64_t
+dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+				 const char *name, unsigned int *id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	dlb2_xstats_fn fn;
+
+	for (i = 0; i < dlb2->xstats_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (strncmp(xs->name.name, name,
+			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
+			if (id != NULL)
+				*id = i;
+
+			switch (xs->fn_id) {
+			case DLB2_XSTATS_FN_DEV:
+				fn = get_dev_stat;
+				break;
+			case DLB2_XSTATS_FN_PORT:
+				fn = get_port_stat;
+				break;
+			case DLB2_XSTATS_FN_QUEUE:
+				fn = get_queue_stat;
+				break;
+			default:
+				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
+					  xs->fn_id);
+				return (uint64_t)-1;
+			}
+
+			return fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+		}
+	}
+	if (id != NULL)
+		*id = (uint32_t)-1;
+	return (uint64_t)-1;
+}
+
+static void
+dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
+			uint32_t num)
+{
+	uint32_t i;
+	dlb2_xstats_fn fn;
+
+	for (i = start; i < start + num; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (!xs->reset_allowed)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			return;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
+		xs->reset_value = val;
+	}
+}
+
+static int
+dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
+			const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_QUEUE,
+					queue_id, ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	if (ids == NULL)
+		dlb2_xstats_reset_range(dlb2,
+			dlb2->xstats_offset_for_qid[queue_id],
+			dlb2->xstats_count_per_qid[queue_id]);
+
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
+		       const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+	int offset = dlb2->xstats_offset_for_port[port_id];
+	int nb_stat = dlb2->xstats_count_per_port[port_id];
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_PORT, port_id,
+					ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
+		      uint32_t nb_ids)
+{
+	uint32_t i;
+
+	if (ids) {
+		for (i = 0; i < nb_ids; i++) {
+			uint32_t id = ids[i];
+
+			if (id >= dlb2->xstats_count_mode_dev)
+				return -EINVAL;
+			dlb2_xstats_reset_range(dlb2, id, 1);
+		}
+	} else {
+		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
+			dlb2_xstats_reset_range(dlb2, i, 1);
+	}
+
+	return 0;
+}
+
+int
+dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+			   enum rte_event_dev_xstats_mode mode,
+			   int16_t queue_port_id,
+			   const uint32_t ids[],
+			   uint32_t nb_ids)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	uint32_t i;
+
+	/* handle -1 for queue_port_id here, looping over all ports/queues */
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
+			return -EINVAL;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+				if (dlb2_xstats_reset_port(dlb2, i,
+							   ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
+			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
+						   ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
+				if (dlb2_xstats_reset_queue(dlb2, i,
+							    ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
+			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
+						    ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	};
+
+	return 0;
+}
+
+void
+dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	int i;
+
+	if (f == NULL) {
+		DLB2_LOG_ERR("Invalid file pointer\n");
+		return;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (dlb2 == NULL) {
+		fprintf(f, "DLB2 Event device cannot be dumped!\n");
+		return;
+	}
+
+	if (!dlb2->configured)
+		fprintf(f, "DLB2 Event device is not configured\n");
+
+	handle = &dlb2->qm_instance;
+
+	fprintf(f, "================\n");
+	fprintf(f, "DLB2 Device Dump\n");
+	fprintf(f, "================\n");
+
+	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
+		dlb2->umwait_allowed ? "yes" : "no");
+
+	/* Generic top level device information */
+
+	fprintf(f, "device is configured and run state =");
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		fprintf(f, "STOPPED\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
+		fprintf(f, "STOPPING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
+		fprintf(f, "STARTING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
+		fprintf(f, "STARTED\n");
+	else
+		fprintf(f, "UNEXPECTED\n");
+
+	fprintf(f, "domain ID=%u, socket_id=%u, evdev=%p\n",
+		handle->domain_id, handle->info.socket_id, dlb2->event_dev);
+
+	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
+		dlb2->num_dir_ports, dlb2->num_dir_queues);
+
+	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
+		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
+
+	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
+		handle->cfg.resources.num_atomic_inflights,
+		handle->cfg.resources.num_hist_list_entries);
+
+	fprintf(f, "results from most recent hw resource query:\n");
+
+	fprintf(f, "\tnum_sched_domains = %u\n",
+		dlb2->hw_rsrc_query_results.num_sched_domains);
+
+	fprintf(f, "\tnum_ldb_queues = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_queues);
+
+	fprintf(f, "\tnum_ldb_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_ports);
+
+	fprintf(f, "\tnum_dir_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_ports);
+
+	fprintf(f, "\tnum_atomic_inflights = %u\n",
+		dlb2->hw_rsrc_query_results.num_atomic_inflights);
+
+	fprintf(f, "\tnum_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.num_hist_list_entries);
+
+	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
+
+	fprintf(f, "\tnum_ldb_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_credits);
+
+	fprintf(f, "\tnum_dir_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_credits);
+
+	/* Port level information */
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
+		int j;
+
+		if (!p->enq_configured)
+			fprintf(f, "Port_%d is not configured\n", i);
+
+		fprintf(f, "Port_%d\n", i);
+		fprintf(f, "=======\n");
+
+		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
+			p->id, p->setup_done);
+
+		fprintf(f, "\tconfig state=%d, port state=%d\n",
+			p->qm_port.config_state, p->qm_port.state);
+
+		fprintf(f, "\tport is %s\n",
+			p->qm_port.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\toutstanding releases=%u\n",
+			p->outstanding_releases);
+
+		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
+			p->inflight_max, p->inflight_credits);
+
+		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
+			p->credit_update_quanta, p->implicit_release);
+
+		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (p->link[j].valid)
+				fprintf(f, "id=%u prio=%u ",
+					p->link[j].queue_id,
+					p->link[j].priority);
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
+
+		fprintf(f, "\tcached_ldb_credits=%u\n",
+			p->qm_port.cached_ldb_credits);
+
+		fprintf(f, "\tldb_credits = %u\n",
+			p->qm_port.ldb_credits);
+
+		fprintf(f, "\tcached_dir_credits = %u\n",
+			p->qm_port.cached_dir_credits);
+
+		fprintf(f, "\tdir_credits = %u\n",
+			p->qm_port.dir_credits);
+
+		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
+			p->qm_port.gen_bit,
+			p->qm_port.cq_idx,
+			p->qm_port.cq_depth);
+
+		fprintf(f, "\tinterrupt armed=%d\n",
+			p->qm_port.int_armed);
+
+		fprintf(f, "\tPort statistics\n");
+
+		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
+			p->stats.traffic.rx_ok);
+
+		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
+			p->stats.traffic.rx_drop);
+
+		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
+			p->stats.traffic.rx_interrupt_wait);
+
+		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
+			p->stats.traffic.rx_umonitor_umwait);
+
+		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
+			p->stats.traffic.tx_ok);
+
+		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
+			p->stats.traffic.total_polls);
+
+		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
+			p->stats.traffic.zero_polls);
+
+		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_ldb_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_dir_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_max);
+
+		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_new_event_limit);
+
+		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_credits);
+
+		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
+
+		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
+
+		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
+
+		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
+			p->stats.tx_implicit_rel);
+
+		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
+			p->stats.tx_invalid);
+
+		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
+			p->stats.rx_sched_invalid);
+	}
+
+	/* Queue level information */
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
+		int j, k;
+
+		if (!q->setup_done)
+			fprintf(f, "Queue_%d is not configured\n", i);
+
+		fprintf(f, "Queue_%d\n", i);
+		fprintf(f, "========\n");
+
+		fprintf(f, "\tevqueue_%u is set up\n", q->id);
+
+		fprintf(f, "\tqueue is %s\n",
+			q->qm_queue.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
+
+		for (j = 0; j < dlb2->num_ports; j++) {
+			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
+
+			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++) {
+				if (p->link[k].valid &&
+				    p->link[k].queue_id == q->id)
+					fprintf(f, "id=%u prio=%u ",
+						p->id, p->link[k].priority);
+			}
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\tcurrent depth: %u events\n",
+			dlb2_get_queue_depth(dlb2, q));
+
+		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
+			q->qm_queue.num_qid_inflights, q->qm_queue.sched_type);
+	}
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index d7390ec..a80d7ab 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files('dlb2.c',
 		'dlb2_iface.c',
+		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c'
-- 
2.6.4

[dpdk-dev] [PATCH v11 10/23] event/dlb2: add infos get and configure

[Timothy McDaniel]

Add support for configuring the DLB2 hardware.
In particular, this patch configures the DLB2
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  116 +
 drivers/event/dlb2/dlb2.c                  |  313 +++
 drivers/event/dlb2/dlb2_iface.c            |    5 +
 drivers/event/dlb2/dlb2_iface.h            |    4 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 3237 ++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |   15 +
 drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
 7 files changed, 3734 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index e3091c9..5f6c486 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -32,9 +32,125 @@ detailed understanding of the hardware, but these details are important when
 writing high-performance code. This section describes the places where the
 eventdev API and DLB2 misalign.
 
+Scheduling Domain Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are 32 scheduling domainis the DLB2.
+When one is configured, it allocates load-balanced and
+directed queues, ports, credits, and other hardware resources. Some
+resource allocations are user-controlled -- the number of queues, for example
+-- and others, like credit pools (one directed and one load-balanced pool per
+scheduling domain), are not.
+
+The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
+setup argument and the per-port ``new_event_threshold`` argument apply as
+defined in the eventdev header file. The limit is applied to all enqueues,
+regardless of whether it will consume a directed or load-balanced credit.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Reconfiguration
+~~~~~~~~~~~~~~~
+
+The Eventdev API allows one to reconfigure a device, its ports, and its queues
+by first stopping the device, calling the configuration function(s), then
+restarting the device. The DLB2 does not support configuring an individual queue
+or port without first reconfiguring the entire device, however, so there are
+certain reconfiguration sequences that are valid in the eventdev API but not
+supported by the PMD.
+
+Specifically, the PMD supports the following configuration sequence:
+1. Configure and start the device
+2. Stop the device
+3. (Optional) Reconfigure the device
+4. (Optional) If step 3 is run:
+
+   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
+   b. The reconfigured port(s) lose their previous queue links.
+
+5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
+6. Restart the device. If the device is reconfigured in step 3 but one or more
+   of its ports or queues are not, the PMD will apply their previous
+   configuration (including port->queue links) at this time.
+
+The PMD does not support the following configuration sequences:
+1. Configure and start the device
+2. Stop the device
+3. Setup queue or setup port
+4. Start the device
+
+This sequence is not supported because the event device must be reconfigured
+before its ports or queues can be.
+
+Atomic Inflights Allocation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
+inflight event in a temporary buffer that is divided among load-balanced
+queues. If a queue's atomic buffer storage fills up, this can result in
+head-of-line-blocking. For example:
+
+- An LDB queue allocated N atomic buffer entries
+- All N entries are filled with events from flow X, which is pinned to CQ 0.
+
+Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
+scheduled. The likelihood of this case depends on the eventdev configuration,
+traffic behavior, event processing latency, potential for a worker to be
+interrupted or otherwise delayed, etc.
+
+By default, the PMD allocates 16 buffer entries for each load-balanced queue,
+which provides an even division across all 128 queues but potentially wastes
+buffer space (e.g. if not all queues are used, or aren't used for atomic
+scheduling).
+
+The PMD provides a dev arg to override the default per-queue allocation. To
+increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,atm_inflights=64
+
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 925824e..6798e66 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -85,6 +85,25 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
 	return 0;
 }
 
+static void
+dlb2_free_qe_mem(struct dlb2_port *qm_port)
+{
+	if (qm_port == NULL)
+		return;
+
+	rte_free(qm_port->qe4);
+	qm_port->qe4 = NULL;
+
+	rte_free(qm_port->int_arm_qe);
+	qm_port->int_arm_qe = NULL;
+
+	rte_free(qm_port->consume_qe);
+	qm_port->consume_qe = NULL;
+
+	rte_memzone_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz);
+	dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz = NULL;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -350,10 +369,304 @@ set_qid_depth_thresh(const char *key __rte_unused,
 }
 
 static void
+dlb2_eventdev_info_get(struct rte_eventdev *dev,
+		       struct rte_event_dev_info *dev_info)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret;
+
+	ret = dlb2_hw_query_resources(dlb2);
+	if (ret) {
+		const struct rte_eventdev_data *data = dev->data;
+
+		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+			     ret, data->dev_id);
+		/* fn is void, so fall through and return values set up in
+		 * probe
+		 */
+	}
+
+	/* Add num resources currently owned by this domain.
+	 * These would become available if the scheduling domain were reset due
+	 * to the application recalling eventdev_configure to *reconfigure* the
+	 * domain.
+	 */
+	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
+	evdev_dlb2_default_info.max_event_queues += dlb2->num_ldb_queues;
+	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
+
+	evdev_dlb2_default_info.max_event_queues =
+		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
+			RTE_EVENT_MAX_QUEUES_PER_DEV);
+
+	evdev_dlb2_default_info.max_num_events =
+		RTE_MIN(evdev_dlb2_default_info.max_num_events,
+			dlb2->max_num_events_override);
+
+	*dev_info = evdev_dlb2_default_info;
+}
+
+static int
+dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
+			    const struct dlb2_hw_rsrcs *resources_asked)
+{
+	int ret = 0;
+	struct dlb2_create_sched_domain_args *cfg;
+
+	if (resources_asked == NULL) {
+		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
+		ret = EINVAL;
+		goto error_exit;
+	}
+
+	/* Map generic qm resources to dlb2 resources */
+	cfg = &handle->cfg.resources;
+
+	/* DIR ports and queues */
+
+	cfg->num_dir_ports = resources_asked->num_dir_ports;
+
+	cfg->num_dir_credits = resources_asked->num_dir_credits;
+
+	/* LDB queues */
+
+	cfg->num_ldb_queues = resources_asked->num_ldb_queues;
+
+	/* LDB ports */
+
+	cfg->cos_strict = 0; /* Best effort */
+	cfg->num_cos_ldb_ports[0] = 0;
+	cfg->num_cos_ldb_ports[1] = 0;
+	cfg->num_cos_ldb_ports[2] = 0;
+	cfg->num_cos_ldb_ports[3] = 0;
+
+	switch (handle->cos_id) {
+	case DLB2_COS_0:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[0] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_1:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[1] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_2:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[2] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_3:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[3] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_DEFAULT:
+		/* all ldb ports are don't care ports from a cos perspective */
+		cfg->num_ldb_ports =
+			resources_asked->num_ldb_ports;
+		break;
+	}
+
+	cfg->num_ldb_credits =
+		resources_asked->num_ldb_credits;
+
+	cfg->num_atomic_inflights =
+		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
+		cfg->num_ldb_queues;
+
+	cfg->num_hist_list_entries = resources_asked->num_ldb_ports *
+		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d\n",
+		     cfg->num_ldb_queues,
+		     resources_asked->num_ldb_ports,
+		     cfg->num_dir_ports,
+		     cfg->num_atomic_inflights,
+		     cfg->num_hist_list_entries,
+		     cfg->num_ldb_credits,
+		     cfg->num_dir_credits);
+
+	/* Configure the QM */
+
+	ret = dlb2_iface_sched_domain_create(handle, cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: domain create failed, ret = %d, extra status: %s\n",
+			     ret,
+			     dlb2_error_strings[cfg->response.status]);
+
+		goto error_exit;
+	}
+
+	handle->domain_id = cfg->response.id;
+	handle->cfg.configured = true;
+
+error_exit:
+
+	return ret;
+}
+
+static void
+dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	enum dlb2_configuration_state config_state;
+	int i, j;
+
+	dlb2_iface_domain_reset(dlb2);
+
+	/* Free all dynamically allocated port memory */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
+
+	/* If reconfiguring, mark the device's queues and ports as "previously
+	 * configured." If the user doesn't reconfigure them, the PMD will
+	 * reapply their previous configuration when the device is started.
+	 */
+	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
+		DLB2_NOT_CONFIGURED;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		dlb2->ev_ports[i].qm_port.config_state = config_state;
+		/* Reset setup_done so ports can be reconfigured */
+		dlb2->ev_ports[i].setup_done = false;
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			dlb2->ev_ports[i].link[j].mapped = false;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++)
+		dlb2->ev_queues[i].qm_queue.config_state = config_state;
+
+	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
+		dlb2->ev_queues[i].setup_done = false;
+
+	dlb2->num_ports = 0;
+	dlb2->num_ldb_ports = 0;
+	dlb2->num_dir_ports = 0;
+	dlb2->num_queues = 0;
+	dlb2->num_ldb_queues = 0;
+	dlb2->num_dir_queues = 0;
+	dlb2->configured = false;
+}
+
+/* Note: 1 QM instance per QM device, QM instance/device == event device */
+static int
+dlb2_eventdev_configure(const struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
+	const struct rte_eventdev_data *data = dev->data;
+	const struct rte_event_dev_config *config = &data->dev_conf;
+	int ret;
+
+	/* If this eventdev is already configured, we must release the current
+	 * scheduling domain before attempting to configure a new one.
+	 */
+	if (dlb2->configured) {
+		dlb2_hw_reset_sched_domain(dev, true);
+
+		ret = dlb2_hw_query_resources(dlb2);
+		if (ret) {
+			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+				     ret, data->dev_id);
+			return ret;
+		}
+	}
+
+	if (config->nb_event_queues > rsrcs->num_queues) {
+		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_queues,
+			     rsrcs->num_queues);
+		return -EINVAL;
+	}
+	if (config->nb_event_ports > (rsrcs->num_ldb_ports
+			+ rsrcs->num_dir_ports)) {
+		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_ports,
+			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
+		return -EINVAL;
+	}
+	if (config->nb_events_limit > rsrcs->nb_events_limit) {
+		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_events_limit,
+			     rsrcs->nb_events_limit);
+		return -EINVAL;
+	}
+
+	if (config->event_dev_cfg & RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT)
+		dlb2->global_dequeue_wait = false;
+	else {
+		uint32_t timeout32;
+
+		dlb2->global_dequeue_wait = true;
+
+		/* note size mismatch of timeout vals in eventdev lib. */
+		timeout32 = config->dequeue_timeout_ns;
+
+		dlb2->global_dequeue_wait_ticks =
+			timeout32 * (rte_get_timer_hz() / 1E9);
+	}
+
+	/* Does this platform support umonitor/umwait? */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
+		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
+			DLB2_LOG_ERR("invalid value (%d) for RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
+				     RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
+			return -EINVAL;
+		}
+		dlb2->umwait_allowed = true;
+	}
+
+	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
+	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs->num_dir_ports;
+	/* 1 dir queue per dir port */
+	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports;
+
+	/* Scale down nb_events_limit by 4 for directed credits, since there
+	 * are 4x as many load-balanced credits.
+	 */
+	rsrcs->num_ldb_credits = 0;
+	rsrcs->num_dir_credits = 0;
+
+	if (rsrcs->num_ldb_queues)
+		rsrcs->num_ldb_credits = config->nb_events_limit;
+	if (rsrcs->num_dir_ports)
+		rsrcs->num_dir_credits = config->nb_events_limit / 4;
+	if (dlb2->num_dir_credits_override != -1)
+		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
+
+	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
+		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
+		return -ENODEV;
+	}
+
+	dlb2->new_event_limit = config->nb_events_limit;
+	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
+
+	/* Save number of ports/queues for this event dev */
+	dlb2->num_ports = config->nb_event_ports;
+	dlb2->num_queues = config->nb_event_queues;
+	dlb2->num_dir_ports = rsrcs->num_dir_ports;
+	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
+	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
+	dlb2->num_dir_queues = dlb2->num_dir_ports;
+	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
+	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
+	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
+	dlb2->max_dir_credits = rsrcs->num_dir_credits;
+
+	dlb2->configured = true;
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dev_infos_get    = dlb2_eventdev_info_get,
+		.dev_configure    = dlb2_eventdev_configure,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 0fc9991..a829b9b 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -26,3 +26,8 @@ int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
+
+int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				struct dlb2_create_sched_domain_args *args);
+
+void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4fb416e..6663dab 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -26,4 +26,8 @@ extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
 
+extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				 struct dlb2_create_sched_domain_args *args);
+
+extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 6de8b95..63a68a6 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -12,6 +12,27 @@
 #include "dlb2_regs.h"
 #include "dlb2_resource.h"
 
+#include "../../dlb2_priv.h"
+#include "../../dlb2_inline_fns.h"
+
+#define DLB2_DOM_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, domain_list, iter)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, func_list, iter)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, domain_list, it, it_tmp)
+
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, func_list, it, it_tmp)
+
 static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
 {
 	int i;
@@ -272,3 +293,3219 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
 
 	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
 }
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+	union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+	r0.field.count = domain->num_ldb_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+	r1.field.count = domain->num_dir_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+		       struct dlb2_function_resources *rsrcs,
+		       u32 domain_id,
+		       u32 cos_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	RTE_SET_USED(iter);
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+				   typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				   struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_ports,
+				   u32 cos_id,
+				   struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_ldb_port *port;
+
+		port = dlb2_get_next_ldb_port(hw, rsrcs,
+					      domain->id.phys_id, cos_id);
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_ports[cos_id],
+			      &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_ports[cos_id],
+			      &port->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_create_sched_domain_args *args,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      num,
+						      i,
+						      resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+					ret = __dlb2_attach_ldb_ports(hw,
+								      rsrcs,
+								      domain,
+								      1,
+								      cos_id,
+								      resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret < 0)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      1,
+						      j,
+						      resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 u32 num_ports,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+					   typeof(*port));
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_dir_pq_pairs, &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_dir_pq_pairs, &port->domain_list);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+					struct dlb2_hw_domain *domain,
+					u32 num_atomic_inflights,
+					struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+				     struct dlb2_hw_domain *domain,
+				     u32 num_hist_list_entries,
+				     struct dlb2_cmd_response *resp)
+{
+	struct dlb2_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb2_bitmap_find_set_bit_range(bitmap,
+						      num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+				  struct dlb2_function_resources *rsrcs,
+				  struct dlb2_hw_domain *domain,
+				  u32 num_queues,
+				  struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb2_ldb_queue *queue;
+
+		queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+					    typeof(*queue));
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_queues, &queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_queues, &queue->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+			     struct dlb2_function_resources *rsrcs,
+			     struct dlb2_hw_domain *domain,
+			     struct dlb2_create_sched_domain_args *args,
+			     struct dlb2_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb2_attach_ldb_queues(hw,
+				     rsrcs,
+				     domain,
+				     args->num_ldb_queues,
+				     resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_ports(hw,
+				    rsrcs,
+				    domain,
+				    args,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_ports(hw,
+				    rsrcs,
+				    domain,
+				    args->num_dir_ports,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_credits(rsrcs,
+				      domain,
+				      args->num_ldb_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_credits(rsrcs,
+				      domain,
+				      args->num_dir_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+						   domain,
+						   args->num_hist_list_entries,
+						   resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_atomic_inflights(rsrcs,
+					   domain,
+					   args->num_atomic_inflights,
+					   resp);
+	if (ret < 0)
+		return ret;
+
+	dlb2_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+				  struct dlb2_create_sched_domain_args *args,
+				  struct dlb2_cmd_response *resp)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb2_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+				  struct dlb2_create_sched_domain_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		    args->num_ldb_queues);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		    args->num_ldb_ports);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		    args->num_cos_ldb_ports[0]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		    args->cos_strict);
+	DLB2_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		    args->num_dir_ports);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		    args->num_atomic_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		    args->num_hist_list_entries);
+	DLB2_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		    args->num_ldb_credits);
+	DLB2_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		    args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ *	domain and its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+	if (ret)
+		return ret;
+
+	domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available domains\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (domain->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_domains contains configured domains.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	dlb2_init_domain_rsrc_lists(domain);
+
+	ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to verify args.\n",
+			    __func__);
+
+		return ret;
+	}
+
+	dlb2_list_del(&rsrcs->avail_domains, &domain->func_list);
+
+	dlb2_list_add(&rsrcs->used_domains, &domain->func_list);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+	DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+			     struct dlb2_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb2_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	int ret;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_disable(hw, port);
+
+		ret = dlb2_drain_dir_cq(hw, port);
+		if (ret < 0)
+			return ret;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_dir_pq_pair *queue)
+{
+	union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *queue)
+{
+	return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+				      struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		dlb2_movdir64b(pp_addr, hcw);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int ret, i;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			if (toggle_port)
+				dlb2_ldb_port_cq_disable(hw, port);
+
+			ret = dlb2_drain_ldb_cq(hw, port);
+			if (ret < 0)
+				return ret;
+
+			if (toggle_port)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+
+	return 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_atm_active r1;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_ldb_queue *queue)
+{
+	return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb2_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+					   struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to unmap domain queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = true;
+
+			dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+			   u32 id,
+			   bool vdev_req,
+			   unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iter1) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter2)
+			if (queue->id.virt_id == id)
+				return queue;
+	}
+
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue, iter1)
+		if (queue->id.virt_id == id)
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+						      u32 id,
+						      bool vdev_req,
+						      unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iteration;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	RTE_SET_USED(iteration);
+
+	if (id >= DLB2_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iteration)
+		if (domain->id.virt_id == id)
+			return domain;
+
+	return NULL;
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot,
+					   enum dlb2_qid_map_state new_state)
+{
+	enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb2_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB2_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB2_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB2_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB2_HW_DBG(hw,
+		    "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB2_HW_ERR(hw,
+		    "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+				enum dlb2_qid_map_state state,
+				int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+				      enum dlb2_qid_map_state state,
+				      struct dlb2_ldb_queue *queue,
+				      int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+					      struct dlb2_hw_domain *domain,
+					      struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain,
+					     struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+						struct dlb2_ldb_port *port,
+						int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 0;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 1;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+					struct dlb2_ldb_port *p,
+					struct dlb2_ldb_queue *q,
+					u8 priority)
+{
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_lsp_cq2qid0 r1;
+	union dlb2_atm_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix_00 r3;
+	union dlb2_lsp_qid2cqidix2_00 r4;
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+	    !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+	r0.field.v |= 1 << i;
+	r0.field.prio |= (priority & 0x7) << i * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+	else
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		r1.field.qid_p0 = q->id.phys_id;
+	if (i == 1 || i == 5)
+		r1.field.qid_p1 = q->id.phys_id;
+	if (i == 2 || i == 6)
+		r1.field.qid_p2 = q->id.phys_id;
+	if (i == 3 || i == 7)
+		r1.field.qid_p3 = q->id.phys_id;
+
+	if (i < 4)
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+	else
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r4.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+						  p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		r2.field.cq_p0 |= 1 << i;
+		r3.field.cq_p0 |= 1 << i;
+		r4.field.cq_p0 |= 1 << i;
+		break;
+
+	case 1:
+		r2.field.cq_p1 |= 1 << i;
+		r3.field.cq_p1 |= 1 << i;
+		r4.field.cq_p1 |= 1 << i;
+		break;
+
+	case 2:
+		r2.field.cq_p2 |= 1 << i;
+		r3.field.cq_p2 |= 1 << i;
+		r4.field.cq_p2 |= 1 << i;
+		break;
+
+	case 3:
+		r2.field.cq_p3 |= 1 << i;
+		r3.field.cq_p3 |= 1 << i;
+		r4.field.cq_p3 |= 1 << i;
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r3.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		    r4.val);
+
+	dlb2_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB2_QUEUE_MAPPED;
+
+	return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	/* Set the atomic scheduling haswork bit */
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.rlist_haswork_v = r0.field.count > 0;
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      u8 slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.rlist_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.nalb_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+					      struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+	r0.field.limit = queue->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+						struct dlb2_ldb_queue *queue)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+						struct dlb2_hw_domain *domain,
+						struct dlb2_ldb_port *port,
+						struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_lsp_qid_ldb_infl_cnt r0;
+	enum dlb2_qid_map_state state;
+	int slot, ret, i;
+	u8 prio;
+	RTE_SET_USED(iter);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: non-zero QID inflight count\n",
+			    __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			state = DLB2_QUEUE_MAPPED;
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+					 struct dlb2_ldb_port *port,
+					 struct dlb2_ldb_queue *queue,
+					 u8 priority)
+{
+	union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	int domain_id, slot, ret;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+		DLB2_HW_ERR(hw,
+			    "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		if (port->enabled)
+			dlb2_ldb_port_cq_enable(hw, port);
+
+		dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain,
+					struct dlb2_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		union dlb2_lsp_qid_ldb_infl_cnt r0;
+		struct dlb2_ldb_queue *queue;
+		int qid;
+
+		if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: unable to find queue %d\n",
+				    __func__, qid);
+			continue;
+		}
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count)
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb2_ldb_port_cq_disable(hw, port);
+
+		dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count) {
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+
+			dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port,
+				   struct dlb2_ldb_queue *queue)
+{
+	enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_atm_qid2cqidix_00 r1;
+	union dlb2_lsp_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix2_00 r3;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB2_QUEUE_MAPPED;
+	in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: QID %d isn't mapped\n",
+			    __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+	r0.field.v &= ~(1 << i);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		r1.field.cq_p0 &= ~(1 << i);
+		r2.field.cq_p0 &= ~(1 << i);
+		r3.field.cq_p0 &= ~(1 << i);
+		break;
+
+	case 1:
+		r1.field.cq_p1 &= ~(1 << i);
+		r2.field.cq_p1 &= ~(1 << i);
+		r3.field.cq_p1 &= ~(1 << i);
+		break;
+
+	case 2:
+		r1.field.cq_p2 &= ~(1 << i);
+		r2.field.cq_p2 &= ~(1 << i);
+		r3.field.cq_p2 &= ~(1 << i);
+		break;
+
+	case 3:
+		r1.field.cq_p3 &= ~(1 << i);
+		r2.field.cq_p3 &= ~(1 << i);
+		r3.field.cq_p3 &= ~(1 << i);
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+		    r1.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+		    r3.val);
+
+	dlb2_flush_csr(hw);
+
+	unmapped = DLB2_QUEUE_UNMAPPED;
+
+	return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_ldb_port *port,
+				 struct dlb2_ldb_queue *queue,
+				 u8 prio)
+{
+	if (domain->started)
+		return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   int slot)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb2_ldb_port_qid_map *map;
+		struct dlb2_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain,
+					  struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (r0.field.count > 0)
+		return false;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = false;
+
+			dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_dir_vpp_v r1;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		unsigned int offs;
+		u32 virt_id;
+
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+	}
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_ldb_vpp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			unsigned int offs;
+			u32 virt_id;
+
+			if (hw->virt_mode == DLB2_VIRT_SRIOV)
+				virt_id = port->id.virt_id;
+			else
+				virt_id = port->id.phys_id;
+
+			offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+			DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+	union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+				    r0.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+				    r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+	union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+			    r0.val);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+			    r1.val);
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+		union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+		union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+				    r1.val);
+
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID_V(idx),
+				    r2.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID2QID(idx),
+				    r3.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID_V(idx),
+				    r1.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID2QID(idx),
+				    r2.val);
+		}
+	}
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+					       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_sn_chk_enbl r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.en = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			int i;
+
+			for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+				if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					    __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		port->enabled = false;
+
+		dlb2_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	union dlb2_sys_dir_pp_v r1;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+			    r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_ldb_pp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_ldb_queue *queue;
+	int i;
+	RTE_SET_USED(iter);
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty ldb queue %d\n",
+				    __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port, iter) {
+			if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb2_ldb_cq_token_count(hw, ldb_port)) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to empty ldb port %d\n",
+					    __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir queue %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb2_dir_cq_token_count(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir port %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						   struct dlb2_ldb_port *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP2PP(offs),
+			    DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP_V(offs),
+			    DLB2_SYS_VF_LDB_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    DLB2_SYS_LDB_PP_V_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_DSBL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_BASE_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID0(port->id.phys_id),
+		    DLB2_LSP_CQ2QID0_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID1(port->id.phys_id),
+		    DLB2_LSP_CQ2QID1_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+		    DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			__dlb2_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+				       struct dlb2_dir_pq_pair *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_DSBL_RST);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_FMT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP2PP(offs),
+			    DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP_V(offs),
+			    DLB2_SYS_VF_DIR_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		__dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+			    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_ITS(queue_id),
+			    DLB2_SYS_LDB_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN(queue_id),
+			    DLB2_CHP_ORD_QID_SN_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+			    DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_V(queue_id),
+			    DLB2_SYS_LDB_QID_V_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_CFG_V(queue_id),
+			    DLB2_SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB2_CSR_WR(hw,
+				    offs[queue->sn_group],
+				    DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX2(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX2_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_ATM_QID2CQIDIX(queue_id, i),
+				    DLB2_ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb2_domain_reset_dir_port_registers(hw, domain);
+
+	dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_dir_pq_pair *tmp_dir_port;
+	struct dlb2_ldb_queue *tmp_ldb_queue;
+	struct dlb2_ldb_port *tmp_ldb_port;
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_list_head *list;
+	int ret, i;
+	RTE_SET_USED(tmp_dir_port);
+	RTE_SET_USED(tmp_ldb_queue);
+	RTE_SET_USED(tmp_ldb_port);
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list = &domain->used_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb2_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		dlb2_list_del(&domain->used_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list = &domain->avail_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		ldb_queue->owned = false;
+
+		dlb2_list_del(&domain->avail_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		list = &domain->used_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB2_QUEUE_UNMAPPED;
+
+			dlb2_list_del(&domain->used_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list = &domain->avail_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			ldb_port->owned = false;
+
+			dlb2_list_del(&domain->avail_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list = &domain->used_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		dlb2_list_del(&domain->used_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list = &domain->avail_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				    domain->hist_list_entry_base,
+				    domain->total_hist_list_entries);
+	if (ret) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			    __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	dlb2_list_del(&rsrcs->used_domains, &domain->func_list);
+	dlb2_list_add(&rsrcs->avail_domains, &domain->func_list);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain,
+					    struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_ldb_port *port;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (!dlb2_list_empty(&domain->used_ldb_ports[i]))
+			break;
+	}
+
+	if (i == DLB2_NUM_COS_DOMAINS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: No configured LDB ports\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb2_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+	 * prior to this in dlb2_domain_drain_mapped_queues().
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings != 0 ||
+		    dlb2_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ *	hardware resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain  == NULL || !domain->configured)
+		return -EINVAL;
+
+	/* Disable VPPs */
+	if (vdev_req) {
+		dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+		dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+	}
+
+	/* Disable CQ interrupts */
+	dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+	dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+	dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb2_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_mapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	dlb2_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb2_domain_disable_dir_cqs(hw, domain);
+
+	/* Disable PPs */
+	dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+	ret = dlb2_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Reset the QID and port state. */
+	dlb2_domain_reset_registers(hw, domain);
+
+	/* Hardware reset complete. Reset the domain's software state */
+	ret = dlb2_domain_reset_software_state(hw, domain);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 7bed67f..bd364d4 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -598,3 +598,18 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
 
 	return 0;
 }
+
+int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+			    struct dlb2_create_sched_domain_args *args,
+			    struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_sched_domain(hw, args, resp, NOT_VF_REQ,
+					   PF_ID_ZERO);
+}
+
+int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 7d64309..9310f72 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -108,15 +108,59 @@ dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
 	return 0;
 }
 
+static int
+dlb2_pf_sched_domain_create(struct dlb2_hw_dev *handle,
+			    struct dlb2_create_sched_domain_args *arg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (dlb2_dev->domain_reset_failed) {
+		response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = dlb2_pf_create_sched_domain(&dlb2_dev->hw, arg, &response);
+	if (ret)
+		goto done;
+
+done:
+
+	arg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static void
+dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret;
+
+	dlb2_dev = (struct dlb2_dev *)dlb2->qm_instance.pf_dev;
+	ret = dlb2_pf_reset_domain(&dlb2_dev->hw, dlb2->qm_instance.domain_id);
+	if (ret)
+		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
 	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
 	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_domain_reset = dlb2_pf_domain_reset;
 	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
 	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v11 11/23] event/dlb2: add queue and port default conf

[Timothy McDaniel]

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 6798e66..ef1c000 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -661,12 +661,42 @@ dlb2_eventdev_configure(const struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
+				    uint8_t port_id,
+				    struct rte_event_port_conf *port_conf)
+{
+	RTE_SET_USED(port_id);
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	port_conf->new_event_threshold = dlb2->new_event_limit;
+	port_conf->dequeue_depth = 32;
+	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
+	port_conf->event_port_cfg = 0;
+}
+
+static void
+dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
+				     uint8_t queue_id,
+				     struct rte_event_queue_conf *queue_conf)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(queue_id);
+
+	queue_conf->nb_atomic_flows = 1024;
+	queue_conf->nb_atomic_order_sequences = 64;
+	queue_conf->event_queue_cfg = 0;
+	queue_conf->priority = 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v11 12/23] event/dlb2: add queue setup

[Timothy McDaniel]

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  73 +++--
 drivers/event/dlb2/dlb2.c                  | 312 +++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |  12 +
 drivers/event/dlb2/dlb2_iface.h            |  12 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 464 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  82 +++++
 7 files changed, 926 insertions(+), 39 deletions(-)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 5f6c486..b154cac 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -47,45 +47,40 @@ setup argument and the per-port ``new_event_threshold`` argument apply as
 defined in the eventdev header file. The limit is applied to all enqueues,
 regardless of whether it will consume a directed or load-balanced credit.
 
-Load-balanced and Directed Ports
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
-does not have the same concept, but it has a similar one: ports and queues that
-are singly-linked (i.e. linked to a single queue or port, respectively).
-
-The ``rte_event_dev_info_get()`` function reports the number of available
-event ports and queues (among other things). For the DLB2 PMD, max_event_ports
-and max_event_queues report the number of available load-balanced ports and
-queues, and max_single_link_event_port_queue_pairs reports the number of
-available directed ports and queues.
-
-When a scheduling domain is created in ``rte_event_dev_configure()``, the user
-specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
-control the total number of ports (load-balanced and directed) and the number
-of directed ports. Hence, the number of requested load-balanced ports is
-``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
-specifies the total number of queues (load-balanced and directed). The number
-of directed queues comes from ``nb_single_link_event_port_queues``, since
-directed ports and queues come in pairs.
-
-When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
-whether it should be configured as a directed (the flag is set) or a
-load-balanced (the flag is unset) port. Similarly, the
-``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
-whether it is a directed or load-balanced queue.
-
-Load-balanced ports can only be linked to load-balanced queues, and directed
-ports can only be linked to directed queues. Furthermore, directed ports can
-only be linked to a single directed queue (and vice versa), and that link
-cannot change after the eventdev is started.
-
-The eventdev API does not have a directed scheduling type. To support directed
-traffic, the dlb PMD detects when an event is being sent to a directed queue
-and overrides its scheduling type. Note that the originally selected scheduling
-type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
-will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
-port.
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
 
 Flow ID
 ~~~~~~~
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index ef1c000..48e7245 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -688,6 +688,317 @@ dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
 	queue_conf->priority = 0;
 }
 
+static int32_t
+dlb2_get_sn_allocation(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_allocation_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_set_sn_allocation(struct dlb2_eventdev *dlb2, int group, int num)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_set_sn_allocation_args cfg;
+	int ret;
+
+	cfg.num = num;
+	cfg.group = group;
+
+	ret = dlb2_iface_set_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: set_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int32_t
+dlb2_get_sn_occupancy(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_occupancy_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_occupancy(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_occupancy ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+/* Query the current sequence number allocations and, if they conflict with the
+ * requested LDB queue configuration, attempt to re-allocate sequence numbers.
+ * This is best-effort; if it fails, the PMD will attempt to configure the
+ * load-balanced queue and return an error.
+ */
+static void
+dlb2_program_sn_allocation(struct dlb2_eventdev *dlb2,
+			   const struct rte_event_queue_conf *queue_conf)
+{
+	int grp_occupancy[DLB2_NUM_SN_GROUPS];
+	int grp_alloc[DLB2_NUM_SN_GROUPS];
+	int i, sequence_numbers;
+
+	sequence_numbers = (int)queue_conf->nb_atomic_order_sequences;
+
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		int total_slots;
+
+		grp_alloc[i] = dlb2_get_sn_allocation(dlb2, i);
+		if (grp_alloc[i] < 0)
+			return;
+
+		total_slots = DLB2_MAX_LDB_SN_ALLOC / grp_alloc[i];
+
+		grp_occupancy[i] = dlb2_get_sn_occupancy(dlb2, i);
+		if (grp_occupancy[i] < 0)
+			return;
+
+		/* DLB has at least one available slot for the requested
+		 * sequence numbers, so no further configuration required.
+		 */
+		if (grp_alloc[i] == sequence_numbers &&
+		    grp_occupancy[i] < total_slots)
+			return;
+	}
+
+	/* None of the sequence number groups are configured for the requested
+	 * sequence numbers, so we have to reconfigure one of them. This is
+	 * only possible if a group is not in use.
+	 */
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		if (grp_occupancy[i] == 0)
+			break;
+	}
+
+	if (i == DLB2_NUM_SN_GROUPS) {
+		DLB2_LOG_ERR("[%s()] No groups with %d sequence_numbers are available or have free slots\n",
+		       __func__, sequence_numbers);
+		return;
+	}
+
+	/* Attempt to configure slot i with the requested number of sequence
+	 * numbers. Ignore the return value -- if this fails, the error will be
+	 * caught during subsequent queue configuration.
+	 */
+	dlb2_set_sn_allocation(dlb2, i, sequence_numbers);
+}
+
+static int32_t
+dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 const struct rte_event_queue_conf *evq_conf)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_queue *queue = &ev_queue->qm_queue;
+	struct dlb2_create_ldb_queue_args cfg;
+	int32_t ret;
+	uint32_t qm_qid;
+	int sched_type = -1;
+
+	if (evq_conf == NULL)
+		return -EINVAL;
+
+	if (evq_conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_ALL_TYPES) {
+		if (evq_conf->nb_atomic_order_sequences != 0)
+			sched_type = RTE_SCHED_TYPE_ORDERED;
+		else
+			sched_type = RTE_SCHED_TYPE_PARALLEL;
+	} else
+		sched_type = evq_conf->schedule_type;
+
+	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
+	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
+	cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences;
+
+	if (sched_type != RTE_SCHED_TYPE_ORDERED) {
+		cfg.num_sequence_numbers = 0;
+		cfg.num_qid_inflights = 2048;
+	}
+
+	/* App should set this to the number of hardware flows they want, not
+	 * the overall number of flows they're going to use. E.g. if app is
+	 * using 64 flows and sets compression to 64, best-case they'll get
+	 * 64 unique hashed flows in hardware.
+	 */
+	switch (evq_conf->nb_atomic_flows) {
+	/* Valid DLB2 compression levels */
+	case 64:
+	case 128:
+	case 256:
+	case 512:
+	case (1 * 1024): /* 1K */
+	case (2 * 1024): /* 2K */
+	case (4 * 1024): /* 4K */
+	case (64 * 1024): /* 64K */
+		cfg.lock_id_comp_level = evq_conf->nb_atomic_flows;
+		break;
+	default:
+		/* Invalid compression level */
+		cfg.lock_id_comp_level = 0; /* no compression */
+	}
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create LB event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	qm_qid = cfg.response.id;
+
+	/* Save off queue config for debug, resource lookups, and reconfig */
+	queue->num_qid_inflights = cfg.num_qid_inflights;
+	queue->num_atm_inflights = cfg.num_atomic_inflights;
+
+	queue->sched_type = sched_type;
+	queue->config_state = DLB2_CONFIGURED;
+
+	DLB2_LOG_DBG("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n",
+		     qm_qid,
+		     cfg.num_atomic_inflights,
+		     cfg.num_sequence_numbers,
+		     cfg.num_qid_inflights);
+
+	return qm_qid;
+}
+
+static int
+dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int32_t qm_qid;
+
+	if (queue_conf->nb_atomic_order_sequences)
+		dlb2_program_sn_allocation(dlb2, queue_conf);
+
+	qm_qid = dlb2_hw_create_ldb_queue(dlb2, ev_queue, queue_conf);
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the load-balanced queue\n");
+
+		return qm_qid;
+	}
+
+	dlb2->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int dlb2_num_dir_queues_setup(struct dlb2_eventdev *dlb2)
+{
+	int i, num = 0;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].setup_done &&
+		    dlb2->ev_queues[i].qm_queue.is_directed)
+			num++;
+	}
+
+	return num;
+}
+
+static void
+dlb2_queue_link_teardown(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue)
+{
+	struct dlb2_eventdev_port *ev_port;
+	int i, j;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (!ev_port->link[j].valid ||
+			    ev_port->link[j].queue_id != ev_queue->id)
+				continue;
+
+			ev_port->link[j].valid = false;
+			ev_port->num_links--;
+		}
+	}
+
+	ev_queue->num_links = 0;
+}
+
+static int
+dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
+			  uint8_t ev_qid,
+			  const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_queue *ev_queue;
+	int ret;
+
+	if (queue_conf == NULL)
+		return -EINVAL;
+
+	if (ev_qid >= dlb2->num_queues)
+		return -EINVAL;
+
+	ev_queue = &dlb2->ev_queues[ev_qid];
+
+	ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg &
+		RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+	ev_queue->id = ev_qid;
+	ev_queue->conf = *queue_conf;
+
+	if (!ev_queue->qm_queue.is_directed) {
+		ret = dlb2_eventdev_ldb_queue_setup(dev, ev_queue, queue_conf);
+	} else {
+		/* The directed queue isn't setup until link time, at which
+		 * point we know its directed port ID. Directed queue setup
+		 * will only fail if this queue is already setup or there are
+		 * no directed queues left to configure.
+		 */
+		ret = 0;
+
+		ev_queue->qm_queue.config_state = DLB2_NOT_CONFIGURED;
+
+		if (ev_queue->setup_done ||
+		    dlb2_num_dir_queues_setup(dlb2) == dlb2->num_dir_queues)
+			ret = -EINVAL;
+	}
+
+	/* Tear down pre-existing port->queue links */
+	if (!ret && dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_queue_link_teardown(dlb2, ev_queue);
+
+	if (!ret)
+		ev_queue->setup_done = true;
+
+	return ret;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -696,6 +1007,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a829b9b..4c07574 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -31,3 +31,15 @@ int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				struct dlb2_create_sched_domain_args *args);
 
 void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_ldb_queue_args *cfg);
+
+int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_sn_allocation_args *args);
+
+int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_set_sn_allocation_args *args);
+
+int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_occupancy_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 6663dab..4c88fe0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -30,4 +30,16 @@ extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				 struct dlb2_create_sched_domain_args *args);
 
 extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+extern int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_queue_args *cfg);
+
+extern int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_allocation_args *args);
+
+extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_set_sn_allocation_args *args);
+
+extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				  struct dlb2_get_sn_occupancy_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 63a68a6..e6ea0d7 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3509,3 +3509,467 @@ unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
 
 	return num;
 }
+
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_ldb_queue *queue,
+				     struct dlb2_create_ldb_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+	union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+	union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+	union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+	union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+	union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+	union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+	union dlb2_sys_ldb_qid_its r7 = { {0} };
+	union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+	union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+	union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+	union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+	union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+	union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+	struct dlb2_sn_group *sn_group;
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r3.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	r4.field.limit = args->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+	r5.field.limit = queue->aqed_limit;
+
+	if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+		r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+		    r5.val);
+
+	switch (args->lock_id_comp_level) {
+	case 64:
+		r6.field.compress_code = 1;
+		break;
+	case 128:
+		r6.field.compress_code = 2;
+		break;
+	case 256:
+		r6.field.compress_code = 3;
+		break;
+	case 512:
+		r6.field.compress_code = 4;
+		break;
+	case 1024:
+		r6.field.compress_code = 5;
+		break;
+	case 2048:
+		r6.field.compress_code = 6;
+		break;
+	case 4096:
+		r6.field.compress_code = 7;
+		break;
+	case 0:
+	case 65536:
+		r6.field.compress_code = 0;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+		    r6.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r7.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+		    r7.val);
+
+	r8.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+		    r8.val);
+
+	r9.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+		    r9.val);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	r10.field.qid_fid_limit = 512;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+		    r10.val);
+
+	/* Configure SNs */
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	r11.field.mode = sn_group->mode;
+	r11.field.slot = queue->sn_slot;
+	r11.field.grp  = sn_group->id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+	r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+	r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		r0.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+		r1.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+		r2.field.vqid = queue->id.virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+			    r2.val);
+	}
+
+	r13.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+				  struct dlb2_ldb_queue *queue,
+				  struct dlb2_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb2_sn_group_full(group)) {
+			slot = dlb2_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no sequence number slots available\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+				struct dlb2_hw_domain *domain,
+				struct dlb2_ldb_queue *queue,
+				struct dlb2_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_ldb_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (dlb2_list_empty(&domain->avail_ldb_queues)) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb2_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_ldb_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		    args->num_sequence_numbers);
+	DLB2_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		    args->num_qid_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		    args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+
+	dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			    __func__, __LINE__);
+		return ret;
+	}
+
+	dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_queues, &queue->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_queues, &queue->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+						unsigned int group_id,
+						unsigned long val)
+{
+	DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+	DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+	DLB2_HW_DBG(hw, "\tValue:    %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val)
+{
+	u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+	union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+	struct dlb2_sn_group *group;
+	int mode;
+
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	group = &hw->rsrcs.sn_groups[group_id];
+
+	/*
+	 * Once the first load-balanced queue using an SN group is configured,
+	 * the group cannot be changed.
+	 */
+	if (group->slot_use_bitmap != 0)
+		return -EPERM;
+
+	for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+		if (val == valid_allocations[mode])
+			break;
+
+	if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+		return -EINVAL;
+
+	group->mode = mode;
+	group->sequence_numbers_per_queue = val;
+
+	r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+	r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+	DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+	dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index bd364d4..9c734e7 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -613,3 +613,13 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
 {
 	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_ldb_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 9310f72..a6cd178 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -150,6 +150,84 @@ dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
 		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
 }
 
+static int
+dlb2_pf_ldb_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_ldb_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_ldb_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw_dev *handle,
+			 struct dlb2_get_sn_occupancy_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_number_occupancy(&dlb2_dev->hw,
+						       args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_numbers(&dlb2_dev->hw, args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_set_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_set_group_sequence_numbers(&dlb2_dev->hw, args->group,
+					      args->num);
+
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -161,6 +239,10 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
+	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
+	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
+	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v11 13/23] event/dlb2: add port setup

[Timothy McDaniel]

Configure the load balanced (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  75 +++
 drivers/event/dlb2/dlb2.c                  | 498 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   8 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 922 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  28 +
 drivers/event/dlb2/pf/dlb2_pf.c            | 180 ++++++
 7 files changed, 1719 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index b154cac..bbd6ac8 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -82,6 +82,81 @@ The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
 the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
 load-balanced queues can use the full 16-bit flow ID range.
 
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 48e7245..f0b13a4 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -999,6 +999,503 @@ dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
 	return ret;
 }
 
+static int
+dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_cq_pop_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_cq_pop_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL)	{
+		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->consume_qe = qe;
+
+	qe->qe_valid = 0;
+	qe->qe_frag = 0;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
+	 * and so on.
+	 */
+	qe->tokens = 0;	/* set at run time */
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_enqueue_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_enqueue_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL) {
+		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->int_arm_qe = qe;
+
+	/* V2 - INT ARM is CQ_TOKEN + FRAG */
+	qe->qe_valid = 0;
+	qe->qe_frag = 1;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
+{
+	int ret, sz;
+
+	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct dlb2_enqueue_qe);
+
+	qm_port->qe4 = rte_zmalloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
+
+	if (qm_port->qe4 == NULL) {
+		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	ret = dlb2_init_consume_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	return 0;
+
+error_exit:
+
+	dlb2_free_qe_mem(qm_port);
+
+	return ret;
+}
+
+static int
+dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_ldb_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_CQ_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	cfg.cq_history_list_size = DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	if (handle->cos_id == DLB2_COS_DEFAULT)
+		cfg.cos_id = 0;
+	else
+		cfg.cos_id = handle->cos_id;
+
+	cfg.cos_strict = 0;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* If there are no directed ports, the kernel driver will ignore this
+	 * port's directed credit settings. Don't use enqueue_depth if it would
+	 * require more directed credits than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2; /* back ptr */
+	/*
+	 * Allocate and init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_ldb_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(qm_port->cfg.ldb));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     qm_port->ldb_credits,
+		     qm_port->dir_credits);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
+
+	return ret;
+}
+
+static void
+dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port)
+{
+	struct dlb2_eventdev_queue *ev_queue;
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (!ev_port->link[i].valid)
+			continue;
+
+		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
+
+		ev_port->link[i].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+}
+
+static int
+dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_port_args cfg = { {0} };
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (dlb2 == NULL || handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* Directed queues are configured at link time. */
+	cfg.queue_id = -1;
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* Don't use enqueue_depth if it would require more directed credits
+	 * than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2;  /* back ptr */
+
+	/*
+	 * Init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE to be aligned
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_dir_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = cfg.cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(qm_port->cfg.dir));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     dir_credit_high_watermark,
+		     ldb_credit_high_watermark);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_setup(struct rte_eventdev *dev,
+			 uint8_t ev_port_id,
+			 const struct rte_event_port_conf *port_conf)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_eventdev_port *ev_port;
+	int ret;
+
+	if (dev == NULL || port_conf == NULL) {
+		DLB2_LOG_ERR("Null parameter\n");
+		return -EINVAL;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
+		return -EINVAL;
+
+	if (port_conf->dequeue_depth >
+		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
+	    port_conf->enqueue_depth >
+		evdev_dlb2_default_info.max_event_port_enqueue_depth)
+		return -EINVAL;
+
+	ev_port = &dlb2->ev_ports[ev_port_id];
+	/* configured? */
+	if (ev_port->setup_done) {
+		DLB2_LOG_ERR("evport %d is already configured\n", ev_port_id);
+		return -EINVAL;
+	}
+
+	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
+		RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (!ev_port->qm_port.is_directed) {
+		ret = dlb2_hw_create_ldb_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the lB port ve portId=%d\n",
+				     ev_port_id);
+
+			return ret;
+		}
+	} else {
+		ret = dlb2_hw_create_dir_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the DIR port\n");
+			return ret;
+		}
+	}
+
+	/* Save off port config for reconfig */
+	ev_port->conf = *port_conf;
+
+	ev_port->id = ev_port_id;
+	ev_port->enq_configured = true;
+	ev_port->setup_done = true;
+	ev_port->inflight_max = port_conf->new_event_threshold;
+	ev_port->implicit_release = !(port_conf->event_port_cfg &
+		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
+	ev_port->outstanding_releases = 0;
+	ev_port->inflight_credits = 0;
+	ev_port->credit_update_quanta = RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
+	ev_port->dlb2 = dlb2; /* reverse link */
+
+	/* Tear down pre-existing port->queue links */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
+
+	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1009,6 +1506,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
+		.port_setup       = dlb2_eventdev_port_setup,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 4c07574..01818c7 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -43,3 +43,11 @@ int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				   struct dlb2_get_sn_occupancy_args *args);
+
+int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_dir_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4c88fe0..a6b923a 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -42,4 +42,12 @@ extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				  struct dlb2_get_sn_occupancy_args *args);
+
+extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_ldb_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_dir_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index e6ea0d7..a633d3e 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3973,3 +3973,925 @@ int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_ldb_port *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_pp2vas r0 = { {0} };
+	union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+		union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_ldb_port *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_ldb_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+	union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+	union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+	union dlb2_chp_hist_list_lim r5 = { {0} };
+	union dlb2_chp_hist_list_base r6 = { {0} };
+	union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+	union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+	union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+	union dlb2_sys_ldb_cq_at r10 = { {0} };
+	union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+	union dlb2_chp_ldb_cq2vas r12 = { {0} };
+	union dlb2_lsp_cq2priov r13 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r14.field.token_count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    r14.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	r5.field.limit = port->hist_list_entry_limit - 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+	r6.field.base = port->hist_list_entry_base;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	r7.field.limit = args->cq_history_list_size;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+	r8.field.push_ptr = r6.field.base;
+	r8.field.generation = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    r8.val);
+
+	r9.field.pop_ptr = r6.field.base;
+	r9.field.generation = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+	/* Disable the port's QID mappings */
+	r13.field.v = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+	return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_ldb_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb2_ldb_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+	if (ret < 0)
+		return ret;
+
+	dlb2_ldb_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_ldb_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		    args->cq_history_list_size);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+	DLB2_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		    args->cos_strict);
+}
+
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_ldb_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+		resp->status = DLB2_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		if (dlb2_list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	} else {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			if (!dlb2_list_empty(&domain->avail_ldb_ports[i]))
+				break;
+		}
+
+		if (i == DLB2_NUM_COS_DOMAINS) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ *	its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	int ret, cos_id, i;
+
+	dlb2_log_create_ldb_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->cos_strict) {
+		cos_id = args->cos_id;
+
+		port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+					  typeof(*port));
+	} else {
+		int idx;
+
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+			port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+						  typeof(*port));
+			if (port)
+				break;
+		}
+
+		cos_id = idx;
+	}
+
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (port->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_ldb_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_ports[cos_id], &port->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_ports[cos_id], &port->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_dir_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+}
+
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+			    bool vdev_req,
+			    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+
+	return NULL;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_dir_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the queue is already configured, validate
+	 * the queue ID, its domain, and whether the queue is configured.
+	 */
+	if (args->queue_id != -1) {
+		struct dlb2_dir_pq_pair *queue;
+
+		queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+						    vdev_req,
+						    domain);
+
+		if (queue == NULL || queue->domain_id.phys_id !=
+				domain->id.phys_id ||
+				!queue->queue_configured) {
+			resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the port's queue is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->queue_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_dir_pq_pair *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_dir_pp2vas r0 = { {0} };
+	union dlb2_sys_dir_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+		union dlb2_sys_dir_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_dir_pq_pair *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_dir_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+	union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+	union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+	union dlb2_sys_dir_cq_fmt r9 = { {0} };
+	union dlb2_sys_dir_cq_at r10 = { {0} };
+	union dlb2_sys_dir_cq_pasid r11 = { {0} };
+	union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r13.field.count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    r13.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.disable_wb_opt = 0;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	r9.field.keep_pf_ppid = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+	return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_dir_pq_pair *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_dir_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb2_dir_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+
+	if (ret < 0)
+		return ret;
+
+	dlb2_dir_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ *	and queue. The port/queue pair have the same ID and name.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *port;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (args->queue_id != -1)
+		port = dlb2_get_domain_used_dir_pq(args->queue_id,
+						   vdev_req,
+						   domain);
+	else
+		port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					  typeof(*port));
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_dir_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs, &port->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs, &port->domain_list);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 9c734e7..7104f9b 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -623,3 +623,31 @@ dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_ldb_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
+
+int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_dir_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a6cd178..2a82b8f 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -228,6 +228,184 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static void *
+dlb2_alloc_coherent_aligned(const struct rte_memzone **mz, uintptr_t *phys,
+			    size_t size, int align)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t core_id = rte_lcore_id();
+	unsigned int socket_id;
+
+	snprintf(mz_name, sizeof(mz_name) - 1, "event_dlb2_pf_%lx",
+		 (unsigned long)rte_get_timer_cycles());
+	if (core_id == (unsigned int)LCORE_ID_ANY)
+		core_id = rte_get_main_lcore();
+	socket_id = rte_lcore_to_socket_id(core_id);
+	*mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
+					 RTE_MEMZONE_IOVA_CONTIG, align);
+	if (*mz == NULL) {
+		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu bytes - %s\n",
+			     size, rte_strerror(rte_errno));
+		*phys = 0;
+		return NULL;
+	}
+	*phys = (*mz)->iova;
+	return (*mz)->addr;
+}
+
+static int
+dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_ldb_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret, cq_alloc_depth;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = false;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	cq_alloc_depth = RTE_MAX(cfg->cq_depth, DLB2_MIN_HARDWARE_CQ_DEPTH);
+	alloc_sz = cq_alloc_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].cq_base = (void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+	return ret;
+}
+
+static int
+dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_dir_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = true;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	alloc_sz = cfg->cq_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
+		(void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -240,6 +418,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v11 14/23] event/dlb2: add port link

[Timothy McDaniel]

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 302 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 633 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
 6 files changed, 1007 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index f0b13a4..a7e9a4c 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1496,6 +1496,307 @@ dlb2_eventdev_port_setup(struct rte_eventdev *dev,
 	return 0;
 }
 
+static int16_t
+dlb2_hw_map_ldb_qid_to_port(struct dlb2_hw_dev *handle,
+			    uint32_t qm_port_id,
+			    uint16_t qm_qid,
+			    uint8_t priority)
+{
+	struct dlb2_map_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	/* Build message */
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+	cfg.priority = EV_TO_DLB2_PRIO(priority);
+
+	ret = dlb2_iface_map_qid(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: map qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		DLB2_LOG_ERR("dlb2: grp=%d, qm_port=%d, qm_qid=%d prio=%d\n",
+			     handle->domain_id, cfg.port_id,
+			     cfg.qid,
+			     cfg.priority);
+	} else {
+		DLB2_LOG_DBG("dlb2: mapped queue %d to qm_port %d\n",
+			     qm_qid, qm_port_id);
+	}
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port,
+			  struct dlb2_eventdev_queue *ev_queue,
+			  uint8_t priority)
+{
+	int first_avail = -1;
+	int ret, i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid) {
+			if (ev_port->link[i].queue_id == ev_queue->id &&
+			    ev_port->link[i].priority == priority) {
+				if (ev_port->link[i].mapped)
+					return 0; /* already mapped */
+				first_avail = i;
+			}
+		} else if (first_avail == -1)
+			first_avail = i;
+	}
+	if (first_avail == -1) {
+		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
+			     ev_port->qm_port.id);
+		return -EINVAL;
+	}
+
+	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
+					  ev_port->qm_port.id,
+					  ev_queue->qm_queue.id,
+					  priority);
+
+	if (!ret)
+		ev_port->link[first_avail].mapped = true;
+
+	return ret;
+}
+
+static int32_t
+dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 int32_t qm_port_id)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_queue_args cfg;
+	int32_t ret;
+
+	/* The directed port is always configured before its queue */
+	cfg.port_id = qm_port_id;
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_dir_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	int32_t qm_qid;
+
+	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port->qm_port.id);
+
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the DIR queue\n");
+		return qm_qid;
+	}
+
+	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int
+dlb2_do_port_link(struct rte_eventdev *dev,
+		  struct dlb2_eventdev_queue *ev_queue,
+		  struct dlb2_eventdev_port *ev_port,
+		  uint8_t prio)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int err;
+
+	/* Don't link until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	if (ev_queue->qm_queue.is_directed)
+		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue, ev_port);
+	else
+		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue, prio);
+
+	if (err) {
+		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
+			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
+			     ev_queue->id, ev_port->id);
+
+		rte_errno = err;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
+			uint8_t queue_id,
+			bool link_exists,
+			int index)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	bool port_is_dir, queue_is_dir;
+
+	if (queue_id > dlb2->num_queues) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	ev_queue = &dlb2->ev_queues[queue_id];
+
+	if (!ev_queue->setup_done &&
+	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	port_is_dir = ev_port->qm_port.is_directed;
+	queue_is_dir = ev_queue->qm_queue.is_directed;
+
+	if (port_is_dir != queue_is_dir) {
+		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
+			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
+			     port_is_dir ? "DIR" : "LDB", ev_port->id);
+
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if there is space for the requested link */
+	if (!link_exists && index == -1) {
+		DLB2_LOG_ERR("no space for new link\n");
+		rte_errno = -ENOSPC;
+		return -1;
+	}
+
+	/* Check if the directed port is already linked */
+	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if the directed queue is already linked */
+	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
+			     ev_queue->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
+			const uint8_t queues[], const uint8_t priorities[],
+			uint16_t nb_links)
+
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i, j;
+
+	RTE_SET_USED(dev);
+
+	if (ev_port == NULL) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!ev_port->setup_done &&
+	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
+	 * queues pointer.
+	 */
+	if (nb_links == 0) {
+		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
+		     nb_links,
+		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
+		     ev_port->id);
+
+	for (i = 0; i < nb_links; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		uint8_t queue_id, prio;
+		bool found = false;
+		int index = -1;
+
+		queue_id = queues[i];
+		prio = priorities[i];
+
+		/* Check if the link already exists. */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].valid) {
+				if (ev_port->link[j].queue_id == queue_id) {
+					found = true;
+					index = j;
+					break;
+				}
+			} else if (index == -1) {
+				index = j;
+			}
+
+		/* could not link */
+		if (index == -1)
+			break;
+
+		/* Check if already linked at the requested priority */
+		if (found && ev_port->link[j].priority == prio)
+			continue;
+
+		if (dlb2_validate_port_link(ev_port, queue_id, found, index))
+			break; /* return index of offending queue */
+
+		ev_queue = &dlb2->ev_queues[queue_id];
+
+		if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+			break; /* return index of offending queue */
+
+		ev_queue->num_links++;
+
+		ev_port->link[index].queue_id = queue_id;
+		ev_port->link[index].priority = prio;
+		ev_port->link[index].valid = true;
+		/* Entry already exists?  If so, then must be prio change */
+		if (!found)
+			ev_port->num_links++;
+	}
+	return i;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1507,6 +1808,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_link        = dlb2_eventdev_port_link,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 01818c7..7d1ba73 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -51,3 +51,9 @@ int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 				  struct dlb2_create_dir_port_args *cfg,
 				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_dir_queue_args *cfg);
+
+int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+			  struct dlb2_map_qid_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index a6b923a..4eddd14 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -50,4 +50,10 @@ extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 struct dlb2_create_dir_port_args *cfg,
 					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+					struct dlb2_create_dir_queue_args *cfg);
+
+extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+				 struct dlb2_map_qid_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index a633d3e..14ad262 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -4895,3 +4895,636 @@ int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_dir_pq_pair *queue,
+				     struct dlb2_create_dir_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_dir_vasqid_v r0 = { {0} };
+	union dlb2_sys_dir_qid_its r1 = { {0} };
+	union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+	union dlb2_sys_dir_qid_v r5 = { {0} };
+
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r0.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r1.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+		    r1.val);
+
+	r2.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+		    r2.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		r3.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+		r4.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+	}
+
+	r5.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+	queue->queue_configured = true;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_dir_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_dir_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = dlb2_get_domain_used_dir_pq(args->port_id,
+						   vdev_req,
+						   domain);
+
+		if (port == NULL || port->domain_id.phys_id !=
+				domain->id.phys_id || !port->port_configured) {
+			resp->status = DLB2_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the queue's port is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->port_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->port_id != -1)
+		queue = dlb2_get_domain_used_dir_pq(args->port_id,
+						    vdev_req,
+						    domain);
+	else
+		queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					   typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &queue->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs,
+			      &queue->domain_list);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot,
+					      struct dlb2_map_qid_args *args)
+{
+	union dlb2_lsp_cq2priov r0;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+	r0.field.v |= 1 << slot;
+	r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+	dlb2_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+					      struct dlb2_ldb_queue *queue,
+					      struct dlb2_cmd_response *resp)
+{
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB2_QUEUE_UNMAPPED;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+			  bool vdev_req,
+			  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter)
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+			      bool vdev_req,
+			      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+
+		DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+	}
+
+	return NULL;
+}
+
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+				    u32 domain_id,
+				    struct dlb2_map_qid_args *args,
+				    struct dlb2_cmd_response *resp,
+				    bool vdev_req,
+				    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	struct dlb2_ldb_queue *queue;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB2_QID_PRIORITIES) {
+		resp->status = DLB2_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_map_qid_args *args,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	DLB2_HW_DBG(hw, "\tPriority:  %d\n",
+		    args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	int ret, i, id;
+	u8 prio;
+
+	dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_map_qid_args(hw,
+				       domain_id,
+				       args,
+				       resp,
+				       vdev_req,
+				       vdev_id);
+	if (ret)
+		return ret;
+
+	prio = args->priority;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	if (port->num_pending_removals)
+		dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].pending_priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+		if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb2_qid_map_state st;
+
+			if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+				DLB2_HW_ERR(hw,
+					    "[%s():%d] Internal error: port slot tracking failed\n",
+					    __func__, __LINE__);
+				return -EFAULT;
+			}
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb2_port_slot_state_transition(hw, port, queue,
+							      i, st);
+			if (ret)
+				return ret;
+
+			DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 7104f9b..d91d186 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -651,3 +651,13 @@ dlb2_pf_create_dir_port(struct dlb2_hw *hw,
 				       NOT_VF_REQ,
 				       PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_dir_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 2a82b8f..37be4b1 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -406,6 +406,54 @@ dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_dir_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_dir_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_dir_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
+		struct dlb2_map_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_map_qid(&dlb2_dev->hw,
+			      handle->domain_id,
+			      cfg,
+			      &response,
+			      false,
+			      0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -419,7 +467,9 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
 	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
+	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v11 15/23] event/dlb2: add port unlink and port unlinks in progress

[Timothy McDaniel]

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 163 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 283 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  51 ++++++
 5 files changed, 509 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index a7e9a4c..9357b23 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1797,6 +1797,166 @@ dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
 	return i;
 }
 
+static int16_t
+dlb2_hw_unmap_ldb_qid_from_port(struct dlb2_hw_dev *handle,
+				uint32_t qm_port_id,
+				uint16_t qm_qid)
+{
+	struct dlb2_unmap_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+
+	ret = dlb2_iface_unmap_qid(handle, &cfg);
+	if (ret < 0)
+		DLB2_LOG_ERR("dlb2: unmap qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_detach_ldb(struct dlb2_eventdev *dlb2,
+			    struct dlb2_eventdev_port *ev_port,
+			    struct dlb2_eventdev_queue *ev_queue)
+{
+	int ret, i;
+
+	/* Don't unlink until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid &&
+		    ev_port->link[i].queue_id == ev_queue->id)
+			break; /* found */
+	}
+
+	/* This is expected with eventdev API!
+	 * It blindly attemmpts to unmap all queues.
+	 */
+	if (i == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_LOG_DBG("dlb2: ignoring LB QID %d not mapped for qm_port %d.\n",
+			     ev_queue->qm_queue.id,
+			     ev_port->qm_port.id);
+		return 0;
+	}
+
+	ret = dlb2_hw_unmap_ldb_qid_from_port(&dlb2->qm_instance,
+					      ev_port->qm_port.id,
+					      ev_queue->qm_queue.id);
+	if (!ret)
+		ev_port->link[i].mapped = false;
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
+			  uint8_t queues[], uint16_t nb_unlinks)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (queues == NULL || nb_unlinks == 0) {
+		DLB2_LOG_DBG("dlb2: queues is NULL or nb_unlinks is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	/* FIXME: How to handle unlink on a directed port? */
+	if (ev_port->qm_port.is_directed) {
+		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
+			     ev_port->id);
+		rte_errno = 0;
+		return nb_unlinks; /* as if success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	for (i = 0; i < nb_unlinks; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		int ret, j;
+
+		if (queues[i] >= dlb2->num_queues) {
+			DLB2_LOG_ERR("dlb2: invalid queue id %d\n", queues[i]);
+			rte_errno = -EINVAL;
+			return i; /* return index of offending queue */
+		}
+
+		ev_queue = &dlb2->ev_queues[queues[i]];
+
+		/* Does a link exist? */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].queue_id == queues[i] &&
+			    ev_port->link[j].valid)
+				break;
+
+		if (j == DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+			continue;
+
+		ret = dlb2_event_queue_detach_ldb(dlb2, ev_port, ev_queue);
+		if (ret) {
+			DLB2_LOG_ERR("unlink err=%d for port %d queue %d\n",
+				     ret, ev_port->id, queues[i]);
+			rte_errno = -ENOENT;
+			return i; /* return index of offending queue */
+		}
+
+		ev_port->link[j].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+
+	return nb_unlinks;
+}
+
+static int
+dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
+				       void *event_port)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	struct dlb2_pending_port_unmaps_args cfg;
+	int ret;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	cfg.port_id = ev_port->qm_port.id;
+	dlb2 = ev_port->dlb2;
+	handle = &dlb2->qm_instance;
+	ret = dlb2_iface_pending_port_unmaps(handle, &cfg);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: num_unlinks_in_progress ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1809,6 +1969,9 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
 		.port_link        = dlb2_eventdev_port_link,
+		.port_unlink      = dlb2_eventdev_port_unlink,
+		.port_unlinks_in_progress =
+				    dlb2_eventdev_port_unlinks_in_progress,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 7d1ba73..bd241f3 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -57,3 +57,9 @@ int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 			  struct dlb2_map_qid_args *cfg);
+
+int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+			    struct dlb2_unmap_qid_args *cfg);
+
+int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4eddd14..e892036 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -56,4 +56,10 @@ extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 				 struct dlb2_map_qid_args *cfg);
+
+extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+				   struct dlb2_unmap_qid_args *cfg);
+
+extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 14ad262..4a3d96d 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5528,3 +5528,286 @@ int dlb2_hw_map_qid(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_unmap_qid_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+		DLB2_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			    hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+				      u32 domain_id,
+				      struct dlb2_unmap_qid_args *args,
+				      struct dlb2_cmd_response *resp,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	struct dlb2_ldb_port *port;
+	int slot;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			    __func__, args->qid);
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		return 0;
+
+	resp->status = DLB2_ST_INVALID_QID;
+	return -EINVAL;
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	bool unmap_complete;
+	int i, ret, id;
+
+	dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_unmap_qid_args(hw,
+					 domain_id,
+					 args,
+					 resp,
+					 vdev_req,
+					 vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+		st = DLB2_QUEUE_UNMAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		st = DLB2_QUEUE_UNMAP_IN_PROG;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB2_QUEUE_MAPPED;
+	if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: no available CQ slots\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asynchronous procedure. It requires
+	 * stopping the DLB2 from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb2_ldb_port_cq_disable(hw, port);
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete && !os_worker_active(hw))
+		os_schedule_work(hw);
+
+unmap_qid_done:
+	resp->status = 0;
+
+	return 0;
+}
+
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+				  struct dlb2_pending_port_unmaps_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+
+	dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 37be4b1..33ea73e 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -454,6 +454,55 @@ dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw_dev *handle,
+		  struct dlb2_unmap_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_unmap_qid(&dlb2_dev->hw,
+				handle->domain_id,
+				cfg,
+				&response,
+				false,
+				0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
+			    struct dlb2_pending_port_unmaps_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_pending_port_unmaps(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -470,6 +519,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
+	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v11 16/23] event/dlb2: add eventdev start

[Timothy McDaniel]

Add support for the eventdev start entry point.
We delay initializing some resources until
eventdev start, since the number of linked queues can be
used to determine if we are dealing with a ldb or dir resource.
If this is a device restart, then the previous configuration
will be reapplied.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 133 +++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   3 +
 drivers/event/dlb2/dlb2_iface.h            |   3 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 123 ++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +++
 drivers/event/dlb2/pf/dlb2_pf.c            |  25 ++++++
 6 files changed, 297 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 9357b23..fa6fa7b 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1957,6 +1957,138 @@ dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
 	return cfg.response.id;
 }
 
+static int
+dlb2_eventdev_reapply_configuration(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret, i;
+
+	/* If an event queue or port was previously configured, but hasn't been
+	 * reconfigured, reapply its original configuration.
+	 */
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+
+		ev_queue = &dlb2->ev_queues[i];
+
+		if (ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_queue_setup(dev, i, &ev_queue->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure queue %d", i);
+			return ret;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+
+		if (ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_port_setup(dev, i, &ev_port->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure ev_port %d",
+				     i);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_apply_port_links(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int i;
+
+	/* Perform requested port->queue links */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+		int j;
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			struct dlb2_eventdev_queue *ev_queue;
+			uint8_t prio, queue_id;
+
+			if (!ev_port->link[j].valid)
+				continue;
+
+			prio = ev_port->link[j].priority;
+			queue_id = ev_port->link[j].queue_id;
+
+			if (dlb2_validate_port_link(ev_port, queue_id, true, j))
+				return -EINVAL;
+
+			ev_queue = &dlb2->ev_queues[queue_id];
+
+			if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_start(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_start_domain_args cfg;
+	int ret, i;
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_ERR("bad state %d for dev_start\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return -EINVAL;
+	}
+	dlb2->run_state = DLB2_RUN_STATE_STARTING;
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	/* If the device was configured more than once, some event ports and/or
+	 * queues may need to be reconfigured.
+	 */
+	ret = dlb2_eventdev_reapply_configuration(dev);
+	if (ret)
+		return ret;
+
+	/* The DLB PMD delays port links until the device is started. */
+	ret = dlb2_eventdev_apply_port_links(dev);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		if (!dlb2->ev_ports[i].setup_done) {
+			DLB2_LOG_ERR("dlb2: port %d not setup", i);
+			return -ESTALE;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0) {
+			DLB2_LOG_ERR("dlb2: queue %d is not linked", i);
+			return -ENOLINK;
+		}
+	}
+
+	ret = dlb2_iface_sched_domain_start(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: sched_domain_start ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STARTED;
+	DLB2_LOG_DBG("dlb2: sched_domain_start completed OK\n");
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1964,6 +2096,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.dev_start        = dlb2_eventdev_start,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index bd241f3..a86191d 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -63,3 +63,6 @@ int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				     struct dlb2_start_domain_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index e892036..7ead374 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -62,4 +62,7 @@ extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				struct dlb2_start_domain_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 4a3d96d..c14a2f3 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5811,3 +5811,126 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 struct dlb2_cmd_response *resp,
+					 bool vdev_req,
+					 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_start_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 start domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_hw_start_domain() - Lock the domain configuration
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @arg: User-provided arguments (unused, here for ioctl callback template).
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int
+dlb2_hw_start_domain(struct dlb2_hw *hw,
+		     u32 domain_id,
+		     __attribute((unused)) struct dlb2_start_domain_args *arg,
+		     struct dlb2_cmd_response *resp,
+		     bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_queue;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+	RTE_SET_USED(arg);
+	RTE_SET_USED(iter);
+
+	dlb2_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb2_verify_start_domain_args(hw,
+					    domain_id,
+					    resp,
+					    vdev_req,
+					    vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB2 will drop all
+	 * incoming traffic to those queues.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, ldb_queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r0.val);
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+	}
+
+	dlb2_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index d91d186..06b6aee 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -661,3 +661,13 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_start_domain(struct dlb2_hw *hw,
+		     u32 id,
+		     struct dlb2_start_domain_args *args,
+		     struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_start_domain(hw, id, args, resp, NOT_VF_REQ,
+				    PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 33ea73e..fff2e57 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -503,6 +503,30 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
 
 	return ret;
 }
+
+static int
+dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
+			   struct dlb2_start_domain_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_start_domain(&dlb2_dev->hw,
+				   handle->domain_id,
+				   cfg,
+				   &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -520,6 +544,7 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v11 17/23] event/dlb2: add enqueue and its burst variants

[Timothy McDaniel]

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst | 118 +++++++++
 drivers/event/dlb2/dlb2.c     | 540 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 658 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index bbd6ac8..aa8bf01 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -163,6 +163,124 @@ Flow ID
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Hardware Credits
+~~~~~~~~~~~~~~~~
+
+DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
+event storage, with each unit of storage represented by a credit. A port spends
+a credit to enqueue an event, and hardware refills the ports with credits as the
+events are scheduled to ports. Refills come from credit pools, and each port is
+a member of a load-balanced credit pool and a directed credit pool. The
+load-balanced credits are used to enqueue to load-balanced queues, and directed
+credits are used for directed queues.
+
+A DLB2 eventdev contains one load-balanced and one directed credit pool. These
+pools' sizes are controlled by the nb_events_limit field in struct
+rte_event_dev_config. The load-balanced pool is sized to contain
+nb_events_limit credits, and the directed pool is sized to contain
+nb_events_limit/4 credits. The directed pool size can be overridden with the
+num_dir_credits vdev argument, like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,num_dir_credits=<value>
+
+This can be used if the default allocation is too low or too high for the
+specific application needs. The PMD also supports a vdev arg that limits the
+max_num_events reported by rte_event_dev_info_get():
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,max_num_events=<value>
+
+By default, max_num_events is reported as the total available load-balanced
+credits. If multiple DLB2-based applications are being used, it may be desirable
+to control how many load-balanced credits each application uses, particularly
+when application(s) are written to configure nb_events_limit equal to the
+reported max_num_events.
+
+Each port is a member of both credit pools. A port's credit allocation is
+defined by its low watermark, high watermark, and refill quanta. These three
+parameters are calculated by the dlb PMD like so:
+
+- The load-balanced high watermark is set to the port's enqueue_depth.
+  The directed high watermark is set to the minimum of the enqueue_depth and
+  the directed pool size divided by the total number of ports.
+- The refill quanta is set to half the high watermark.
+- The low watermark is set to the minimum of 16 and the refill quanta.
+
+When the eventdev is started, each port is pre-allocated a high watermark's
+worth of credits. For example, if an eventdev contains four ports with enqueue
+depths of 32 and a load-balanced credit pool size of 4096, each port will start
+with 32 load-balanced credits, and there will be 3968 credits available to
+replenish the ports. Thus, a single port is not capable of enqueueing up to the
+nb_events_limit (without any events being dequeued), since the other ports are
+retaining their initial credit allocation; in short, all ports must enqueue in
+order to reach the limit.
+
+If a port attempts to enqueue and has no credits available, the enqueue
+operation will fail and the application must retry the enqueue. Credits are
+replenished asynchronously by the DLB2 hardware.
+
+Software Credits
+~~~~~~~~~~~~~~~~
+
+The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
+credit scheme on top of the hardware credit scheme in order to comply with
+the per-port backpressure described in the eventdev API.
+
+The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
+credit when it enqueues to a queue, and credits are later replenished after the
+events are dequeued and released.
+
+In the software credit scheme, a credit is consumed when a new (.op =
+RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
+replenished when the event is released from the system (either explicitly with
+RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
+
+In this model, an event is "in the system" from its first enqueue into eventdev
+until it is last dequeued. If the event goes through multiple event queues, it
+is still considered "in the system" while a worker thread is processing it.
+
+A port will fail to enqueue if the number of events in the system exceeds its
+``new_event_threshold`` (specified at port setup time). A port will also fail
+to enqueue if it lacks enough hardware credits to enqueue; load-balanced
+credits are used to enqueue to a load-balanced queue, and directed credits are
+used to enqueue to a directed queue.
+
+The out-of-credit situations are typically transient, and an eventdev
+application using the DLB2 ought to retry its enqueues if they fail.
+If enqueue fails, DLB2 PMD sets rte_errno as follows:
+
+- -ENOSPC: Credit exhaustion (either hardware or software)
+- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
+
+Depending on the pipeline the application has constructed, it's possible to
+enter a credit deadlock scenario wherein the worker thread lacks the credit
+to enqueue an event, and it must dequeue an event before it can recover the
+credit. If the worker thread retries its enqueue indefinitely, it will not
+make forward progress. Such deadlock is possible if the application has event
+"loops", in which an event in dequeued from queue A and later enqueued back to
+queue A.
+
+Due to this, workers should stop retrying after a time, release the events it
+is attempting to enqueue, and dequeue more events. It is important that the
+worker release the events and don't simply set them aside to retry the enqueue
+again later, because the port has limited history list size (by default, twice
+the port's dequeue_depth).
+
+Priority
+~~~~~~~~
+
+The DLB2 supports event priority and per-port queue service priority, as
+described in the eventdev header file. The DLB2 does not support 'global' event
+queue priority established at queue creation time.
+
+DLB2 supports 8 event and queue service priority levels. For both priority
+types, the PMD uses the upper three bits of the priority field to determine the
+DLB2 priority, discarding the 5 least significant bits. The 5 least significant
+event priority bits are not preserved when an event is enqueued.
+
 Reconfiguration
 ~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index fa6fa7b..faf45c8 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2089,6 +2089,540 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
 	return 0;
 }
 
+static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
+	{
+		/* Load-balanced cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
+	},
+	{
+		/* Directed cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
+	},
+};
+
+static inline uint32_t
+dlb2_port_credits_get(struct dlb2_port *qm_port,
+		      enum dlb2_hw_queue_types type)
+{
+	uint32_t credits = *qm_port->credit_pool[type];
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	if (unlikely(credits < batch_size))
+		batch_size = credits;
+
+	if (likely(credits &&
+		   __atomic_compare_exchange_n(
+			qm_port->credit_pool[type],
+			&credits, credits - batch_size, false,
+			__ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
+		return batch_size;
+	else
+		return 0;
+}
+
+static inline void
+dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port)
+{
+	uint16_t quanta = ev_port->credit_update_quanta;
+
+	if (ev_port->inflight_credits >= quanta * 2) {
+		/* Replenish credits, saving one quanta for enqueues */
+		uint16_t val = ev_port->inflight_credits - quanta;
+
+		__atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits -= val;
+	}
+}
+
+static inline int
+dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
+						__ATOMIC_SEQ_CST);
+	const int num = 1;
+
+	if (unlikely(ev_port->inflight_max < sw_inflights)) {
+		DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
+		rte_errno = -ENOSPC;
+		return 1;
+	}
+
+	if (ev_port->inflight_credits < num) {
+		/* check if event enqueue brings ev_port over max threshold */
+		uint32_t credit_update_quanta = ev_port->credit_update_quanta;
+
+		if (sw_inflights + credit_update_quanta >
+				dlb2->new_event_limit) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_new_event_limit,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+
+		__atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
+				   __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits += (credit_update_quanta);
+
+		if (ev_port->inflight_credits < num) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_inflight_credits,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_ldb_credits == 0)) {
+		qm_port->cached_ldb_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_LDB_QUEUE);
+		if (unlikely(qm_port->cached_ldb_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
+			1);
+			DLB2_LOG_DBG("ldb credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_dir_credits == 0)) {
+		qm_port->cached_dir_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_DIR_QUEUE);
+		if (unlikely(qm_port->cached_dir_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
+			1);
+			DLB2_LOG_DBG("dir credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static __rte_always_inline void
+dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
+	      struct process_local_port_data *port_data)
+{
+	dlb2_movdir64b(port_data->pp_addr, qe4);
+}
+
+static inline void
+dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
+		   bool do_sfence,
+		   struct process_local_port_data *port_data)
+{
+	/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
+	 * application writes complete before enqueueing the QE.
+	 */
+	if (do_sfence)
+		rte_wmb();
+
+	dlb2_pp_write(qm_port->qe4, port_data);
+}
+
+static inline void
+dlb2_event_build_hcws(struct dlb2_port *qm_port,
+		      const struct rte_event ev[],
+		      int num,
+		      uint8_t *sched_type,
+		      uint8_t *queue_id)
+{
+	struct dlb2_enqueue_qe *qe;
+	uint16_t sched_word[4];
+	__m128i sse_qe[2];
+	int i;
+
+	qe = qm_port->qe4;
+
+	sse_qe[0] = _mm_setzero_si128();
+	sse_qe[1] = _mm_setzero_si128();
+
+	switch (num) {
+	case 4:
+		/* Construct the metadata portion of two HCWs in one 128b SSE
+		 * register. HCW metadata is constructed in the SSE registers
+		 * like so:
+		 * sse_qe[0][63:0]:   qe[0]'s metadata
+		 * sse_qe[0][127:64]: qe[1]'s metadata
+		 * sse_qe[1][63:0]:   qe[2]'s metadata
+		 * sse_qe[1][127:64]: qe[3]'s metadata
+		 */
+
+		/* Convert the event operation into a command byte and store it
+		 * in the metadata:
+		 * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
+		 * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
+		 * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
+		 * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
+		 */
+#define DLB2_QE_CMD_BYTE 7
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[0].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[1].op],
+				DLB2_QE_CMD_BYTE + 8);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[2].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[3].op],
+				DLB2_QE_CMD_BYTE + 8);
+
+		/* Store priority, scheduling type, and queue ID in the sched
+		 * word array because these values are re-used when the
+		 * destination is a directed queue.
+		 */
+		sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
+				sched_type[0] << 8 |
+				queue_id[0];
+		sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
+				sched_type[1] << 8 |
+				queue_id[1];
+		sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
+				sched_type[2] << 8 |
+				queue_id[2];
+		sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
+				sched_type[3] << 8 |
+				queue_id[3];
+
+		/* Store the event priority, scheduling type, and queue ID in
+		 * the metadata:
+		 * sse_qe[0][31:16] = sched_word[0]
+		 * sse_qe[0][95:80] = sched_word[1]
+		 * sse_qe[1][31:16] = sched_word[2]
+		 * sse_qe[1][95:80] = sched_word[3]
+		 */
+#define DLB2_QE_QID_SCHED_WORD 1
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[0],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[1],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[2],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[3],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+
+		/* If the destination is a load-balanced queue, store the lock
+		 * ID. If it is a directed queue, DLB places this field in
+		 * bytes 10-11 of the received QE, so we format it accordingly:
+		 * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
+		 * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
+		 * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
+		 * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
+		 */
+#define DLB2_QE_LOCK_ID_WORD 2
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[0] == DLB2_SCHED_DIRECTED) ?
+					sched_word[0] : ev[0].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[1] == DLB2_SCHED_DIRECTED) ?
+					sched_word[1] : ev[1].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[2] == DLB2_SCHED_DIRECTED) ?
+					sched_word[2] : ev[2].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[3] == DLB2_SCHED_DIRECTED) ?
+					sched_word[3] : ev[3].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+
+		/* Store the event type and sub event type in the metadata:
+		 * sse_qe[0][15:0]  = flow_id[0]
+		 * sse_qe[0][79:64] = flow_id[1]
+		 * sse_qe[1][15:0]  = flow_id[2]
+		 * sse_qe[1][79:64] = flow_id[3]
+		 */
+#define DLB2_QE_EV_TYPE_WORD 0
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[0].sub_event_type << 8 |
+						ev[0].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[1].sub_event_type << 8 |
+						ev[1].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[2].sub_event_type << 8 |
+						ev[2].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[3].sub_event_type << 8 |
+						ev[3].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+
+		/* Store the metadata to memory (use the double-precision
+		 * _mm_storeh_pd because there is no integer function for
+		 * storing the upper 64b):
+		 * qe[0] metadata = sse_qe[0][63:0]
+		 * qe[1] metadata = sse_qe[0][127:64]
+		 * qe[2] metadata = sse_qe[1][63:0]
+		 * qe[3] metadata = sse_qe[1][127:64]
+		 */
+		_mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
+		_mm_storeh_pd((double *)&qe[1].u.opaque_data,
+			      (__m128d)sse_qe[0]);
+		_mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
+		_mm_storeh_pd((double *)&qe[3].u.opaque_data,
+			      (__m128d)sse_qe[1]);
+
+		qe[0].data = ev[0].u64;
+		qe[1].data = ev[1].u64;
+		qe[2].data = ev[2].u64;
+		qe[3].data = ev[3].u64;
+
+		break;
+	case 3:
+	case 2:
+	case 1:
+		/* At least one QE will be valid, so only zero out three */
+		qe[1].cmd_byte = 0;
+		qe[2].cmd_byte = 0;
+		qe[3].cmd_byte = 0;
+
+		for (i = 0; i < num; i++) {
+			qe[i].cmd_byte =
+				cmd_byte_map[qm_port->is_directed][ev[i].op];
+			qe[i].sched_type = sched_type[i];
+			qe[i].data = ev[i].u64;
+			qe[i].qid = queue_id[i];
+			qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
+			qe[i].lock_id = ev[i].flow_id;
+			if (sched_type[i] == DLB2_SCHED_DIRECTED) {
+				struct dlb2_msg_info *info =
+					(struct dlb2_msg_info *)&qe[i].lock_id;
+
+				info->qid = queue_id[i];
+				info->sched_type = DLB2_SCHED_DIRECTED;
+				info->priority = qe[i].priority;
+			}
+			qe[i].u.event_type.major = ev[i].event_type;
+			qe[i].u.event_type.sub = ev[i].sub_event_type;
+		}
+		break;
+	}
+}
+
+static inline int
+dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
+			struct dlb2_port *qm_port,
+			const struct rte_event ev[],
+			uint8_t *sched_type,
+			uint8_t *queue_id)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	uint16_t *cached_credits = NULL;
+	struct dlb2_queue *qm_queue;
+
+	ev_queue = &dlb2->ev_queues[ev->queue_id];
+	qm_queue = &ev_queue->qm_queue;
+	*queue_id = qm_queue->id;
+
+	/* Ignore sched_type and hardware credits on release events */
+	if (ev->op == RTE_EVENT_OP_RELEASE)
+		goto op_check;
+
+	if (!qm_queue->is_directed) {
+		/* Load balanced destination queue */
+
+		if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_ldb_credits;
+
+		switch (ev->sched_type) {
+		case RTE_SCHED_TYPE_ORDERED:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
+			if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
+				DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
+					     *queue_id);
+				rte_errno = -EINVAL;
+				return 1;
+			}
+			*sched_type = DLB2_SCHED_ORDERED;
+			break;
+		case RTE_SCHED_TYPE_ATOMIC:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
+			*sched_type = DLB2_SCHED_ATOMIC;
+			break;
+		case RTE_SCHED_TYPE_PARALLEL:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
+			if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
+				*sched_type = DLB2_SCHED_ORDERED;
+			else
+				*sched_type = DLB2_SCHED_UNORDERED;
+			break;
+		default:
+			DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
+			DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
+			rte_errno = -EINVAL;
+			return 1;
+		}
+	} else {
+		/* Directed destination queue */
+
+		if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_dir_credits;
+
+		DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
+
+		*sched_type = DLB2_SCHED_DIRECTED;
+	}
+
+op_check:
+	switch (ev->op) {
+	case RTE_EVENT_OP_NEW:
+		/* Check that a sw credit is available */
+		if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		ev_port->inflight_credits--;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_FORWARD:
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_RELEASE:
+		ev_port->inflight_credits++;
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+
+		/* Replenish s/w credits if enough are cached */
+		dlb2_replenish_sw_credits(dlb2, ev_port);
+		break;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+	if (ev->op != RTE_EVENT_OP_RELEASE) {
+		DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
+		DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
+	}
+#endif
+
+	return 0;
+}
+
+static inline uint16_t
+dlb2_event_enqueue_burst(void *event_port,
+			 const struct rte_event events[],
+			 uint16_t num)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct process_local_port_data *port_data;
+	int i, cnt;
+
+	RTE_ASSERT(ev_port->enq_configured);
+	RTE_ASSERT(events != NULL);
+
+	cnt = 0;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
+		int j = 0;
+
+		for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
+			const struct rte_event *ev = &events[i + j];
+
+			if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
+						    &sched_types[j],
+						    &queue_ids[j]))
+				break;
+		}
+
+		if (j == 0)
+			break;
+
+		dlb2_event_build_hcws(qm_port, &events[i], j,
+				      sched_types, queue_ids);
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+
+		if (j < DLB2_NUM_QES_PER_CACHE_LINE)
+			break;
+	}
+
+	return cnt;
+}
+
+static inline uint16_t
+dlb2_event_enqueue(void *event_port,
+		   const struct rte_event events[])
+{
+	return dlb2_event_enqueue_burst(event_port, events, 1);
+}
+
+static uint16_t
+dlb2_event_enqueue_new_burst(void *event_port,
+			     const struct rte_event events[],
+			     uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
+static uint16_t
+dlb2_event_enqueue_forward_burst(void *event_port,
+				 const struct rte_event events[],
+				 uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2112,7 +2646,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
 	};
 
+	/* Expose PMD's eventdev interface */
+
 	dev->dev_ops = &dlb2_eventdev_entry_ops;
+	dev->enqueue = dlb2_event_enqueue;
+	dev->enqueue_burst = dlb2_event_enqueue_burst;
+	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
+	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v11 18/23] event/dlb2: add dequeue and its burst variants

[Timothy McDaniel]

Add support for dequeue, dequeue_burst, ...

DLB2 does not currently support interrupts, but instead use
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB2 supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst |  21 ++
 drivers/event/dlb2/dlb2.c     | 783 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 804 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index aa8bf01..b9f57fd 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -314,6 +314,27 @@ The PMD does not support the following configuration sequences:
 This sequence is not supported because the event device must be reconfigured
 before its ports or queues can be.
 
+Deferred Scheduling
+~~~~~~~~~~~~~~~~~~~
+
+The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
+dequeued event before returning from rte_event_dequeue_burst(). This frees the
+corresponding entries in the CQ, which enables the DLB2 to schedule more events
+to it.
+
+To support applications seeking finer-grained scheduling control -- for example
+deferring scheduling to get the best possible priority scheduling and
+load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
+the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
+call. This mode only applies to load-balanced event ports with dequeue depth of
+1.
+
+To enable deferred scheduling, use the defer_sched vdev argument like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,defer_sched=on
+
 Atomic Inflights Allocation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index faf45c8..631fe52 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -26,6 +26,7 @@
 #include <rte_log.h>
 #include <rte_malloc.h>
 #include <rte_mbuf.h>
+#include <rte_power_intrinsics.h>
 #include <rte_prefetch.h>
 #include <rte_ring.h>
 #include <rte_string_fns.h>
@@ -2227,6 +2228,32 @@ dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
 	dlb2_movdir64b(port_data->pp_addr, qe4);
 }
 
+static inline int
+dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_cq_pop_qe *qe;
+
+	RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
+
+	qe = qm_port->consume_qe;
+
+	qe->tokens = num - 1;
+
+	/* No store fence needed since no pointer is being sent, and CQ token
+	 * pops can be safely reordered with other HCWs.
+	 */
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	dlb2_movntdq_single(port_data->pp_addr, qe);
+
+	DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
+
+	qm_port->owed_tokens = 0;
+
+	return 0;
+}
+
 static inline void
 dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
 		   bool do_sfence,
@@ -2623,9 +2650,756 @@ dlb2_event_enqueue_forward_burst(void *event_port,
 	return dlb2_event_enqueue_burst(event_port, events, num);
 }
 
+static inline void
+dlb2_port_credits_inc(struct dlb2_port *qm_port, int num)
+{
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	/* increment port credits, and return to pool if exceeds threshold */
+	if (!qm_port->is_directed) {
+		qm_port->cached_ldb_credits += num;
+		if (qm_port->cached_ldb_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_LDB_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_ldb_credits -= batch_size;
+		}
+	} else {
+		qm_port->cached_dir_credits += num;
+		if (qm_port->cached_dir_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_DIR_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_dir_credits -= batch_size;
+		}
+	}
+}
+
+static inline int
+dlb2_dequeue_wait(struct dlb2_eventdev *dlb2,
+		  struct dlb2_eventdev_port *ev_port,
+		  struct dlb2_port *qm_port,
+		  uint64_t timeout,
+		  uint64_t start_ticks)
+{
+	struct process_local_port_data *port_data;
+	uint64_t elapsed_ticks;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	elapsed_ticks = rte_get_timer_cycles() - start_ticks;
+
+	/* Wait/poll time expired */
+	if (elapsed_ticks >= timeout) {
+		return 1;
+	} else if (dlb2->umwait_allowed) {
+		volatile struct dlb2_dequeue_qe *cq_base;
+		union {
+			uint64_t raw_qe[2];
+			struct dlb2_dequeue_qe qe;
+		} qe_mask;
+		uint64_t expected_value;
+		volatile uint64_t *monitor_addr;
+
+		qe_mask.qe.cq_gen = 1; /* set mask */
+
+		cq_base = port_data->cq_base;
+		monitor_addr = (volatile uint64_t *)(volatile void *)
+			&cq_base[qm_port->cq_idx];
+		monitor_addr++; /* cq_gen bit is in second 64bit location */
+
+		if (qm_port->gen_bit)
+			expected_value = qe_mask.raw_qe[1];
+		else
+			expected_value = 0;
+
+		rte_power_monitor(monitor_addr, expected_value,
+				  qe_mask.raw_qe[1], timeout + start_ticks,
+				  sizeof(uint64_t));
+
+		DLB2_INC_STAT(ev_port->stats.traffic.rx_umonitor_umwait, 1);
+	} else {
+		uint64_t poll_interval = RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL;
+		uint64_t curr_ticks = rte_get_timer_cycles();
+		uint64_t init_ticks = curr_ticks;
+
+		while ((curr_ticks - start_ticks < timeout) &&
+		       (curr_ticks - init_ticks < poll_interval))
+			curr_ticks = rte_get_timer_cycles();
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
+			 struct dlb2_port *qm_port,
+			 struct rte_event *events,
+			 struct dlb2_dequeue_qe *qes,
+			 int cnt)
+{
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	int i, num, evq_id;
+
+	for (i = 0, num = 0; i < cnt; i++) {
+		struct dlb2_dequeue_qe *qe = &qes[i];
+		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
+			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+			[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+		};
+
+		/* Fill in event information.
+		 * Note that flow_id must be embedded in the data by
+		 * the app, such as the mbuf RSS hash field if the data
+		 * buffer is a mbuf.
+		 */
+		if (unlikely(qe->error)) {
+			DLB2_LOG_ERR("QE error bit ON\n");
+			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
+			dlb2_consume_qe_immediate(qm_port, 1);
+			continue; /* Ignore */
+		}
+
+		events[num].u64 = qe->data;
+		events[num].flow_id = qe->flow_id;
+		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
+		events[num].event_type = qe->u.event_type.major;
+		events[num].sub_event_type = qe->u.event_type.sub;
+		events[num].sched_type = sched_type_map[qe->sched_type];
+		events[num].impl_opaque = qe->qid_depth;
+
+		/* qid not preserved for directed queues */
+		if (qm_port->is_directed)
+			evq_id = ev_port->link[0].queue_id;
+		else
+			evq_id = qid_mappings[qe->qid];
+
+		events[num].queue_id = evq_id;
+		DLB2_INC_STAT(
+			ev_port->stats.queue[evq_id].qid_depth[qe->qid_depth],
+			1);
+		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type], 1);
+		num++;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num);
+
+	return num;
+}
+
+static inline int
+dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
+			      struct dlb2_port *qm_port,
+			      struct rte_event *events,
+			      struct dlb2_dequeue_qe *qes)
+{
+	int sched_type_map[] = {
+		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+	};
+	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	__m128i sse_evt[2];
+
+	/* In the unlikely case that any of the QE error bits are set, process
+	 * them one at a time.
+	 */
+	if (unlikely(qes[0].error || qes[1].error ||
+		     qes[2].error || qes[3].error))
+		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
+						 qes, num_events);
+
+	events[0].u64 = qes[0].data;
+	events[1].u64 = qes[1].data;
+	events[2].u64 = qes[2].data;
+	events[3].u64 = qes[3].data;
+
+	/* Construct the metadata portion of two struct rte_events
+	 * in one 128b SSE register. Event metadata is constructed in the SSE
+	 * registers like so:
+	 * sse_evt[0][63:0]:   event[0]'s metadata
+	 * sse_evt[0][127:64]: event[1]'s metadata
+	 * sse_evt[1][63:0]:   event[2]'s metadata
+	 * sse_evt[1][127:64]: event[3]'s metadata
+	 */
+	sse_evt[0] = _mm_setzero_si128();
+	sse_evt[1] = _mm_setzero_si128();
+
+	/* Convert the hardware queue ID to an event queue ID and store it in
+	 * the metadata:
+	 * sse_evt[0][47:40]   = qid_mappings[qes[0].qid]
+	 * sse_evt[0][111:104] = qid_mappings[qes[1].qid]
+	 * sse_evt[1][47:40]   = qid_mappings[qes[2].qid]
+	 * sse_evt[1][111:104] = qid_mappings[qes[3].qid]
+	 */
+#define DLB_EVENT_QUEUE_ID_BYTE 5
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[0].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[1].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[2].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[3].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+
+	/* Convert the hardware priority to an event priority and store it in
+	 * the metadata, while also returning the queue depth status
+	 * value captured by the hardware, storing it in impl_opaque, which can
+	 * be read by the application but not modified
+	 * sse_evt[0][55:48]   = DLB2_TO_EV_PRIO(qes[0].priority)
+	 * sse_evt[0][63:56]   = qes[0].qid_depth
+	 * sse_evt[0][119:112] = DLB2_TO_EV_PRIO(qes[1].priority)
+	 * sse_evt[0][127:120] = qes[1].qid_depth
+	 * sse_evt[1][55:48]   = DLB2_TO_EV_PRIO(qes[2].priority)
+	 * sse_evt[1][63:56]   = qes[2].qid_depth
+	 * sse_evt[1][119:112] = DLB2_TO_EV_PRIO(qes[3].priority)
+	 * sse_evt[1][127:120] = qes[3].qid_depth
+	 */
+#define DLB_EVENT_PRIO_IMPL_OPAQUE_WORD 3
+#define DLB_BYTE_SHIFT 8
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[0].priority) |
+			(qes[0].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[1].priority) |
+			(qes[1].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[2].priority) |
+			(qes[2].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[3].priority) |
+			(qes[3].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+
+	/* Write the event type, sub event type, and flow_id to the event
+	 * metadata.
+	 * sse_evt[0][31:0]   = qes[0].flow_id |
+	 *			qes[0].u.event_type.major << 28 |
+	 *			qes[0].u.event_type.sub << 20;
+	 * sse_evt[0][95:64]  = qes[1].flow_id |
+	 *			qes[1].u.event_type.major << 28 |
+	 *			qes[1].u.event_type.sub << 20;
+	 * sse_evt[1][31:0]   = qes[2].flow_id |
+	 *			qes[2].u.event_type.major << 28 |
+	 *			qes[2].u.event_type.sub << 20;
+	 * sse_evt[1][95:64]  = qes[3].flow_id |
+	 *			qes[3].u.event_type.major << 28 |
+	 *			qes[3].u.event_type.sub << 20;
+	 */
+#define DLB_EVENT_EV_TYPE_DW 0
+#define DLB_EVENT_EV_TYPE_SHIFT 28
+#define DLB_EVENT_SUB_EV_TYPE_SHIFT 20
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[0].flow_id |
+			qes[0].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[0].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[1].flow_id |
+			qes[1].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[1].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[2].flow_id |
+			qes[2].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[2].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[3].flow_id |
+			qes[3].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT  |
+			qes[3].u.event_type.sub << DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+
+	/* Write the sched type to the event metadata. 'op' and 'rsvd' are not
+	 * set:
+	 * sse_evt[0][39:32]  = sched_type_map[qes[0].sched_type] << 6
+	 * sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6
+	 * sse_evt[1][39:32]  = sched_type_map[qes[2].sched_type] << 6
+	 * sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6
+	 */
+#define DLB_EVENT_SCHED_TYPE_BYTE 4
+#define DLB_EVENT_SCHED_TYPE_SHIFT 6
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[0].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[1].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[2].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[3].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+
+	/* Store the metadata to the event (use the double-precision
+	 * _mm_storeh_pd because there is no integer function for storing the
+	 * upper 64b):
+	 * events[0].event = sse_evt[0][63:0]
+	 * events[1].event = sse_evt[0][127:64]
+	 * events[2].event = sse_evt[1][63:0]
+	 * events[3].event = sse_evt[1][127:64]
+	 */
+	_mm_storel_epi64((__m128i *)&events[0].event, sse_evt[0]);
+	_mm_storeh_pd((double *)&events[1].event, (__m128d) sse_evt[0]);
+	_mm_storel_epi64((__m128i *)&events[2].event, sse_evt[1]);
+	_mm_storeh_pd((double *)&events[3].event, (__m128d) sse_evt[1]);
+
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[0].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[1].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[2].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[3].sched_type], 1);
+
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[0].queue_id].
+			qid_depth[qes[0].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[1].queue_id].
+			qid_depth[qes[1].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[2].queue_id].
+			qid_depth[qes[2].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[3].queue_id].
+			qid_depth[qes[3].qid_depth],
+		1);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num_events);
+
+	return num_events;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe_sparse(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe)
+{
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	uint8_t xor_mask[2] = {0x0F, 0x00};
+	const uint8_t and_mask = 0x0F;
+	__m128i *qes = (__m128i *)qe;
+	uint8_t gen_bits, gen_bit;
+	uintptr_t addr[4];
+	uint16_t idx;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+
+	idx = qm_port->cq_idx;
+
+	/* Load the next 4 QEs */
+	addr[0] = (uintptr_t)&cq_addr[idx];
+	addr[1] = (uintptr_t)&cq_addr[(idx +  4) & qm_port->cq_depth_mask];
+	addr[2] = (uintptr_t)&cq_addr[(idx +  8) & qm_port->cq_depth_mask];
+	addr[3] = (uintptr_t)&cq_addr[(idx + 12) & qm_port->cq_depth_mask];
+
+	/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */
+	rte_prefetch0(&cq_addr[(idx + 16) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 20) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 24) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 28) & qm_port->cq_depth_mask]);
+
+	/* Correct the xor_mask for wrap-around QEs */
+	gen_bit = qm_port->gen_bit;
+	xor_mask[gen_bit] ^= !!((idx +  4) > qm_port->cq_depth_mask) << 1;
+	xor_mask[gen_bit] ^= !!((idx +  8) > qm_port->cq_depth_mask) << 2;
+	xor_mask[gen_bit] ^= !!((idx + 12) > qm_port->cq_depth_mask) << 3;
+
+	/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is
+	 * valid, then QEs[0:N-1] are too.
+	 */
+	qes[3] = _mm_load_si128((__m128i *)(void *)addr[3]);
+	rte_compiler_barrier();
+	qes[2] = _mm_load_si128((__m128i *)(void *)addr[2]);
+	rte_compiler_barrier();
+	qes[1] = _mm_load_si128((__m128i *)(void *)addr[1]);
+	rte_compiler_barrier();
+	qes[0] = _mm_load_si128((__m128i *)(void *)addr[0]);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[gen_bit];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask;
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline void
+dlb2_inc_cq_idx(struct dlb2_port *qm_port, int cnt)
+{
+	uint16_t idx = qm_port->cq_idx_unmasked + cnt;
+
+	qm_port->cq_idx_unmasked = idx;
+	qm_port->cq_idx = idx & qm_port->cq_depth_mask;
+	qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) & 0x1;
+}
+
+static int
+dlb2_event_release(struct dlb2_eventdev *dlb2,
+		   uint8_t port_id,
+		   int n)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_eventdev_port *ev_port;
+	struct dlb2_port *qm_port;
+	int i, cnt;
+
+	if (port_id > dlb2->num_ports) {
+		DLB2_LOG_ERR("Invalid port id %d in dlb2-event_release\n",
+			     port_id);
+		rte_errno = -EINVAL;
+		return rte_errno;
+	}
+
+	ev_port = &dlb2->ev_ports[port_id];
+	qm_port = &ev_port->qm_port;
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	cnt = 0;
+
+	if (qm_port->is_directed) {
+		cnt = n;
+		goto sw_credit_update;
+	}
+
+	for (i = 0; i < n; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		int j;
+
+		/* Zero-out QEs */
+		qm_port->qe4[0].cmd_byte = 0;
+		qm_port->qe4[1].cmd_byte = 0;
+		qm_port->qe4[2].cmd_byte = 0;
+		qm_port->qe4[3].cmd_byte = 0;
+
+		for (j = 0; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++)
+			qm_port->qe4[j].cmd_byte = DLB2_COMP_CMD_BYTE;
+
+		qm_port->issued_releases += j;
+
+		if (j == 0)
+			break;
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+	}
+
+sw_credit_update:
+	/* each release returns one credit */
+	if (!ev_port->outstanding_releases) {
+		DLB2_LOG_ERR("Unrecoverable application error. Outstanding releases underflowed.\n");
+		rte_errno = -ENOTRECOVERABLE;
+		return rte_errno;
+	}
+
+	ev_port->outstanding_releases -= cnt;
+	ev_port->inflight_credits += cnt;
+
+	/* Replenish s/w credits if enough releases are performed */
+	dlb2_replenish_sw_credits(dlb2, ev_port);
+	return 0;
+}
+
+static inline int16_t
+dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
+		       struct dlb2_eventdev_port *ev_port,
+		       struct rte_event *events,
+		       uint16_t max_num,
+		       uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe_sparse(qm_port, qes);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail << 2);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							      qm_port,
+							      &events[num],
+							      &qes[0]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							 qm_port,
+							 &events[num],
+							 &qes[0],
+							 num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe,
+	     uint8_t *offset)
+{
+	uint8_t xor_mask[2][4] = { {0x0F, 0x0E, 0x0C, 0x08},
+				   {0x00, 0x01, 0x03, 0x07} };
+	uint8_t and_mask[4] = {0x0F, 0x0E, 0x0C, 0x08};
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	__m128i *qes = (__m128i *)qe;
+	uint64_t *cache_line_base;
+	uint8_t gen_bits;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	cq_addr = &cq_addr[qm_port->cq_idx];
+
+	cache_line_base = (void *)(((uintptr_t)cq_addr) & ~0x3F);
+	*offset = ((uintptr_t)cq_addr & 0x30) >> 4;
+
+	/* Load the next CQ cache line from memory. Pack these reads as tight
+	 * as possible to reduce the chance that DLB invalidates the line while
+	 * the CPU is reading it. Read the cache line backwards to ensure that
+	 * if QE[N] (N > 0) is valid, then QEs[0:N-1] are too.
+	 *
+	 * (Valid QEs start at &qe[offset])
+	 */
+	qes[3] = _mm_load_si128((__m128i *)&cache_line_base[6]);
+	qes[2] = _mm_load_si128((__m128i *)&cache_line_base[4]);
+	qes[1] = _mm_load_si128((__m128i *)&cache_line_base[2]);
+	qes[0] = _mm_load_si128((__m128i *)&cache_line_base[0]);
+
+	/* Evict the cache line ASAP */
+	rte_cldemote(cache_line_base);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[qm_port->gen_bit][*offset];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask[*offset];
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline int16_t
+dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
+		struct dlb2_eventdev_port *ev_port,
+		struct rte_event *events,
+		uint16_t max_num,
+		uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t offset;
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe(qm_port, qes, &offset);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							     qm_port,
+							     &events[num],
+							     &qes[offset]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							qm_port,
+							&events[num],
+							&qes[offset],
+							num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static uint16_t
+dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
+			 uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue(void *event_port, struct rte_event *ev, uint64_t wait)
+{
+	return dlb2_event_dequeue_burst(event_port, ev, 1, wait);
+}
+
+static uint16_t
+dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
+				uint16_t num, uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
+			  uint64_t wait)
+{
+	return dlb2_event_dequeue_burst_sparse(event_port, ev, 1, wait);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
+	struct dlb2_eventdev *dlb2;
+
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
@@ -2653,6 +3427,15 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	dev->enqueue_burst = dlb2_event_enqueue_burst;
 	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
 	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
+
+	dlb2 = dev->data->dev_private;
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE) {
+		dev->dequeue = dlb2_event_dequeue_sparse;
+		dev->dequeue_burst = dlb2_event_dequeue_burst_sparse;
+	} else {
+		dev->dequeue = dlb2_event_dequeue;
+		dev->dequeue_burst = dlb2_event_dequeue_burst;
+	}
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v11 19/23] event/dlb2: add eventdev stop and close

[Timothy McDaniel]

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 257 +++++++++++++++++++++++++++--
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c |  91 ++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  52 ++++++
 5 files changed, 396 insertions(+), 16 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 631fe52..4b8f45f 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -71,21 +71,6 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
-/*
- * DUMMY - added so that xstats path will compile/link.
- * Will be replaced by real version in a subsequent
- * patch.
- */
-uint32_t
-dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
-		     struct dlb2_eventdev_queue *queue)
-{
-	RTE_SET_USED(dlb2);
-	RTE_SET_USED(queue);
-
-	return 0;
-}
-
 static void
 dlb2_free_qe_mem(struct dlb2_port *qm_port)
 {
@@ -1878,7 +1863,6 @@ dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
 		return 0; /* Ignore and return success */
 	}
 
-	/* FIXME: How to handle unlink on a directed port? */
 	if (ev_port->qm_port.is_directed) {
 		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
 			     ev_port->id);
@@ -3396,6 +3380,245 @@ dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
 }
 
 static void
+dlb2_flush_port(struct rte_eventdev *dev, int port_id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	eventdev_stop_flush_t flush;
+	struct rte_event ev;
+	uint8_t dev_id;
+	void *arg;
+	int i;
+
+	flush = dev->dev_ops->dev_stop_flush;
+	dev_id = dev->data->dev_id;
+	arg = dev->data->dev_stop_flush_arg;
+
+	while (rte_event_dequeue_burst(dev_id, port_id, &ev, 1, 0)) {
+		if (flush)
+			flush(dev_id, ev, arg);
+
+		if (dlb2->ev_ports[port_id].qm_port.is_directed)
+			continue;
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+	}
+
+	/* Enqueue any additional outstanding releases */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = dlb2->ev_ports[port_id].outstanding_releases; i > 0; i--)
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+}
+
+static uint32_t
+dlb2_get_ldb_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_ldb_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_ldb_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_ldb_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static uint32_t
+dlb2_get_dir_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_dir_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_dir_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_dir_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	if (queue->qm_queue.is_directed)
+		return dlb2_get_dir_queue_depth(dlb2, queue);
+	else
+		return dlb2_get_ldb_queue_depth(dlb2, queue);
+}
+
+static bool
+dlb2_queue_is_empty(struct dlb2_eventdev *dlb2,
+		    struct dlb2_eventdev_queue *queue)
+{
+	return dlb2_get_queue_depth(dlb2, queue) == 0;
+}
+
+static bool
+dlb2_linked_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0)
+			continue;
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static bool
+dlb2_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static void
+dlb2_drain(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_port *ev_port = NULL;
+	uint8_t dev_id;
+	int i;
+
+	dev_id = dev->data->dev_id;
+
+	while (!dlb2_linked_queues_empty(dlb2)) {
+		/* Flush all the ev_ports, which will drain all their connected
+		 * queues.
+		 */
+		for (i = 0; i < dlb2->num_ports; i++)
+			dlb2_flush_port(dev, i);
+	}
+
+	/* The queues are empty, but there may be events left in the ports. */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_flush_port(dev, i);
+
+	/* If the domain's queues are empty, we're done. */
+	if (dlb2_queues_empty(dlb2))
+		return;
+
+	/* Else, there must be at least one unlinked load-balanced queue.
+	 * Select a load-balanced port with which to drain the unlinked
+	 * queue(s).
+	 */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		if (!ev_port->qm_port.is_directed)
+			break;
+	}
+
+	if (i == dlb2->num_ports) {
+		DLB2_LOG_ERR("internal error: no LDB ev_ports\n");
+		return;
+	}
+
+	rte_errno = 0;
+	rte_event_port_unlink(dev_id, ev_port->id, NULL, 0);
+
+	if (rte_errno) {
+		DLB2_LOG_ERR("internal error: failed to unlink ev_port %d\n",
+			     ev_port->id);
+		return;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		uint8_t qid, prio;
+		int ret;
+
+		if (dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			continue;
+
+		qid = i;
+		prio = 0;
+
+		/* Link the ev_port to the queue */
+		ret = rte_event_port_link(dev_id, ev_port->id, &qid, &prio, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to link ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+
+		/* Flush the queue */
+		while (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			dlb2_flush_port(dev, ev_port->id);
+
+		/* Drain any extant events in the ev_port. */
+		dlb2_flush_port(dev, ev_port->id);
+
+		/* Unlink the ev_port from the queue */
+		ret = rte_event_port_unlink(dev_id, ev_port->id, &qid, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to unlink ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+	}
+}
+
+static void
+dlb2_eventdev_stop(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_DBG("Internal error: already stopped\n");
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	} else if (dlb2->run_state != DLB2_RUN_STATE_STARTED) {
+		DLB2_LOG_ERR("Internal error: bad state %d for dev_stop\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPING;
+
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	dlb2_drain(dev);
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPED;
+}
+
+static int
+dlb2_eventdev_close(struct rte_eventdev *dev)
+{
+	dlb2_hw_reset_sched_domain(dev, false);
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3405,6 +3628,8 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.dev_start        = dlb2_eventdev_start,
+		.dev_stop         = dlb2_eventdev_stop,
+		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a86191d..5471dd8 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -66,3 +66,9 @@ int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				     struct dlb2_start_domain_args *cfg);
+
+int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 7ead374..b508eb0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -65,4 +65,10 @@ extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				struct dlb2_start_domain_args *cfg);
+
+extern int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+extern int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index c14a2f3..ae5ef2f 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5934,3 +5934,94 @@ dlb2_hw_start_domain(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_dir_queue_depth(hw, queue);
+
+	return 0;
+}
+
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+
+	dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index fff2e57..632c4e0 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -527,6 +527,56 @@ dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_ldb_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_ldb_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_dir_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret = 0;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_dir_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -544,6 +594,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth;
+	dlb2_iface_get_dir_queue_depth = dlb2_pf_get_dir_queue_depth;
 	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v11 20/23] event/dlb2: add PMD's token pop public interface

[Timothy McDaniel]

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/api/doxy-api-index.md         |  3 +-
 doc/api/doxy-api.conf.in          |  1 +
 drivers/event/dlb2/dlb2.c         | 65 +++++++++++++++++++++++++++++++----
 drivers/event/dlb2/dlb2_priv.h    |  3 ++
 drivers/event/dlb2/meson.build    |  6 ++--
 drivers/event/dlb2/rte_pmd_dlb2.c | 39 +++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2.h | 72 +++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/version.map    |  6 ++++
 8 files changed, 186 insertions(+), 9 deletions(-)
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index b865a51..e1836dc 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -52,7 +52,8 @@ The public API headers are grouped by topics:
   [dpaa2_mempool]      (@ref rte_dpaa2_mempool.h),
   [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
-  [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
+  [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h),
+  [dlb2]               (@ref rte_pmd_dlb2.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index c5b01a1..535321f 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -7,6 +7,7 @@ USE_MDFILE_AS_MAINPAGE  = @TOPDIR@/doc/api/doxy-api-index.md
 INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/drivers/bus/vdev \
                           @TOPDIR@/drivers/crypto/scheduler \
+                          @TOPDIR@/drivers/event/dlb2 \
                           @TOPDIR@/drivers/mempool/dpaa2 \
                           @TOPDIR@/drivers/net/ark \
                           @TOPDIR@/drivers/net/bnxt \
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 4b8f45f..e8b4446 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1197,7 +1197,7 @@ dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
 	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
 
 	qm_port->dequeue_depth = dequeue_depth;
-
+	qm_port->token_pop_thresh = dequeue_depth;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -1365,6 +1365,8 @@ dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
 
 	qm_port->dequeue_depth = dequeue_depth;
 
+	/* Directed ports are auto-pop, by default. */
+	qm_port->token_pop_mode = AUTO_POP;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -2253,6 +2255,18 @@ dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
 }
 
 static inline void
+dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
+{
+	struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
+	int num = qm_port->owed_tokens;
+
+	qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
+	qe[idx].tokens = num - 1;
+
+	qm_port->owed_tokens = 0;
+}
+
+static inline void
 dlb2_event_build_hcws(struct dlb2_port *qm_port,
 		      const struct rte_event ev[],
 		      int num,
@@ -2600,6 +2614,14 @@ dlb2_event_enqueue_burst(void *event_port,
 		dlb2_event_build_hcws(qm_port, &events[i], j,
 				      sched_types, queue_ids);
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
@@ -2608,6 +2630,11 @@ dlb2_event_enqueue_burst(void *event_port,
 			break;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
 	return cnt;
 }
 
@@ -3085,11 +3112,25 @@ dlb2_event_release(struct dlb2_eventdev *dlb2,
 		if (j == 0)
 			break;
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
 sw_credit_update:
 	/* each release returns one credit */
 	if (!ev_port->outstanding_releases) {
@@ -3173,8 +3214,8 @@ dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3300,8 +3341,8 @@ dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3316,6 +3357,7 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 			 uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3331,6 +3373,9 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3350,6 +3395,7 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 				uint16_t num, uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3365,6 +3411,9 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3669,7 +3718,7 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    struct dlb2_devargs *dlb2_args)
 {
 	struct dlb2_eventdev *dlb2;
-	int err;
+	int err, i;
 
 	dlb2 = dev->data->dev_private;
 
@@ -3719,6 +3768,10 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Initialize each port's token pop mode */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
+		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 61567a6..b73cf3f 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -12,6 +12,7 @@
 #include <rte_config.h>
 #include "dlb2_user.h"
 #include "dlb2_log.h"
+#include "rte_pmd_dlb2.h"
 
 #ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
 #define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
@@ -290,6 +291,7 @@ struct dlb2_port {
 	bool gen_bit;
 	uint16_t dir_credits;
 	uint32_t dequeue_depth;
+	enum dlb2_token_pop_mode token_pop_mode;
 	union dlb2_port_config cfg;
 	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
 	uint16_t cached_ldb_credits;
@@ -298,6 +300,7 @@ struct dlb2_port {
 	bool int_armed;
 	uint16_t owed_tokens;
 	int16_t issued_releases;
+	int16_t token_pop_thresh;
 	int cq_depth;
 	uint16_t cq_idx;
 	uint16_t cq_idx_unmasked;
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index a80d7ab..2af72e2 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,3 +1,4 @@
+
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
@@ -12,9 +13,10 @@ sources = files('dlb2.c',
 		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
-		'pf/base/dlb2_resource.c'
+		'pf/base/dlb2_resource.c',
+		'rte_pmd_dlb2.c'
 )
 
-headers = files()
+headers = files('rte_pmd_dlb2.h')
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
new file mode 100644
index 0000000..b09b585
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "rte_pmd_dlb2.h"
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode)
+{
+	struct dlb2_eventdev *dlb2;
+	struct rte_eventdev *dev;
+
+	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+	dev = &rte_eventdevs[dev_id];
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (mode >= NUM_TOKEN_POP_MODES)
+		return -EINVAL;
+
+	/* The event device must be configured, but not yet started */
+	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
+		return -EINVAL;
+
+	/* The token pop mode must be set before configuring the port */
+	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
+		return -EINVAL;
+
+	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
new file mode 100644
index 0000000..74399db
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+/*!
+ *  @file      rte_pmd_dlb2.h
+ *
+ *  @brief     DLB PMD-specific functions
+ */
+
+#ifndef _RTE_PMD_DLB2_H_
+#define _RTE_PMD_DLB2_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Selects the token pop mode for a DLB2 port.
+ */
+enum dlb2_token_pop_mode {
+	/* Pop the CQ tokens immediately after dequeueing. */
+	AUTO_POP,
+	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
+	 * Supported on load-balanced ports only.
+	 */
+	DELAYED_POP,
+	/* Pop the CQ tokens during next dequeue operation. */
+	DEFERRED_POP,
+
+	/* NUM_TOKEN_POP_MODES must be last */
+	NUM_TOKEN_POP_MODES
+};
+
+/*!
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Configure the token pop mode for a DLB2 port. By default, all ports use
+ * AUTO_POP. This function must be called before calling rte_event_port_setup()
+ * for the port, but after calling rte_event_dev_configure().
+ *
+ * @param dev_id
+ *    The identifier of the event device.
+ * @param port_id
+ *    The identifier of the event port.
+ * @param mode
+ *    The token pop mode.
+ *
+ * @return
+ * - 0: Success
+ * - EINVAL: Invalid dev_id, port_id, or mode
+ * - EINVAL: The DLB2 is not configured, is already running, or the port is
+ *   already setup
+ */
+
+__rte_experimental
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_DLB2_H_ */
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
index 4a76d1d..b1e4dff 100644
--- a/drivers/event/dlb2/version.map
+++ b/drivers/event/dlb2/version.map
@@ -1,3 +1,9 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmd_dlb2_set_token_pop_mode;
+};
-- 
2.6.4

[dpdk-dev] [PATCH v11 21/23] event/dlb2: add PMD self-tests

[Timothy McDaniel]

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 app/test/test_eventdev.c           |    7 +
 drivers/event/dlb2/dlb2.c          |    1 +
 drivers/event/dlb2/dlb2_selftest.c | 1524 ++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build     |    3 +-
 4 files changed, 1534 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c

diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
index 62019c1..bcfaa53 100644
--- a/app/test/test_eventdev.c
+++ b/app/test/test_eventdev.c
@@ -1030,6 +1030,12 @@ test_eventdev_selftest_dpaa2(void)
 	return test_eventdev_selftest_impl("event_dpaa2", "");
 }
 
+static int
+test_eventdev_selftest_dlb2(void)
+{
+	return test_eventdev_selftest_impl("dlb2_event", "");
+}
+
 REGISTER_TEST_COMMAND(eventdev_common_autotest, test_eventdev_common);
 REGISTER_TEST_COMMAND(eventdev_selftest_sw, test_eventdev_selftest_sw);
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
@@ -1037,3 +1043,4 @@ REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
 		test_eventdev_selftest_octeontx2);
 REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2, test_eventdev_selftest_dpaa2);
+REGISTER_TEST_COMMAND(eventdev_selftest_dlb2, test_eventdev_selftest_dlb2);
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index e8b4446..86299dd 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3692,6 +3692,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
 		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+		.dev_selftest     = test_dlb2_eventdev,
 	};
 
 	/* Expose PMD's eventdev interface */
diff --git a/drivers/event/dlb2/dlb2_selftest.c b/drivers/event/dlb2/dlb2_selftest.c
new file mode 100644
index 0000000..e300067
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_selftest.c
@@ -0,0 +1,1524 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <unistd.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_eventdev.h>
+#include <rte_pause.h>
+
+#include "dlb2_priv.h"
+#include "rte_pmd_dlb2.h"
+
+#define MAX_PORTS 32
+#define MAX_QIDS 32
+#define DEFAULT_NUM_SEQ_NUMS 64
+
+static struct rte_mempool *eventdev_func_mempool;
+static int evdev;
+
+struct test {
+	struct rte_mempool *mbuf_pool;
+	int nb_qids;
+};
+
+/* initialization and config */
+static inline int
+init(struct test *t, int nb_queues, int nb_ports)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	memset(t, 0, sizeof(*t));
+
+	t->mbuf_pool = eventdev_func_mempool;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = nb_queues;
+	config.nb_event_ports = nb_ports;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0)
+		printf("%d: Error configuring device\n", __LINE__);
+
+	return ret;
+}
+
+static inline int
+create_ports(int num_ports)
+{
+	int i;
+
+	if (num_ports > MAX_PORTS)
+		return -1;
+
+	for (i = 0; i < num_ports; i++) {
+		struct rte_event_port_conf conf;
+
+		if (rte_event_port_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default port conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		if (rte_event_port_setup(evdev, i, &conf) < 0) {
+			printf("%d: Error setting up port %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+create_lb_qids(struct test *t, int num_qids, uint32_t flags)
+{
+	int i;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		struct rte_event_queue_conf conf;
+
+		if (rte_event_queue_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default queue conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		conf.schedule_type = flags;
+
+		if (conf.schedule_type == RTE_SCHED_TYPE_PARALLEL)
+			conf.nb_atomic_order_sequences = 0;
+		else
+			conf.nb_atomic_order_sequences = DEFAULT_NUM_SEQ_NUMS;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+static inline int
+create_atomic_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ATOMIC);
+}
+
+/* destruction */
+static inline void
+cleanup(void)
+{
+	int ret = 0;
+
+	rte_event_dev_stop(evdev);
+	ret = rte_event_dev_close(evdev);
+
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+};
+
+static inline int
+enqueue_timeout(uint8_t port_id, struct rte_event *ev, uint64_t tmo_us)
+{
+	const uint64_t start = rte_get_timer_cycles();
+	const uint64_t ticks = (tmo_us * rte_get_timer_hz()) / 1E6;
+
+	while ((rte_get_timer_cycles() - start) < ticks) {
+		if (rte_event_enqueue_burst(evdev, port_id, ev, 1) == 1)
+			return 0;
+
+		if (rte_errno != -ENOSPC) {
+			printf("enqueue_burst returned rte_errno %d\n",
+			       rte_errno);
+			return -1;
+		}
+	}
+	printf("%s time out\n", __func__);
+	return -1;
+}
+
+static void
+flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused)
+{
+	rte_pktmbuf_free(event.mbuf);
+}
+
+static int
+test_stop_flush(struct test *t) /* test to check we can properly flush events */
+{
+	struct rte_event ev;
+	uint32_t dequeue_depth;
+	unsigned int i, count;
+	uint8_t queue_id;
+
+	ev.op = RTE_EVENT_OP_NEW;
+
+	if (init(t, 2, 1) < 0 ||
+	    create_ports(1) < 0 ||
+	    create_atomic_qids(t, 2) < 0) {
+		printf("%d: Error initializing device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_port_link(evdev, 0, NULL, NULL, 0) != 2) {
+		printf("%d: Error linking queues to the port\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: Error with start call\n", __LINE__);
+		goto err;
+	}
+
+	/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked
+	 * queue.
+	 */
+	queue_id = 1;
+
+	if (rte_event_port_unlink(evdev, 0, &queue_id, 1) != 1) {
+		printf("%d: Error unlinking queue 1 from port\n", __LINE__);
+		goto err;
+	}
+
+	count = rte_mempool_avail_count(t->mbuf_pool);
+
+	if (rte_event_port_attr_get(evdev,
+				    0,
+				    RTE_EVENT_PORT_ATTR_DEQ_DEPTH,
+				    &dequeue_depth)) {
+		printf("%d: Error retrieveing dequeue depth\n", __LINE__);
+		goto err;
+	}
+
+	/* Send QEs to queue 0 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 0;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Send QEs to queue 1 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 1;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Now the DLB is scheduling events from the port to the IQ, and at
+	 * least one event should be remaining in each queue.
+	 */
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, flush, NULL)) {
+		printf("%d: Error installing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+
+	if (count != rte_mempool_avail_count(t->mbuf_pool)) {
+		printf("%d: Error executing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, NULL, NULL)) {
+		printf("%d: Error uninstalling the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	return 0;
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_single_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Create a directed port */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) == 0) {
+		printf("%d: port 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Create a load-balanced port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a directed queue */
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed queue */
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) == 0) {
+		printf("%d: queue 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a load-balanced queue */
+	queue_conf.event_queue_cfg = 0;
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to link directed and load-balanced resources */
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 0 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 1 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Link ports to queues */
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+#define NUM_LDB_PORTS 64
+#define NUM_LDB_QUEUES 32
+
+static int
+test_info_get(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	config.nb_event_ports = info.max_event_ports;
+	config.nb_event_queues = NUM_LDB_QUEUES + info.max_event_ports / 2;
+	config.nb_single_link_event_port_queues = info.max_event_ports / 2;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		goto err;
+	}
+
+	/* The DLB2 PMD only reports load-balanced ports and queues in its
+	 * info_get function. Confirm that these values don't include the
+	 * directed port or queue counts.
+	 */
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret) {
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+		return -1;
+	}
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+static int
+test_reconfiguration_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 2 LDB queues */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+			printf("%d: P0->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P1->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+			printf("%d: P1->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Configure device with 2 DIR ports and 2 DIR queues */
+	config.nb_single_link_event_port_queues = 2;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q1 */
+	queue_id = 1;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: P0->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Link P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: P1->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_queues = 5;
+	config.nb_event_ports = 5;
+	config.nb_single_link_event_port_queues = 1;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < config.nb_event_queues - 1; i++) {
+		port_conf.event_port_cfg = 0;
+		queue_conf.event_queue_cfg = 0;
+
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: P%d->Q%d link expected to succeed\n",
+			       __LINE__, i, i);
+			goto err;
+		}
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+		printf("%d: port %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+		printf("%d: queue %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	queue_id = i;
+
+	if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+		printf("%d: P%d->Q%d link expected to succeed\n",
+		       __LINE__, i, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_ports += 1;
+
+	/* Reconfigure device with 1 more load-balanced port */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Configure the new port */
+	if (rte_event_port_setup(evdev, config.nb_event_ports - 1,
+				 &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_load_balanced_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_directed_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 DIR port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (ev.queue_id != 0) {
+		printf("%d: invalid dequeued event queue ID (%d)\n",
+		       __LINE__, ev.queue_id);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_deferred_sched(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	const int num_events = 128;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 1 queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 1, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.dequeue_depth = 1;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.schedule_type = RTE_SCHED_TYPE_PARALLEL;
+	queue_conf.nb_atomic_order_sequences = 0;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 128 NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_PARALLEL;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue one event from port 0 */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue (and release) all other events from port 1. Deferred
+	 * scheduling ensures no other events are scheduled to port 0 without a
+	 * subsequent rte_event_dequeue_burst() call.
+	 */
+	for (i = 0; i < num_events - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 1, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		if (rte_event_enqueue_burst(evdev, 1, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_delayed_pop(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	int ret, i, num_events;
+	struct rte_event ev;
+	uint8_t queue_id;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DELAYED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	num_events = 2 * port_conf.dequeue_depth;
+
+	/* Enqueue 2 * dequeue_depth NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue dequeue_depth events but only release dequeue_depth - 2.
+	 * Delayed pop won't perform the pop and no more events will be
+	 * scheduled.
+	 */
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = 0; i < port_conf.dequeue_depth - 2; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	ret = rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout);
+	if (ret != 0) {
+		printf("%d: event dequeue expected to fail (ret = %d)\n",
+		       __LINE__, ret);
+		goto err;
+	}
+
+	/* Release one more event. This will trigger the token pop, and
+	 * dequeue_depth - 1 more events will be scheduled to the device.
+	 */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: RELEASE enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 0) {
+		printf("%d: event dequeue expected to fail\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+do_selftest(void)
+{
+	struct test t;
+	int ret;
+
+	/* Only create mbuf pool once, reuse for each test run */
+	if (!eventdev_func_mempool) {
+		eventdev_func_mempool =
+			rte_pktmbuf_pool_create("EVENTDEV_DLB2_ST_POOL",
+						(1 << 12), /* 4k buffers */
+						32 /*MBUF_CACHE_SIZE*/,
+						0,
+						512, /* use very small mbufs */
+						rte_socket_id());
+		if (!eventdev_func_mempool) {
+			printf("ERROR creating mempool\n");
+			goto test_fail;
+		}
+	}
+	t.mbuf_pool = eventdev_func_mempool;
+
+	printf("*** Running Stop Flush test...\n");
+	ret = test_stop_flush(&t);
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Single Link test...\n");
+	ret = test_single_link();
+	if (ret != 0) {
+		printf("ERROR - Single Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Info Get test...\n");
+	ret = test_info_get();
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Reconfiguration Link test...\n");
+	ret = test_reconfiguration_link();
+	if (ret != 0) {
+		printf("ERROR - Reconfiguration Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Load-Balanced Traffic test...\n");
+	ret = test_load_balanced_traffic();
+	if (ret != 0) {
+		printf("ERROR - Load-Balanced Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Directed Traffic test...\n");
+	ret = test_directed_traffic();
+	if (ret != 0) {
+		printf("ERROR - Directed Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Deferred Scheduling test...\n");
+	ret = test_deferred_sched();
+	if (ret != 0) {
+		printf("ERROR - Deferred Scheduling test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Delayed Pop test...\n");
+	ret = test_delayed_pop();
+	if (ret != 0) {
+		printf("ERROR - Delayed Pop test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	return 0;
+
+test_fail:
+	return -1;
+}
+
+int
+test_dlb2_eventdev(void)
+{
+	const char *dlb2_eventdev_name = "dlb2_event";
+	uint8_t num_evdevs = rte_event_dev_count();
+	int i, ret = 0;
+	int found = 0, skipped = 0, passed = 0, failed = 0;
+	struct rte_event_dev_info info;
+
+	for (i = 0; found + skipped < num_evdevs && i < RTE_EVENT_MAX_DEVS;
+	     i++) {
+		ret = rte_event_dev_info_get(i, &info);
+		if (ret < 0)
+			continue;
+
+		/* skip non-dlb2 event devices */
+		if (strncmp(info.driver_name, dlb2_eventdev_name,
+			    sizeof(*info.driver_name)) != 0) {
+			skipped++;
+			continue;
+		}
+
+		evdev = rte_event_dev_get_dev_id(info.driver_name);
+		if (evdev < 0) {
+			printf("Could not get dev_id for eventdev with name %s, i=%d\n",
+			       info.driver_name, i);
+			skipped++;
+			continue;
+		}
+		found++;
+		printf("Running selftest on eventdev %s\n", info.driver_name);
+		ret = do_selftest();
+		if (ret == 0) {
+			passed++;
+			printf("Selftest passed for eventdev %s\n",
+			       info.driver_name);
+		} else {
+			failed++;
+			printf("Selftest failed for eventdev %s, err=%d\n",
+			       info.driver_name, ret);
+		}
+	}
+
+	printf("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n",
+	       found, skipped, passed, failed);
+	return ret;
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 2af72e2..bcda3dd 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -14,7 +14,8 @@ sources = files('dlb2.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c',
-		'rte_pmd_dlb2.c'
+		'rte_pmd_dlb2.c',
+		'dlb2_selftest.c'
 )
 
 headers = files('rte_pmd_dlb2.h')
-- 
2.6.4

[dpdk-dev] [PATCH v11 22/23] event/dlb2: add queue and port release

[Timothy McDaniel]

DLB does not support reconfiguring individual queues
or ports on the fly. The entire device must be reconfigured.
Previously allocated port QE and memzone memory
is freed in this patch.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 86299dd..248296d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3668,6 +3668,28 @@ dlb2_eventdev_close(struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(id);
+
+	/* This function intentionally left blank. */
+}
+
+static void
+dlb2_eventdev_port_release(void *port)
+{
+	struct dlb2_eventdev_port *ev_port = port;
+	struct dlb2_port *qm_port;
+
+	if (ev_port) {
+		qm_port = &ev_port->qm_port;
+		if (qm_port->config_state == DLB2_CONFIGURED)
+			dlb2_free_qe_mem(qm_port);
+	}
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3681,8 +3703,10 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
+		.queue_release    = dlb2_eventdev_queue_release,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_release     = dlb2_eventdev_port_release,
 		.port_link        = dlb2_eventdev_port_link,
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
-- 
2.6.4

[dpdk-dev] [PATCH v11 23/23] event/dlb2: add timeout ticks entry point

[Timothy McDaniel]

Adds the timeout ticks conversion function.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 248296d..d42e48b 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3689,6 +3689,18 @@ dlb2_eventdev_port_release(void *port)
 	}
 }
 
+static int
+dlb2_eventdev_timeout_ticks(struct rte_eventdev *dev, uint64_t ns,
+			    uint64_t *timeout_ticks)
+{
+	RTE_SET_USED(dev);
+	uint64_t cycles_per_ns = rte_get_timer_hz() / 1E9;
+
+	*timeout_ticks = ns * cycles_per_ns;
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -3711,6 +3723,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
 				    dlb2_eventdev_port_unlinks_in_progress,
+		.timeout_ticks    = dlb2_eventdev_timeout_ticks,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

Re: [dpdk-dev] [PATCH v11 00/23] Add DLB2 PMD

[Jerin Jacob]

On Mon, Nov 2, 2020 at 5:07 AM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> The following patch series adds support for a new eventdev PMD. The DLB2
> PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
> hardware.
> The DLB2 is a PCIe device that provides load-balanced, prioritized
> scheduling of core-to-core communication. The device consists of
> queues and arbiters that connect producer and consumer cores, and
> implements load-balanced queueing features including:
> - Lock-free multi-producer/multi-consumer operation.
> - Multiple priority levels for varying traffic types.
> - 'Direct' traffic (i.e. multi-producer/single-consumer)
> - Simple unordered load-balanced distribution.
> - Atomic lock-free load balancing across multiple consumers.
> - Queue element reordering feature allowing ordered load-balanced
>   distribution.
>
> The DLB2 hardware supports both load balanced and directed ports and
> queues. Unlike other eventdev devices already in the repo,  not all
> DLB2 ports and queues are equally capable. In particular, directed
> ports are limited to a single link, and must be connected to a
> directed queue. Additionally, even though LDB ports may link multiple
> queues,
> the number of queues that may be linked is limited by hardware.
>
> While reviewing the code, please be aware that this PMD has full
> control over the DLB2 hardware. Intel will be extending the DLB2 PMD
> in the future (not as part of this first series) with a mode that we
> refer to as the bifurcated PMD. The bifurcated PMD communicates with a
> kernel driver to configure the device, ports, and queues, and memory
> maps device MMIO so datapath operations occur purely in user-space.
> Note that the DLB2 hardware is a successor of the DLB hardware, and
> as such is structured similarly, both in terms of code layout and
> implementation.
>
> The framework to support both the PF PMD and bifurcated PMD exists in
> this patchset, and is why the iface.[ch] layer is present.


Finally, Series applied to dpdk-next-eventdev/for-main with the
following fix[1]. Thanks.

Thanks, @David Marchand for your final review cycles.


[1]

 [for-main]dell[dpdk-next-eventdev] $ ./devtools/check-git-log.sh -n 23
Wrong tag:
        Reviewed-by: Chen, Mike Ximing <mike.ximing.chen@intel.com>



>
> Changes since V10
> =================
> - converted printfs in dlb2/pf/dlb2_main.c to DLB2_LOG
> - fixed a repeated word error in dlb2/pf/base/osdep_bitmap.h
> - caught up with marking the patches that Gage reviewed
>
> Changes since V9
> ================
> Address comments from David Marchand:
> - this patch-set is based on Nov 1, 2020 dpdk-next-eventdev
> - fix docs build (doxy-api.conf.in and doxy-api-index.md)
> - restore blank line in MAINTAINERS file
> - move dlb announcement in release_20_11.rst after ethdev
> - use headers = files() for exported meson public headers
> - fix a typo in 'add documentation ..." commit message
> - use eal version of cldemote
> - convert a couple of printfs to LOG messages
> - all patches build incrementally (gcc), and checkpatches reports
>   success
> - I am not able to run clang locally. If clang errors are still
>   present I will ask IT to install clang on a build server tomorrow.
>
> Changes since V8:
> =================
> - Fixed format errors in doc/api/doxy-api-index.md
> - Delayed introduction of dlb2_consume_qe_immediate until
>   add-dequeue-and-its-burst-variants.patch
> - Delayed introduction of dlb2_construct_token_pop_qe until
>   add-PMD-s-token-pop-public-interface.patch
>
> Changes since V7:
> ================
> - fix CENTOS build error: use __m128i instead of __v2di with
>   _mm_stream_si128
> - deleted unused dlb2_umwait and dlb2_umonitor functions
>
> Changes since V6:
> =================
> - removed unused function, fixing build error
> - fixed typo in port_setup commit message
> - this patch series is based on dpdk-next-eventdev
>
> Changes since V5:
> ================
> - convert to use rte_power_monitor patches
> - replace __builtin_ia32_movntdq() with _mm_stream_si128()
> - remove unused functions in dlb_selftest.c
> - workaround for RHEL 7 gcc brace bug
>
> Changes since V4:
> ================
> - moved introduction of dlb in relnotes_20_11 to first patch in series
> - fixed underlines in dlb.rst that were too short
> - note that the code still uses its private byte-encoded versions of
>   umonitor/umwait, rather than the new functions in the power
>   patch that are built on top of those intrinsics. This is intentional.
>
> Changes since V3:
> ================
> - updated MAINTAINERS file to alphabetically insert DLB
> - don't create RTE_ symbols in pmd
> - converted to use version.map scheme
> - converted to use .._master_lcore instead of .._main_lcore
> - this patch set is based on dpdk-next-eventdev
>
> Changes since V2:
> ================
> - fixed meson conditional build. Moved test into driver’s meson.build
>   file instead of event/meson.build
> - documentation is populated as associated code is introduced
> - add log_register in add dynamic logging patch
> - rename RTE_xxx symbol(s) as DLB2_xxx
> - replaced function ptr enqueue_four with direct call to movdir64b
> - remove unused port_pages
> - broke up probe patch into 3 smaller patches for easier review
> - changed param order of movdir64b/movntdq to match intrinsics
> - added self to MAINTAINERS files
> - squashed announcement of availability into last patch in series
> - correct spelling errors and delete repeated words
> - DPDK_21.0 -> DPDK 21 in map file
> - add experimental banner to public structs and APIs
>
> Changes since V1:
> =================
> - implement changes requested in code reviews by Gage Eads and Mike
>   Chen
> - fix a memzone leak
> - convert to use eal rte-cpuflags patch from Liang Ma
>
> Known Issues:
> - the documentation contained in dlb2.rst is not complete, and
>   should be updated to include command line parameters (class of service,
>   etc ...).
>
> Depends-on: patch-82202 ("eventdev: increase MAX QUEUES PER DEV to 255")
>
> Timothy McDaniel (23):
>   event/dlb2: add documentation and meson build infrastructure
>   event/dlb2: add dynamic logging
>   event/dlb2: add private data structures and constants
>   event/dlb2: add definitions shared with LKM or shared code
>   event/dlb2: add inline functions
>   event/dlb2: add eventdev probe
>   event/dlb2: add flexible interface
>   event/dlb2: add probe-time hardware init
>   event/dlb2: add xstats
>   event/dlb2: add infos get and configure
>   event/dlb2: add queue and port default conf
>   event/dlb2: add queue setup
>   event/dlb2: add port setup
>   event/dlb2: add port link
>   event/dlb2: add port unlink and port unlinks in progress
>   event/dlb2: add eventdev start
>   event/dlb2: add enqueue and its burst variants
>   event/dlb2: add dequeue and its burst variants
>   event/dlb2: add eventdev stop and close
>   event/dlb2: add PMD's token pop public interface
>   event/dlb2: add PMD self-tests
>   event/dlb2: add queue and port release
>   event/dlb2: add timeout ticks entry point
>
>  MAINTAINERS                                    |    5 +
>  app/test/test_eventdev.c                       |    7 +
>  config/rte_config.h                            |    7 +
>  doc/api/doxy-api-index.md                      |    3 +-
>  doc/api/doxy-api.conf.in                       |    1 +
>  doc/guides/eventdevs/dlb2.rst                  |  365 ++
>  doc/guides/eventdevs/index.rst                 |    1 +
>  doc/guides/rel_notes/release_20_11.rst         |    5 +
>  drivers/event/dlb2/dlb2.c                      | 3947 ++++++++++++++++
>  drivers/event/dlb2/dlb2_iface.c                |   74 +
>  drivers/event/dlb2/dlb2_iface.h                |   74 +
>  drivers/event/dlb2/dlb2_inline_fns.h           |   33 +
>  drivers/event/dlb2/dlb2_log.h                  |   25 +
>  drivers/event/dlb2/dlb2_priv.h                 |  578 +++
>  drivers/event/dlb2/dlb2_selftest.c             | 1524 ++++++
>  drivers/event/dlb2/dlb2_user.h                 |  679 +++
>  drivers/event/dlb2/dlb2_xstats.c               | 1235 +++++
>  drivers/event/dlb2/meson.build                 |   23 +
>  drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++
>  drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 +++
>  drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +
>  drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 ++
>  drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 +
>  drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
>  drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++
>  drivers/event/dlb2/pf/base/dlb2_resource.c     | 6027 ++++++++++++++++++++++++
>  drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++
>  drivers/event/dlb2/pf/dlb2_main.c              |  673 +++
>  drivers/event/dlb2/pf/dlb2_main.h              |   97 +
>  drivers/event/dlb2/pf/dlb2_pf.c                |  728 +++
>  drivers/event/dlb2/rte_pmd_dlb2.c              |   39 +
>  drivers/event/dlb2/rte_pmd_dlb2.h              |   72 +
>  drivers/event/dlb2/version.map                 |    9 +
>  drivers/event/meson.build                      |    3 +-
>  34 files changed, 22467 insertions(+), 2 deletions(-)
>  create mode 100644 doc/guides/eventdevs/dlb2.rst
>  create mode 100644 drivers/event/dlb2/dlb2.c
>  create mode 100644 drivers/event/dlb2/dlb2_iface.c
>  create mode 100644 drivers/event/dlb2/dlb2_iface.h
>  create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
>  create mode 100644 drivers/event/dlb2/dlb2_log.h
>  create mode 100644 drivers/event/dlb2/dlb2_priv.h
>  create mode 100644 drivers/event/dlb2/dlb2_selftest.c
>  create mode 100644 drivers/event/dlb2/dlb2_user.h
>  create mode 100644 drivers/event/dlb2/dlb2_xstats.c
>  create mode 100644 drivers/event/dlb2/meson.build
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
>  create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
>  create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
>  create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
>  create mode 100644 drivers/event/dlb2/version.map
>
> --
> 2.6.4
>

[dpdk-dev] [PATCH v2 02/22] event/dlb2: add dynamic logging

[Timothy McDaniel]

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_log.h | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_log.h

diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
new file mode 100644
index 0000000..dc1481e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_log.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_EVDEV_LOG_H_
+#define _DLB2_EVDEV_LOG_H_
+
+extern int eventdev_dlb2_log_level;
+
+/* Dynamic logging */
+#define DLB2_LOG_IMPL(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
+		__func__, ##args)
+
+#define DLB2_LOG_INFO(fmt, args...) \
+	DLB2_LOG_IMPL(INFO, fmt, ## args)
+
+#define DLB2_LOG_ERR(fmt, args...) \
+	DLB2_LOG_IMPL(ERR, fmt, ## args)
+
+/* remove debug logs at compile time unless actually debugging */
+#define DLB2_LOG_DBG(fmt, args...) \
+	RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
+
+#endif /* _DLB2_EVDEV_LOG_H_ */
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 02/22] event/dlb2: add dynamic logging

[Jerin Jacob]

On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> This commit adds base support for dynamic logging.
> The default log level is NOTICE. Dynamic logging
> is used exclusively throughout this patchset.
>
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> Reviewed-by: Gage Eads <gage.eads@intel.com>
> ---
>  drivers/event/dlb2/dlb2_log.h | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
>  create mode 100644 drivers/event/dlb2/dlb2_log.h
>
> diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
> new file mode 100644
> index 0000000..dc1481e
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2_log.h
> @@ -0,0 +1,25 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef _DLB2_EVDEV_LOG_H_
> +#define _DLB2_EVDEV_LOG_H_
> +
> +extern int eventdev_dlb2_log_level;

Move rte_log_register() calling part to this patch. Without
registration this patch
is incomplete.


> +/* Dynamic logging */
> +#define DLB2_LOG_IMPL(level, fmt, args...) \
> +       rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
> +               __func__, ##args)
> +
> +#define DLB2_LOG_INFO(fmt, args...) \
> +       DLB2_LOG_IMPL(INFO, fmt, ## args)
> +
> +#define DLB2_LOG_ERR(fmt, args...) \
> +       DLB2_LOG_IMPL(ERR, fmt, ## args)
> +
> +/* remove debug logs at compile time unless actually debugging */
> +#define DLB2_LOG_DBG(fmt, args...) \
> +       RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
> +
> +#endif /* _DLB2_EVDEV_LOG_H_ */
> --
> 2.6.4
>

Re: [dpdk-dev] [PATCH v2 02/22] event/dlb2: add dynamic logging

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Sunday, October 18, 2020 3:57 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <Erik.G.Carrillo@intel.com>;
> Eads, Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH v2 02/22] event/dlb2: add dynamic logging
> 
> On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > This commit adds base support for dynamic logging.
> > The default log level is NOTICE. Dynamic logging
> > is used exclusively throughout this patchset.
> >
> > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > ---
> >  drivers/event/dlb2/dlb2_log.h | 25 +++++++++++++++++++++++++
> >  1 file changed, 25 insertions(+)
> >  create mode 100644 drivers/event/dlb2/dlb2_log.h
> >
> > diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
> > new file mode 100644
> > index 0000000..dc1481e
> > --- /dev/null
> > +++ b/drivers/event/dlb2/dlb2_log.h
> > @@ -0,0 +1,25 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef _DLB2_EVDEV_LOG_H_
> > +#define _DLB2_EVDEV_LOG_H_
> > +
> > +extern int eventdev_dlb2_log_level;
> 
> Move rte_log_register() calling part to this patch. Without
> registration this patch
> is incomplete.
> 

will do

[dpdk-dev] [PATCH v2 03/22] event/dlb2: add private data structures and constants

[Timothy McDaniel]

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_priv.h | 575 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 575 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_priv.h

diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
new file mode 100644
index 0000000..61567a6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -0,0 +1,575 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_PRIV_H_
+#define _DLB2_PRIV_H_
+
+#include <emmintrin.h>
+#include <stdbool.h>
+
+#include <rte_eventdev.h>
+#include <rte_config.h>
+#include "dlb2_user.h"
+#include "dlb2_log.h"
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
+#else
+#define DLB2_INC_STAT(_stat, _incr_val)
+#endif
+
+#define EVDEV_DLB2_NAME_PMD dlb2_event
+
+/*  command line arg strings */
+#define NUMA_NODE_ARG "numa_node"
+#define DLB2_MAX_NUM_EVENTS "max_num_events"
+#define DLB2_NUM_DIR_CREDITS "num_dir_credits"
+#define DEV_ID_ARG "dev_id"
+#define DLB2_DEFER_SCHED_ARG "defer_sched"
+#define DLB2_QID_DEPTH_THRESH_ARG "qid_depth_thresh"
+#define DLB2_COS_ARG "cos"
+
+/* Begin HW related defines and structs */
+
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_VFS 16
+#define DLB2_MAX_NUM_LDB_QUEUES 32
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS 64
+#define DLB2_MAX_NUM_DIR_QUEUES 64
+#define DLB2_MAX_NUM_FLOWS (64 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDITS (8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS (2 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_DIR_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQOS_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_MAX_DEVICE_PATH 32
+#define DLB2_MIN_DEQUEUE_TIMEOUT_NS 1
+/* Note: "- 1" here to support the timeout range check in eventdev_autotest */
+#define DLB2_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1)
+#define DLB2_SW_CREDIT_BATCH_SZ 32
+#define DLB2_NUM_SN_GROUPS 2
+#define DLB2_MAX_LDB_SN_ALLOC 1024
+#define DLB2_MAX_QUEUE_DEPTH_THRESHOLD 8191
+
+/* 2048 total hist list entries and 64 total ldb ports, which
+ * makes for 2048/64 == 32 hist list entries per port. However, CQ
+ * depth must be a power of 2 and must also be >= HIST LIST entries.
+ * As a result we just limit the maximum dequeue depth to 32.
+ */
+#define DLB2_MIN_CQ_DEPTH 1
+#define DLB2_MAX_CQ_DEPTH 32
+#define DLB2_MIN_HARDWARE_CQ_DEPTH 8
+#define DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \
+	DLB2_MAX_CQ_DEPTH
+
+/*
+ * Static per queue/port provisioning values
+ */
+#define DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 64
+
+#define CQ_BASE(is_dir) ((is_dir) ? DLB2_DIR_CQ_BASE : DLB2_LDB_CQ_BASE)
+#define CQ_SIZE(is_dir) ((is_dir) ? DLB2_DIR_CQ_MAX_SIZE : \
+				    DLB2_LDB_CQ_MAX_SIZE)
+#define PP_BASE(is_dir) ((is_dir) ? DLB2_DIR_PP_BASE : DLB2_LDB_PP_BASE)
+
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+
+#define DLB2_NUM_QES_PER_CACHE_LINE 4
+
+#define DLB2_MAX_ENQUEUE_DEPTH 64
+#define DLB2_MIN_ENQUEUE_DEPTH 4
+
+#define DLB2_NAME_SIZE 64
+
+#define DLB2_1K 1024
+#define DLB2_2K (2 * DLB2_1K)
+#define DLB2_4K (4 * DLB2_1K)
+#define DLB2_16K (16 * DLB2_1K)
+#define DLB2_32K (32 * DLB2_1K)
+#define DLB2_1MB (DLB2_1K * DLB2_1K)
+#define DLB2_16MB (16 * DLB2_1MB)
+
+/* Use the upper 3 bits of the event priority to select the DLB2 priority */
+#define EV_TO_DLB2_PRIO(x) ((x) >> 5)
+#define DLB2_TO_EV_PRIO(x) ((x) << 5)
+
+enum dlb2_hw_port_types {
+	DLB2_LDB_PORT,
+	DLB2_DIR_PORT,
+	DLB2_NUM_PORT_TYPES /* Must be last */
+};
+
+enum dlb2_hw_queue_types {
+	DLB2_LDB_QUEUE,
+	DLB2_DIR_QUEUE,
+	DLB2_NUM_QUEUE_TYPES /* Must be last */
+};
+
+#define PORT_TYPE(p) ((p)->is_directed ? DLB2_DIR_PORT : DLB2_LDB_PORT)
+
+/* Do not change - must match hardware! */
+enum dlb2_hw_sched_type {
+	DLB2_SCHED_ATOMIC = 0,
+	DLB2_SCHED_UNORDERED,
+	DLB2_SCHED_ORDERED,
+	DLB2_SCHED_DIRECTED,
+	/* DLB2_NUM_HW_SCHED_TYPES must be last */
+	DLB2_NUM_HW_SCHED_TYPES
+};
+
+struct dlb2_hw_rsrcs {
+	int32_t nb_events_limit;
+	uint32_t num_queues;		/* Total queues (lb + dir) */
+	uint32_t num_ldb_queues;	/* Number of available ldb queues */
+	uint32_t num_ldb_ports;         /* Number of load balanced ports */
+	uint32_t num_dir_ports;         /* Number of directed ports */
+	uint32_t num_ldb_credits;       /* Number of load balanced credits */
+	uint32_t num_dir_credits;       /* Number of directed credits */
+	uint32_t reorder_window_size;   /* Size of reorder window */
+};
+
+struct dlb2_hw_resource_info {
+	/**> Max resources that can be provided */
+	struct dlb2_hw_rsrcs hw_rsrc_max;
+	int num_sched_domains;
+	uint32_t socket_id;
+};
+
+enum dlb2_enqueue_type {
+	/**>
+	 * New : Used to inject a new packet into the QM.
+	 */
+	DLB2_ENQ_NEW,
+	/**>
+	 * Forward : Enqueues a packet, and
+	 *  - if atomic: release any lock it holds in the QM
+	 *  - if ordered: release the packet for egress re-ordering
+	 */
+	DLB2_ENQ_FWD,
+	/**>
+	 * Enqueue Drop : Release an inflight packet. Must be called with
+	 * event == NULL. Used to drop a packet.
+	 *
+	 * Note that all packets dequeued from a load-balanced port must be
+	 * released, either with DLB2_ENQ_DROP or DLB2_ENQ_FWD.
+	 */
+	DLB2_ENQ_DROP,
+
+	/* marker for array sizing etc. */
+	_DLB2_NB_ENQ_TYPES
+};
+
+/* hw-specific format - do not change */
+
+struct dlb2_event_type {
+	uint8_t major:4;
+	uint8_t unused:4;
+	uint8_t sub;
+};
+
+union dlb2_opaque_data {
+	uint16_t opaque_data;
+	struct dlb2_event_type event_type;
+};
+
+struct dlb2_msg_info {
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+};
+
+#define DLB2_NEW_CMD_BYTE 0x08
+#define DLB2_FWD_CMD_BYTE 0x0A
+#define DLB2_COMP_CMD_BYTE 0x02
+#define DLB2_POP_CMD_BYTE 0x01
+#define DLB2_NOOP_CMD_BYTE 0x00
+
+/* hw-specific format - do not change */
+struct dlb2_enqueue_qe {
+	uint64_t data;
+	/* Word 3 */
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	/* Word 4 */
+	uint16_t lock_id;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_cq_pop_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t tokens:10;
+	uint16_t rsvd2:6;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_dequeue_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t flow_id:16; /* was pp_id in v1 */
+	uint8_t debug;
+	uint8_t cq_gen:1;
+	uint8_t qid_depth:2; /* 2 bits in v2 */
+	uint8_t rsvd1:2;
+	uint8_t error:1;
+	uint8_t rsvd2:2;
+};
+
+union dlb2_port_config {
+	struct dlb2_create_ldb_port_args ldb;
+	struct dlb2_create_dir_port_args dir;
+};
+
+enum dlb2_port_state {
+	PORT_CLOSED,
+	PORT_STARTED,
+	PORT_STOPPED
+};
+
+enum dlb2_configuration_state {
+	/* The resource has not been configured */
+	DLB2_NOT_CONFIGURED,
+	/* The resource was configured, but the device was stopped */
+	DLB2_PREV_CONFIGURED,
+	/* The resource is currently configured */
+	DLB2_CONFIGURED
+};
+
+struct dlb2_port {
+	uint32_t id;
+	bool is_directed;
+	bool gen_bit;
+	uint16_t dir_credits;
+	uint32_t dequeue_depth;
+	union dlb2_port_config cfg;
+	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
+	uint16_t cached_ldb_credits;
+	uint16_t ldb_credits;
+	uint16_t cached_dir_credits;
+	bool int_armed;
+	uint16_t owed_tokens;
+	int16_t issued_releases;
+	int cq_depth;
+	uint16_t cq_idx;
+	uint16_t cq_idx_unmasked;
+	uint16_t cq_depth_mask;
+	uint16_t gen_bit_shift;
+	enum dlb2_port_state state;
+	enum dlb2_configuration_state config_state;
+	int num_mapped_qids;
+	uint8_t *qid_mappings;
+	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
+	struct dlb2_enqueue_qe *int_arm_qe;
+	struct dlb2_cq_pop_qe *consume_qe;
+	struct dlb2_eventdev *dlb2; /* back ptr */
+	struct dlb2_eventdev_port *ev_port; /* back ptr */
+};
+
+/* Per-process per-port mmio and memory pointers */
+struct process_local_port_data {
+	uint64_t *pp_addr;
+	struct dlb2_dequeue_qe *cq_base;
+	const struct rte_memzone *mz;
+	bool mmaped;
+};
+
+struct dlb2_eventdev;
+
+struct dlb2_config {
+	int configured;
+	int reserved;
+	uint32_t num_ldb_credits;
+	uint32_t num_dir_credits;
+	struct dlb2_create_sched_domain_args resources;
+};
+
+enum dlb2_cos {
+	DLB2_COS_DEFAULT = -1,
+	DLB2_COS_0 = 0,
+	DLB2_COS_1,
+	DLB2_COS_2,
+	DLB2_COS_3
+};
+
+struct dlb2_hw_dev {
+	struct dlb2_config cfg;
+	struct dlb2_hw_resource_info info;
+	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
+	uint32_t domain_id;
+	enum dlb2_cos cos_id;
+	rte_spinlock_t resource_lock; /* for MP support */
+} __rte_cache_aligned;
+
+/* End HW related defines and structs */
+
+/* Begin DLB2 PMD Eventdev related defines and structs */
+
+#define DLB2_MAX_NUM_QUEUES \
+	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
+
+#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS + DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
+
+/** Structure to hold the queue to port link establishment attributes */
+
+struct dlb2_event_queue_link {
+	uint8_t queue_id;
+	uint8_t priority;
+	bool mapped;
+	bool valid;
+};
+
+struct dlb2_traffic_stats {
+	uint64_t rx_ok;
+	uint64_t rx_drop;
+	uint64_t rx_interrupt_wait;
+	uint64_t rx_umonitor_umwait;
+	uint64_t tx_ok;
+	uint64_t total_polls;
+	uint64_t zero_polls;
+	uint64_t tx_nospc_ldb_hw_credits;
+	uint64_t tx_nospc_dir_hw_credits;
+	uint64_t tx_nospc_inflight_max;
+	uint64_t tx_nospc_new_event_limit;
+	uint64_t tx_nospc_inflight_credits;
+};
+
+/* DLB2 HW sets the 2bit qid_depth in rx QEs based on the programmable depth
+ * threshold. The global default value in config/common_base (or rte_config.h)
+ * can be overridden on a per-qid basis using a vdev command line parameter.
+ * 3: depth > threshold
+ * 2: threshold >= depth > 3/4 threshold
+ * 1: 3/4 threshold >= depth > 1/2 threshold
+ * 0: depth <= 1/2 threshold.
+ */
+#define DLB2_QID_DEPTH_LE50 0
+#define DLB2_QID_DEPTH_GT50_LE75 1
+#define DLB2_QID_DEPTH_GT75_LE100 2
+#define DLB2_QID_DEPTH_GT100 3
+#define DLB2_NUM_QID_DEPTH_STAT_VALS 4 /* 2 bits */
+
+struct dlb2_queue_stats {
+	uint64_t enq_ok;
+	uint64_t qid_depth[DLB2_NUM_QID_DEPTH_STAT_VALS];
+};
+
+struct dlb2_port_stats {
+	struct dlb2_traffic_stats traffic;
+	uint64_t tx_op_cnt[4]; /* indexed by rte_event.op */
+	uint64_t tx_implicit_rel;
+	uint64_t tx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t tx_invalid;
+	uint64_t rx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t rx_sched_invalid;
+	struct dlb2_queue_stats queue[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_eventdev_port {
+	struct dlb2_port qm_port; /* hw specific data structure */
+	struct rte_event_port_conf conf; /* user-supplied configuration */
+	uint16_t inflight_credits; /* num credits this port has right now */
+	uint16_t credit_update_quanta;
+	struct dlb2_eventdev *dlb2; /* backlink optimization */
+	struct dlb2_port_stats stats __rte_cache_aligned;
+	struct dlb2_event_queue_link link[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	int num_links;
+	uint32_t id; /* port id */
+	/* num releases yet to be completed on this port.
+	 * Only applies to load-balanced ports.
+	 */
+	uint16_t outstanding_releases;
+	uint16_t inflight_max; /* app requested max inflights for this port */
+	/* setup_done is set when the event port is setup */
+	bool setup_done;
+	/* enq_configured is set when the qm port is created */
+	bool enq_configured;
+	uint8_t implicit_release; /* release events before dequeueing */
+}  __rte_cache_aligned;
+
+struct dlb2_queue {
+	uint32_t num_qid_inflights; /* User config */
+	uint32_t num_atm_inflights; /* User config */
+	enum dlb2_configuration_state config_state;
+	int sched_type; /* LB queue only */
+	uint32_t id;
+	bool is_directed;
+};
+
+struct dlb2_eventdev_queue {
+	struct dlb2_queue qm_queue;
+	struct rte_event_queue_conf conf; /* User config */
+	int depth_threshold; /* use default if 0 */
+	uint32_t id;
+	bool setup_done;
+	uint8_t num_links;
+};
+
+enum dlb2_run_state {
+	DLB2_RUN_STATE_STOPPED = 0,
+	DLB2_RUN_STATE_STOPPING,
+	DLB2_RUN_STATE_STARTING,
+	DLB2_RUN_STATE_STARTED
+};
+
+struct dlb2_eventdev {
+	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
+	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each queue */
+	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
+	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each port */
+	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
+	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
+	struct dlb2_get_num_resources_args hw_rsrc_query_results;
+	uint32_t xstats_count_mode_queue;
+	struct dlb2_hw_dev qm_instance; /* strictly hw related */
+	uint64_t global_dequeue_wait_ticks;
+	struct dlb2_xstats_entry *xstats;
+	struct rte_eventdev *event_dev; /* backlink to dev */
+	uint32_t xstats_count_mode_dev;
+	uint32_t xstats_count_mode_port;
+	uint32_t xstats_count;
+	uint32_t inflights; /* use __atomic builtins */
+	uint32_t new_event_limit;
+	int max_num_events_override;
+	int num_dir_credits_override;
+	volatile enum dlb2_run_state run_state;
+	uint16_t num_dir_queues; /* total num of evdev dir queues requested */
+	uint16_t num_dir_credits;
+	uint16_t num_ldb_credits;
+	uint16_t num_queues; /* total queues */
+	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested */
+	uint16_t num_ports; /* total num of evdev ports requested */
+	uint16_t num_ldb_ports; /* total num of ldb ports requested */
+	uint16_t num_dir_ports; /* total num of dir ports requested */
+	bool umwait_allowed;
+	bool global_dequeue_wait; /* Not using per dequeue wait if true */
+	bool defer_sched;
+	enum dlb2_cq_poll_modes poll_mode;
+	uint8_t revision;
+	bool configured;
+	uint16_t max_ldb_credits;
+	uint16_t max_dir_credits;
+
+	/* force hw credit pool counters into exclusive cache lines */
+
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t ldb_credit_pool __rte_cache_aligned;
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t dir_credit_pool __rte_cache_aligned;
+};
+
+/* used for collecting and passing around the dev args */
+struct dlb2_qid_depth_thresholds {
+	int val[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_devargs {
+	int socket_id;
+	int max_num_events;
+	int num_dir_credits_override;
+	int dev_id;
+	int defer_sched;
+	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
+	enum dlb2_cos cos_id;
+};
+
+/* End Eventdev related defines and structs */
+
+/* Forwards for non-inlined functions */
+
+void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
+
+int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
+
+void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
+
+int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n);
+
+int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstat_names,
+		unsigned int *ids, unsigned int size);
+
+uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+					  const char *name, unsigned int *id);
+
+int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode,
+		int16_t queue_port_id,
+		const uint32_t ids[],
+		uint32_t nb_ids);
+
+int test_dlb2_eventdev(void);
+
+int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+				const char *name,
+				struct dlb2_devargs *dlb2_args);
+
+int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+				  const char *name);
+
+uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *queue);
+
+int dlb2_parse_params(const char *params,
+		      const char *name,
+		      struct dlb2_devargs *dlb2_args);
+
+/* Extern globals */
+extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
+
+#endif	/* _DLB2_PRIV_H_ */
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 03/22] event/dlb2: add private data structures and constants

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Saturday, October 17, 2020 1:21 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH v2 03/22] event/dlb2: add private data structures and constants
> 
> The header file dlb2_priv.h is used internally by the PMD.
> It include constants, macros for device resources,
> structure definitions for hardware interfaces and
> software state, and various forward-declarations.
> The header file rte_pmd_dlb2.h will be exported in a
> subsequent patch, but is included here due to a data
> structure dependency.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v2 04/22] event/dlb2: add definitions shared with LKM or shared code

[Timothy McDaniel]

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_user.h | 679 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 679 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_user.h

diff --git a/drivers/event/dlb2/dlb2_user.h b/drivers/event/dlb2/dlb2_user.h
new file mode 100644
index 0000000..b5d39be
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_user.h
@@ -0,0 +1,679 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#define DLB2_MAX_NAME_LEN 64
+
+#include <linux/types.h>
+
+enum dlb2_error {
+	DLB2_ST_SUCCESS = 0,
+	DLB2_ST_NAME_EXISTS,
+	DLB2_ST_DOMAIN_UNAVAILABLE,
+	DLB2_ST_LDB_PORTS_UNAVAILABLE,
+	DLB2_ST_DIR_PORTS_UNAVAILABLE,
+	DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB2_ST_INVALID_DOMAIN_ID,
+	DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB2_ST_INVALID_LDB_QUEUE_ID,
+	DLB2_ST_INVALID_CQ_DEPTH,
+	DLB2_ST_INVALID_CQ_VIRT_ADDR,
+	DLB2_ST_INVALID_PORT_ID,
+	DLB2_ST_INVALID_QID,
+	DLB2_ST_INVALID_PRIORITY,
+	DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+	DLB2_ST_INVALID_DIR_QUEUE_ID,
+	DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB2_ST_DOMAIN_NOT_CONFIGURED,
+	DLB2_ST_INTERNAL_ERROR,
+	DLB2_ST_DOMAIN_IN_USE,
+	DLB2_ST_DOMAIN_NOT_FOUND,
+	DLB2_ST_QUEUE_NOT_FOUND,
+	DLB2_ST_DOMAIN_STARTED,
+	DLB2_ST_DOMAIN_NOT_STARTED,
+	DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB2_ST_DOMAIN_RESET_FAILED,
+	DLB2_ST_MBOX_ERROR,
+	DLB2_ST_INVALID_HIST_LIST_DEPTH,
+	DLB2_ST_NO_MEMORY,
+	DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB2_ST_INVALID_COS_ID,
+};
+
+static const char dlb2_error_strings[][128] = {
+	"DLB2_ST_SUCCESS",
+	"DLB2_ST_NAME_EXISTS",
+	"DLB2_ST_DOMAIN_UNAVAILABLE",
+	"DLB2_ST_LDB_PORTS_UNAVAILABLE",
+	"DLB2_ST_DIR_PORTS_UNAVAILABLE",
+	"DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+	"DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+	"DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+	"DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+	"DLB2_ST_INVALID_DOMAIN_ID",
+	"DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+	"DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+	"DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+	"DLB2_ST_INVALID_LDB_QUEUE_ID",
+	"DLB2_ST_INVALID_CQ_DEPTH",
+	"DLB2_ST_INVALID_CQ_VIRT_ADDR",
+	"DLB2_ST_INVALID_PORT_ID",
+	"DLB2_ST_INVALID_QID",
+	"DLB2_ST_INVALID_PRIORITY",
+	"DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+	"DLB2_ST_INVALID_DIR_QUEUE_ID",
+	"DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+	"DLB2_ST_DOMAIN_NOT_CONFIGURED",
+	"DLB2_ST_INTERNAL_ERROR",
+	"DLB2_ST_DOMAIN_IN_USE",
+	"DLB2_ST_DOMAIN_NOT_FOUND",
+	"DLB2_ST_QUEUE_NOT_FOUND",
+	"DLB2_ST_DOMAIN_STARTED",
+	"DLB2_ST_DOMAIN_NOT_STARTED",
+	"DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+	"DLB2_ST_DOMAIN_RESET_FAILED",
+	"DLB2_ST_MBOX_ERROR",
+	"DLB2_ST_INVALID_HIST_LIST_DEPTH",
+	"DLB2_ST_NO_MEMORY",
+	"DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+	"DLB2_ST_INVALID_COS_ID",
+};
+
+struct dlb2_cmd_response {
+	__u32 status; /* Interpret using enum dlb2_error */
+	__u32 id;
+};
+
+/*******************/
+/* 'dlb2' commands */
+/*******************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+	DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	This ioctl interface is the same in all driver versions and is always
+ *	the first ioctl.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	from any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+	/* Output parameters */
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ *	load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ *		percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ *	load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ *   ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+enum dlb2_cq_poll_modes {
+	DLB2_CQ_POLL_MODE_STD,
+	DLB2_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB2_CQ_POLL_MODE must be last */
+	NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/********************************/
+/* 'scheduling domain' commands */
+/********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_start_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
+#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
+
+
+#endif /* __DLB2_USER_H */
-- 
2.6.4

[dpdk-dev] [PATCH v2 05/22] event/dlb2: add inline functions

[Timothy McDaniel]

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_inline_fns.h | 81 ++++++++++++++++++++++++++++++++++++
 1 file changed, 81 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h

diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
new file mode 100644
index 0000000..9c3c36f
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_inline_fns.h
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_INLINE_FNS_H_
+#define _DLB2_INLINE_FNS_H_
+
+/* Inline functions required in more than one source file. */
+
+static inline struct dlb2_eventdev *
+dlb2_pmd_priv(const struct rte_eventdev *eventdev)
+{
+	return eventdev->data->dev_private;
+}
+
+static inline void
+dlb2_umonitor(volatile void *addr)
+{
+	asm volatile(".byte 0xf3, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (addr));
+}
+
+static inline void
+dlb2_umwait(int state, uint64_t timeout)
+{
+	uint32_t eax = timeout & UINT32_MAX;
+	uint32_t edx = timeout >> 32;
+
+	asm volatile(".byte 0xf2, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (state),  "a" (eax), "d" (edx));
+}
+
+static inline void
+dlb2_movntdq(void *qe4, void *pp_addr)
+{
+	/* Move entire 64B cache line of QEs, 128 bits (16B) at a time. */
+	long long *_qe  = (long long *)qe4;
+	__v2di src_data0 = (__v2di){_qe[0], _qe[1]};
+	__v2di src_data1 = (__v2di){_qe[2], _qe[3]};
+	__v2di src_data2 = (__v2di){_qe[4], _qe[5]};
+	__v2di src_data3 = (__v2di){_qe[6], _qe[7]};
+
+	__builtin_ia32_movntdq((__v2di *)pp_addr + 0, (__v2di)src_data0);
+	rte_wmb();
+	__builtin_ia32_movntdq((__v2di *)pp_addr + 1, (__v2di)src_data1);
+	rte_wmb();
+	__builtin_ia32_movntdq((__v2di *)pp_addr + 2, (__v2di)src_data2);
+	rte_wmb();
+	__builtin_ia32_movntdq((__v2di *)pp_addr + 3, (__v2di)src_data3);
+	rte_wmb();
+}
+
+static inline void
+dlb2_movntdq_single(void *qe4, void *pp_addr)
+{
+	long long *_qe  = (long long *)qe4;
+	__v2di src_data0 = (__v2di){_qe[0], _qe[1]};
+
+	__builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
+}
+
+static inline void
+dlb2_cldemote(void *addr)
+{
+	/* Load addr into RSI, then demote the cache line of the address
+	 * contained in that register.
+	 */
+	asm volatile(".byte 0x0f, 0x1c, 0x06" :: "S" (addr));
+}
+
+static inline void
+dlb2_movdir64b(void *qe4, void *pp_addr)
+{
+	asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
+		     :
+		     : "a" (pp_addr), "d" (qe4));
+}
+
+#endif /* _DLB2_INLINE_FNS_H_ */
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 05/22] event/dlb2: add inline functions

[Jerin Jacob]

On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> Add miscellaneous inline functions that may be called
> from multiple files.  These functions include inline
> assembly of new x86 instructions, such as movdir64b,
> since they are not available as builtin functions in
> the minimum supported GCC version.
>
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> Reviewed-by: Gage Eads <gage.eads@intel.com>
> ---
>  drivers/event/dlb2/dlb2_inline_fns.h | 81 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 81 insertions(+)
>  create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
>
> diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
> new file mode 100644
> index 0000000..9c3c36f
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2_inline_fns.h
> @@ -0,0 +1,81 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef _DLB2_INLINE_FNS_H_
> +#define _DLB2_INLINE_FNS_H_
> +
> +/* Inline functions required in more than one source file. */
> +
> +static inline struct dlb2_eventdev *
> +dlb2_pmd_priv(const struct rte_eventdev *eventdev)
> +{
> +       return eventdev->data->dev_private;
> +}
> +
> +static inline void
> +dlb2_umonitor(volatile void *addr)
> +{
> +       asm volatile(".byte 0xf3, 0x0f, 0xae, 0xf7\t\n"
> +                       :
> +                       : "D" (addr));
> +}
> +
> +static inline void
> +dlb2_umwait(int state, uint64_t timeout)
> +{
> +       uint32_t eax = timeout & UINT32_MAX;
> +       uint32_t edx = timeout >> 32;
> +
> +       asm volatile(".byte 0xf2, 0x0f, 0xae, 0xf7\t\n"
> +                       :
> +                       : "D" (state),  "a" (eax), "d" (edx));
> +}
> +

Please change this instruction to use new rte public API once it gets merged.

> +static inline void
> +dlb2_movntdq(void *qe4, void *pp_addr)
> +{
> +       /* Move entire 64B cache line of QEs, 128 bits (16B) at a time. */
> +       long long *_qe  = (long long *)qe4;
> +       __v2di src_data0 = (__v2di){_qe[0], _qe[1]};
> +       __v2di src_data1 = (__v2di){_qe[2], _qe[3]};
> +       __v2di src_data2 = (__v2di){_qe[4], _qe[5]};
> +       __v2di src_data3 = (__v2di){_qe[6], _qe[7]};
> +
> +       __builtin_ia32_movntdq((__v2di *)pp_addr + 0, (__v2di)src_data0);
> +       rte_wmb();
> +       __builtin_ia32_movntdq((__v2di *)pp_addr + 1, (__v2di)src_data1);
> +       rte_wmb();
> +       __builtin_ia32_movntdq((__v2di *)pp_addr + 2, (__v2di)src_data2);
> +       rte_wmb();
> +       __builtin_ia32_movntdq((__v2di *)pp_addr + 3, (__v2di)src_data3);
> +       rte_wmb();
> +}
> +
> +static inline void
> +dlb2_movntdq_single(void *qe4, void *pp_addr)
> +{
> +       long long *_qe  = (long long *)qe4;
> +       __v2di src_data0 = (__v2di){_qe[0], _qe[1]};
> +
> +       __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
> +}
> +
> +static inline void
> +dlb2_cldemote(void *addr)
> +{
> +       /* Load addr into RSI, then demote the cache line of the address
> +        * contained in that register.
> +        */
> +       asm volatile(".byte 0x0f, 0x1c, 0x06" :: "S" (addr));
> +}
> +
> +static inline void
> +dlb2_movdir64b(void *qe4, void *pp_addr)
> +{
> +       asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
> +                    :
> +                    : "a" (pp_addr), "d" (qe4));
> +}
> +
> +#endif /* _DLB2_INLINE_FNS_H_ */
> --
> 2.6.4
>

Re: [dpdk-dev] [PATCH v2 05/22] event/dlb2: add inline functions

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Sunday, October 18, 2020 4:00 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <Erik.G.Carrillo@intel.com>;
> Eads, Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH v2 05/22] event/dlb2: add inline functions
> 
> On Sat, Oct 17, 2020 at 11:50 PM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > Add miscellaneous inline functions that may be called
> > from multiple files.  These functions include inline
> > assembly of new x86 instructions, such as movdir64b,
> > since they are not available as builtin functions in
> > the minimum supported GCC version.
> >
> > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > Reviewed-by: Gage Eads <gage.eads@intel.com>
> > ---
> >  drivers/event/dlb2/dlb2_inline_fns.h | 81
> ++++++++++++++++++++++++++++++++++++
> >  1 file changed, 81 insertions(+)
> >  create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
> >
> > diff --git a/drivers/event/dlb2/dlb2_inline_fns.h
> b/drivers/event/dlb2/dlb2_inline_fns.h
> > new file mode 100644
> > index 0000000..9c3c36f
> > --- /dev/null
> > +++ b/drivers/event/dlb2/dlb2_inline_fns.h
> > @@ -0,0 +1,81 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef _DLB2_INLINE_FNS_H_
> > +#define _DLB2_INLINE_FNS_H_
> > +
> > +/* Inline functions required in more than one source file. */
> > +
> > +static inline struct dlb2_eventdev *
> > +dlb2_pmd_priv(const struct rte_eventdev *eventdev)
> > +{
> > +       return eventdev->data->dev_private;
> > +}
> > +
> > +static inline void
> > +dlb2_umonitor(volatile void *addr)
> > +{
> > +       asm volatile(".byte 0xf3, 0x0f, 0xae, 0xf7\t\n"
> > +                       :
> > +                       : "D" (addr));
> > +}
> > +
> > +static inline void
> > +dlb2_umwait(int state, uint64_t timeout)
> > +{
> > +       uint32_t eax = timeout & UINT32_MAX;
> > +       uint32_t edx = timeout >> 32;
> > +
> > +       asm volatile(".byte 0xf2, 0x0f, 0xae, 0xf7\t\n"
> > +                       :
> > +                       : "D" (state),  "a" (eax), "d" (edx));
> > +}
> > +
> 
> Please change this instruction to use new rte public API once it gets merged.
> 

Yes, we will do that.

[dpdk-dev] [PATCH v2 06/22] event/dlb2: add probe

[Timothy McDaniel]

The DLB2 hardware is a PCI device. This commit adds
support for probe and other initialization. The
dlb2_iface.[ch] files implement a flexible interface
that supports both the PF PMD and the bifurcated PMD.
The bifurcated PMD will be released in a future
patch set. Note that the flexible interface is only
used for configuration, and is not used in the data
path. The shared code is added in pf/base.
This PMD supports command line parameters, and those
are parsed at probe-time.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                      |  532 +++++
 drivers/event/dlb2/dlb2_iface.c                |   28 +
 drivers/event/dlb2/dlb2_iface.h                |   29 +
 drivers/event/dlb2/meson.build                 |    6 +-
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  247 +++
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c     |  274 +++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  615 ++++++
 drivers/event/dlb2/pf/dlb2_main.h              |  106 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  244 +++
 16 files changed, 8085 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
new file mode 100644
index 0000000..26985b9
--- /dev/null
+++ b/drivers/event/dlb2/dlb2.c
@@ -0,0 +1,532 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <assert.h>
+#include <errno.h>
+#include <nmmintrin.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_cycles.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_io.h>
+#include <rte_kvargs.h>
+#include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_ring.h>
+#include <rte_string_fns.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_iface.h"
+#include "dlb2_inline_fns.h"
+
+#if !defined RTE_ARCH_X86_64
+#error "This implementation only supports RTE_ARCH_X86_64 architecture."
+#endif
+
+/*
+ * Resources exposed to eventdev. Some values overridden at runtime using
+ * values returned by the DLB kernel driver.
+ */
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
+#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
+#endif
+static struct rte_event_dev_info evdev_dlb2_default_info = {
+	.driver_name = "", /* probe will set */
+	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
+	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
+	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
+#else
+	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
+#endif
+	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
+	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
+	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
+	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
+	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
+	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+	.max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
+	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
+			  RTE_EVENT_DEV_CAP_EVENT_QOS |
+			  RTE_EVENT_DEV_CAP_BURST_MODE |
+			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
+			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
+			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
+};
+
+struct process_local_port_data
+dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
+
+/* override defaults with value(s) provided on command line */
+static void
+dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
+				 int *qid_depth_thresholds)
+{
+	int q;
+
+	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
+		if (qid_depth_thresholds[q] != 0)
+			dlb2->ev_queues[q].depth_threshold =
+				qid_depth_thresholds[q];
+	}
+}
+
+static int
+dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
+	int ret;
+
+	/* Query driver resources provisioned for this device */
+
+	ret = dlb2_iface_get_num_resources(handle,
+					   &dlb2->hw_rsrc_query_results);
+	if (ret) {
+		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
+		return ret;
+	}
+
+	/* Complete filling in device resource info returned to evdev app,
+	 * overriding any default values.
+	 * The capabilities (CAPs) were set at compile time.
+	 */
+
+	evdev_dlb2_default_info.max_event_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	evdev_dlb2_default_info.max_event_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	evdev_dlb2_default_info.max_num_events =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	/* Save off values used when creating the scheduling domain. */
+
+	handle->info.num_sched_domains =
+		dlb2->hw_rsrc_query_results.num_sched_domains;
+
+	handle->info.hw_rsrc_max.nb_events_limit =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	handle->info.hw_rsrc_max.num_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues +
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.num_ldb_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	handle->info.hw_rsrc_max.num_ldb_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	handle->info.hw_rsrc_max.num_dir_ports =
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.reorder_window_size =
+		dlb2->hw_rsrc_query_results.num_hist_list_entries;
+
+	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
+
+	return 0;
+}
+
+#define RTE_BASE_10 10
+
+static int
+dlb2_string_to_int(int *result, const char *str)
+{
+	long ret;
+	char *endptr;
+
+	if (str == NULL || result == NULL)
+		return -EINVAL;
+
+	errno = 0;
+	ret = strtol(str, &endptr, RTE_BASE_10);
+	if (errno)
+		return -errno;
+
+	/* long int and int may be different width for some architectures */
+	if (ret < INT_MIN || ret > INT_MAX || endptr == str)
+		return -EINVAL;
+
+	*result = ret;
+	return 0;
+}
+
+static int
+set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
+{
+	int *socket_id = opaque;
+	int ret;
+
+	ret = dlb2_string_to_int(socket_id, value);
+	if (ret < 0)
+		return ret;
+
+	if (*socket_id > RTE_MAX_NUMA_NODES)
+		return -EINVAL;
+	return 0;
+}
+
+static int
+set_max_num_events(const char *key __rte_unused,
+		   const char *value,
+		   void *opaque)
+{
+	int *max_num_events = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(max_num_events, value);
+	if (ret < 0)
+		return ret;
+
+	if (*max_num_events < 0 || *max_num_events >
+			DLB2_MAX_NUM_LDB_CREDITS) {
+		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
+			     DLB2_MAX_NUM_LDB_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_num_dir_credits(const char *key __rte_unused,
+		    const char *value,
+		    void *opaque)
+{
+	int *num_dir_credits = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(num_dir_credits, value);
+	if (ret < 0)
+		return ret;
+
+	if (*num_dir_credits < 0 ||
+	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
+		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
+			     DLB2_MAX_NUM_DIR_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_dev_id(const char *key __rte_unused,
+	   const char *value,
+	   void *opaque)
+{
+	int *dev_id = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(dev_id, value);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+set_cos(const char *key __rte_unused,
+	const char *value,
+	void *opaque)
+{
+	enum dlb2_cos *cos_id = opaque;
+	int x = 0;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(&x, value);
+	if (ret < 0)
+		return ret;
+
+	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
+		DLB2_LOG_ERR(
+			"COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
+			x);
+		return -EINVAL;
+	}
+
+	*cos_id = x;
+
+	return 0;
+}
+
+
+static int
+set_qid_depth_thresh(const char *key __rte_unused,
+		     const char *value,
+		     void *opaque)
+{
+	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
+	int first, last, thresh, i;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	/* command line override may take one of the following 3 forms:
+	 * qid_depth_thresh=all:<threshold_value> ... all queues
+	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
+	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
+	 */
+	if (sscanf(value, "all:%d", &thresh) == 1) {
+		first = 0;
+		last = DLB2_MAX_NUM_QUEUES - 1;
+	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
+		/* we have everything we need */
+	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
+		last = first;
+	} else {
+		DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val, qid-qid:val, or qid:val\n");
+		return -EINVAL;
+	}
+
+	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
+		return -EINVAL;
+	}
+
+	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
+			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
+		return -EINVAL;
+	}
+
+	for (i = first; i <= last; i++)
+		qid_thresh->val[i] = thresh; /* indexed by qid */
+
+	return 0;
+}
+
+static void
+dlb2_entry_points_init(struct rte_eventdev *dev)
+{
+	RTE_SET_USED(dev);
+
+	/* Eventdev PMD entry points */
+}
+
+int
+dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+			    const char *name,
+			    struct dlb2_devargs *dlb2_args)
+{
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	dlb2->event_dev = dev; /* backlink */
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	dlb2->max_num_events_override = dlb2_args->max_num_events;
+	dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
+	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
+					    &dlb2->revision);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_hardware_init(&dlb2->qm_instance);
+
+	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
+			     err);
+		return err;
+	}
+
+	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	dlb2_init_queue_depth_thresholds(dlb2,
+					 dlb2_args->qid_depth_thresholds.val);
+
+	return 0;
+}
+
+int
+dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+			      const char *name)
+{
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	return 0;
+}
+
+int
+dlb2_parse_params(const char *params,
+		  const char *name,
+		  struct dlb2_devargs *dlb2_args)
+{
+	int ret = 0;
+	static const char * const args[] = { NUMA_NODE_ARG,
+					     DLB2_MAX_NUM_EVENTS,
+					     DLB2_NUM_DIR_CREDITS,
+					     DEV_ID_ARG,
+					     DLB2_QID_DEPTH_THRESH_ARG,
+					     DLB2_COS_ARG,
+					     NULL };
+
+	if (params != NULL && params[0] != '\0') {
+		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
+
+		if (kvlist == NULL) {
+			RTE_LOG(INFO, PMD,
+				"Ignoring unsupported parameters when creating device '%s'\n",
+				name);
+		} else {
+			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
+						     set_numa_node,
+						     &dlb2_args->socket_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing numa node parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
+						 set_max_num_events,
+						 &dlb2_args->max_num_events);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist,
+					DLB2_NUM_DIR_CREDITS,
+					set_num_dir_credits,
+					&dlb2_args->num_dir_credits_override);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
+						 set_dev_id,
+						 &dlb2_args->dev_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(
+					kvlist,
+					DLB2_QID_DEPTH_THRESH_ARG,
+					set_qid_depth_thresh,
+					&dlb2_args->qid_depth_thresholds);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
+						 set_cos,
+						 &dlb2_args->cos_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing cos parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			rte_kvargs_free(kvlist);
+		}
+	}
+	return ret;
+}
+RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
new file mode 100644
index 0000000..0d93faf
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+
+#include "dlb2_priv.h"
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+void (*dlb2_iface_low_level_io_init)(void);
+
+int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+				     uint8_t *revision);
+
+void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+				   enum dlb2_cq_poll_modes *mode);
+
+int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_num_resources_args *rsrcs);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
new file mode 100644
index 0000000..4fb416e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_IFACE_H_
+#define _DLB2_IFACE_H_
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+extern void (*dlb2_iface_low_level_io_init)(void);
+
+extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+					    uint8_t *revision);
+
+extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+					  enum dlb2_cq_poll_modes *mode);
+
+extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
+
+#endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 54ba2c8..99b71f9 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,7 +1,11 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
-sources = files(
+sources = files('dlb2.c',
+		'dlb2_iface.c',
+		'pf/dlb2_main.c',
+		'pf/dlb2_pf.c',
+		'pf/base/dlb2_resource.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
new file mode 100644
index 0000000..428a5e8
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_osdep_list.h"
+#include "dlb2_osdep_types.h"
+
+#define DLB2_MAX_NUM_VDEVS			16
+#define DLB2_MAX_NUM_DOMAINS			32
+#define DLB2_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS			64
+#define DLB2_MAX_NUM_DIR_PORTS			64
+#define DLB2_MAX_NUM_LDB_CREDITS		(8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS		(2 * 1024)
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB2_MAX_NUM_AQED_ENTRIES		2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB2_QID_PRIORITIES			8
+#define DLB2_NUM_ARB_WEIGHTS			8
+#define DLB2_MAX_WEIGHT				255
+#define DLB2_NUM_COS_DOMAINS			4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#ifdef FPGA
+#define DLB2_HZ					2000000
+#else
+#define DLB2_HZ					800000000
+#endif
+
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
+
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+	DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+	DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID		3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID		4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW		5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ	1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
+#define DLB2_VF_BASE_CQ_VECTOR_ID	     0
+#define DLB2_VF_LAST_CQ_VECTOR_ID	     30
+#define DLB2_VF_MBOX_VECTOR_ID		     31
+#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
+
+#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
+					     DLB2_MAX_NUM_DIR_PORTS + 1)
+
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE   0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
+				DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE   0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
+				DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE       0x2100000
+#define DLB2_LDB_PP_STRIDE     0x1000
+#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
+				DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE       0x2000000
+#define DLB2_DIR_PP_STRIDE     0x1000
+#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
+				DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
+struct dlb2_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb2_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb2_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
+struct dlb2_ldb_queue {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB2_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB2_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB2_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB2_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+	enum dlb2_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb2_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+	u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+	u32 bound[6] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & (1 << i))) {
+			group->slot_use_bitmap |= 1 << i;
+			return i;
+		}
+	}
+
+	return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+	int i, cnt = 0;
+
+	for (i = 0; i < 32; i++)
+		cnt += !!(group->slot_use_bitmap & (1 << i));
+
+	return cnt;
+}
+
+struct dlb2_hw_domain {
+	struct dlb2_function_resources *parent_func;
+	struct dlb2_list_entry func_list;
+	struct dlb2_list_head used_ldb_queues;
+	struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head used_dir_pq_pairs;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb2_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb2_bitmap;
+
+struct dlb2_function_resources {
+	struct dlb2_list_head avail_domains;
+	struct dlb2_list_head used_domains;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	struct dlb2_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+	struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+	struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+	struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
+struct dlb2_mbox {
+	u32 *mbox;
+	u32 *isr_in_progress;
+};
+
+struct dlb2_sw_mbox {
+	struct dlb2_mbox vdev_to_pf;
+	struct dlb2_mbox pf_to_vdev;
+	void (*pf_to_vdev_inject)(void *arg);
+	void *pf_to_vdev_inject_arg;
+};
+
+struct dlb2_hw {
+	/* BAR 0 address */
+	void  *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void  *func_kva;
+	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb2_hw_resources rsrcs;
+	struct dlb2_function_resources pf;
+	struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+	struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+	/* Virtualization */
+	int virt_mode;
+	struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
+	unsigned int pasid[DLB2_MAX_NUM_VDEVS];
+};
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
new file mode 100644
index 0000000..ce462c0
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
@@ -0,0 +1,596 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_BASE_DLB2_MBOX_H
+#define __DLB2_BASE_DLB2_MBOX_H
+
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+
+#define DLB2_MBOX_INTERFACE_VERSION 1
+
+/*
+ * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
+ * to send requests of its own (e.g. notifying a VF of an impending FLR).
+ * To avoid communication race conditions, e.g. the PF sends a response and then
+ * sends a request before the VF reads the response, the PF->VF mailbox is
+ * divided into two sections:
+ * - Bytes 0-47: PF responses
+ * - Bytes 48-63: PF requests
+ *
+ * Partitioning the PF->VF mailbox allows responses and requests to occupy the
+ * mailbox simultaneously.
+ */
+#define DLB2_PF2VF_RESP_BYTES	  48
+#define DLB2_PF2VF_RESP_BASE	  0
+#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
+
+#define DLB2_PF2VF_REQ_BYTES	  16
+#define DLB2_PF2VF_REQ_BASE	  (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
+#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
+
+/*
+ * Similarly, the VF->PF mailbox is divided into two sections:
+ * - Bytes 0-239: VF requests
+ * -- (Bytes 0-3 are unused due to a hardware errata)
+ * - Bytes 240-255: VF responses
+ */
+#define DLB2_VF2PF_REQ_BYTES	 236
+#define DLB2_VF2PF_REQ_BASE	 4
+#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
+
+#define DLB2_VF2PF_RESP_BYTES	  16
+#define DLB2_VF2PF_RESP_BASE	  (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
+#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
+
+/* VF-initiated commands */
+enum dlb2_mbox_cmd_type {
+	DLB2_MBOX_CMD_REGISTER,
+	DLB2_MBOX_CMD_UNREGISTER,
+	DLB2_MBOX_CMD_GET_NUM_RESOURCES,
+	DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
+	DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
+	DLB2_MBOX_CMD_CREATE_LDB_PORT,
+	DLB2_MBOX_CMD_CREATE_DIR_PORT,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT,
+	DLB2_MBOX_CMD_DISABLE_LDB_PORT,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT,
+	DLB2_MBOX_CMD_DISABLE_DIR_PORT,
+	DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_MAP_QID,
+	DLB2_MBOX_CMD_UNMAP_QID,
+	DLB2_MBOX_CMD_START_DOMAIN,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
+	DLB2_MBOX_CMD_ARM_CQ_INTR,
+	DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
+	DLB2_MBOX_CMD_GET_SN_ALLOCATION,
+	DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
+	DLB2_MBOX_CMD_GET_COS_BW,
+	DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
+	DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_QE_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_CMD_TYPES,
+};
+
+static const char dlb2_mbox_cmd_type_strings[][128] = {
+	"DLB2_MBOX_CMD_REGISTER",
+	"DLB2_MBOX_CMD_UNREGISTER",
+	"DLB2_MBOX_CMD_GET_NUM_RESOURCES",
+	"DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_LDB_PORT",
+	"DLB2_MBOX_CMD_CREATE_DIR_PORT",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_DISABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_DISABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_MAP_QID",
+	"DLB2_MBOX_CMD_UNMAP_QID",
+	"DLB2_MBOX_CMD_START_DOMAIN",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
+	"DLB2_MBOX_CMD_ARM_CQ_INTR",
+	"DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
+	"DLB2_MBOX_CMD_GET_SN_ALLOCATION",
+	"DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
+	"DLB2_MBOX_CMD_GET_COS_BW",
+	"DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
+	"DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
+};
+
+/* PF-initiated commands */
+enum dlb2_mbox_vf_cmd_type {
+	DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
+	DLB2_MBOX_VF_CMD_NOTIFICATION,
+	DLB2_MBOX_VF_CMD_IN_USE,
+
+	/* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_VF_CMD_TYPES,
+};
+
+static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
+	"DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
+	"DLB2_MBOX_VF_CMD_NOTIFICATION",
+	"DLB2_MBOX_VF_CMD_IN_USE",
+};
+
+#define DLB2_MBOX_CMD_TYPE(hdr) \
+	(((struct dlb2_mbox_req_hdr *)hdr)->type)
+#define DLB2_MBOX_CMD_STRING(hdr) \
+	dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
+
+enum dlb2_mbox_status_type {
+	DLB2_MBOX_ST_SUCCESS,
+	DLB2_MBOX_ST_INVALID_CMD_TYPE,
+	DLB2_MBOX_ST_VERSION_MISMATCH,
+	DLB2_MBOX_ST_INVALID_OWNER_VF,
+};
+
+static const char dlb2_mbox_status_type_strings[][128] = {
+	"DLB2_MBOX_ST_SUCCESS",
+	"DLB2_MBOX_ST_INVALID_CMD_TYPE",
+	"DLB2_MBOX_ST_VERSION_MISMATCH",
+	"DLB2_MBOX_ST_INVALID_OWNER_VF",
+};
+
+#define DLB2_MBOX_ST_TYPE(hdr) \
+	(((struct dlb2_mbox_resp_hdr *)hdr)->status)
+#define DLB2_MBOX_ST_STRING(hdr) \
+	dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
+
+/* This structure is always the first field in a request structure */
+struct dlb2_mbox_req_hdr {
+	u32 type;
+};
+
+/* This structure is always the first field in a response structure */
+struct dlb2_mbox_resp_hdr {
+	u32 status;
+};
+
+struct dlb2_mbox_register_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 min_interface_version;
+	u16 max_interface_version;
+};
+
+struct dlb2_mbox_register_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 interface_version;
+	u8 pf_id;
+	u8 vf_id;
+	u8 is_auxiliary_vf;
+	u8 primary_vf_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u16 num_sched_domains;
+	u16 num_ldb_queues;
+	u16 num_ldb_ports;
+	u16 num_cos_ldb_ports[4];
+	u16 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 max_contiguous_hist_list_entries;
+	u16 num_ldb_credits;
+	u16 num_dir_credits;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 num_ldb_queues;
+	u32 num_ldb_ports;
+	u32 num_cos_ldb_ports[4];
+	u32 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u8 cos_strict;
+	u8 padding0[3];
+	u32 padding1;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 num_sequence_numbers;
+	u32 num_qid_inflights;
+	u32 num_atomic_inflights;
+	u32 lock_id_comp_level;
+	u32 depth_threshold;
+	u32 padding;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 depth_threshold;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u16 cq_depth;
+	u16 cq_history_list_size;
+	u8 cos_id;
+	u8 cos_strict;
+	u16 padding1;
+	u64 cq_base_address;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u64 cq_base_address;
+	u16 cq_depth;
+	u16 padding0;
+	s32 queue_id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_map_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+	u32 priority;
+	u32 padding0;
+};
+
+struct dlb2_mbox_map_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_start_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+};
+
+struct dlb2_mbox_start_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 is_ldb;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding0;
+};
+
+/*
+ * The alert_id and aux_alert_data follows the format of the alerts defined in
+ * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
+ * the aux_alert_data value varies depending on the alert.
+ */
+struct dlb2_mbox_vf_alert_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 alert_id;
+	u32 aux_alert_data;
+};
+
+enum dlb2_mbox_vf_notification_type {
+	DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
+	DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
+
+	/* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
+	NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
+};
+
+struct dlb2_mbox_vf_notification_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 notification;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 in_use;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 num;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 cos_id;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 mode;
+};
+
+#endif /* __DLB2_BASE_DLB2_MBOX_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
new file mode 100644
index 0000000..43f2125
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
@@ -0,0 +1,247 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_H
+#define __DLB2_OSDEP_H
+
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <rte_string_fns.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include "../dlb2_main.h"
+#include "dlb2_resource.h"
+#include "../../dlb2_log.h"
+#include "../../dlb2_user.h"
+
+
+#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
+#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
+#define DLB2_CSR_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
+#define DLB2_FUNC_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
+/* Map to PMDs logging interface */
+#define DLB2_ERR(dev, fmt, args...) \
+	DLB2_LOG_ERR(fmt, ## args)
+
+#define DLB2_INFO(dev, fmt, args...) \
+	DLB2_LOG_INFO(fmt, ## args)
+
+#define DLB2_DEBUG(dev, fmt, args...) \
+	DLB2_LOG_DBG(fmt, ## args)
+
+/**
+ * os_udelay() - busy-wait for a number of microseconds
+ * @usecs: delay duration.
+ */
+static inline void os_udelay(int usecs)
+{
+	rte_delay_us(usecs);
+}
+
+/**
+ * os_msleep() - sleep for a number of milliseconds
+ * @usecs: delay duration.
+ */
+static inline void os_msleep(int msecs)
+{
+	rte_delay_ms(msecs);
+}
+
+#define DLB2_PP_BASE(__is_ldb) \
+	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
+
+/**
+ * os_map_producer_port() - map a producer port into the caller's address space
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ *
+ * This function maps the requested producer port memory into the caller's
+ * address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void *os_map_producer_port(struct dlb2_hw *hw,
+					 u8 port_id,
+					 bool is_ldb)
+{
+	uint64_t addr;
+	uint64_t pp_dma_base;
+
+	pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
+	addr = (pp_dma_base + (PAGE_SIZE * port_id));
+
+	return (void *)(uintptr_t)addr;
+}
+
+/**
+ * os_unmap_producer_port() - unmap a producer port
+ * @addr: mapped producer port address
+ *
+ * This function undoes os_map_producer_port() by unmapping the producer port
+ * memory from the caller's address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(addr);
+}
+
+/**
+ * os_fence_hcw() - fence an HCW to ensure it arrives at the device
+ * @hw: dlb2_hw handle for a particular device.
+ * @pp_addr: producer port address
+ */
+static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
+{
+	RTE_SET_USED(hw);
+
+	/* To ensure outstanding HCWs reach the device, read the PP address. IA
+	 * memory ordering prevents reads from passing older writes, and the
+	 * mfence also ensures this.
+	 */
+	rte_mb();
+
+	*(volatile u64 *)pp_addr;
+}
+
+/**
+ * os_enqueue_four_hcws() - enqueue four HCWs to DLB
+ * @hw: dlb2_hw handle for a particular device.
+ * @hcw: pointer to the 64B-aligned contiguous HCW memory
+ * @addr: producer port address
+ */
+static inline void os_enqueue_four_hcws(struct dlb2_hw *hw,
+					struct dlb2_hcw *hcw,
+					void *addr)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	dlb2_dev->enqueue_four(hcw, addr);
+}
+
+/**
+ * DLB2_HW_ERR() - log an error message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_ERR(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_ERR(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/**
+ * DLB2_HW_DBG() - log an info message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_DBG(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_DEBUG(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/* The callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void *dlb2_complete_queue_map_unmap(void *__args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
+	int ret;
+
+	while (1) {
+		rte_spinlock_lock(&dlb2_dev->resource_mutex);
+
+		ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+		ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+		if (ret != 0) {
+			rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+			/* Relinquish the CPU so the application can process
+			 * its CQs, so this function doesn't deadlock.
+			 */
+			sched_yield();
+		} else {
+			break;
+		}
+	}
+
+	dlb2_dev->worker_launched = false;
+
+	rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+
+	return NULL;
+}
+
+
+/**
+ * os_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static inline void os_schedule_work(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+	pthread_t complete_queue_map_unmap_thread;
+	int ret;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
+				     "dlb_queue_unmap_waiter",
+				     NULL,
+				     dlb2_complete_queue_map_unmap,
+				     dlb2_dev);
+	if (ret)
+		DLB2_ERR(dlb2_dev,
+			 "Could not create queue complete map/unmap thread, err=%d\n",
+			 ret);
+	else
+		dlb2_dev->worker_launched = true;
+}
+
+/**
+ * os_worker_active() - query whether the map/unmap worker thread is active
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether a thread (launched by
+ * os_schedule_work()) is active. This function is used to determine
+ * whether or not to launch a worker thread.
+ */
+static inline bool os_worker_active(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	return dlb2_dev->worker_launched;
+}
+
+#endif /*  __DLB2_OSDEP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
new file mode 100644
index 0000000..423233b
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <rte_bitmap.h>
+#include <rte_string_fns.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include "../dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+	struct rte_bitmap *map;
+	unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+				    unsigned int len)
+{
+	struct dlb2_bitmap *bm;
+	void *mem;
+	uint32_t alloc_size;
+	uint32_t nbits = (uint32_t)len;
+
+	if (bitmap == NULL || nbits == 0)
+		return -EINVAL;
+
+	/* Allocate DLB2 bitmap control struct */
+	bm = rte_malloc("DLB2_PF",
+			sizeof(struct dlb2_bitmap),
+			RTE_CACHE_LINE_SIZE);
+
+	if (bm == NULL)
+		return -ENOMEM;
+
+	/* Allocate bitmap memory */
+	alloc_size = rte_bitmap_get_memory_footprint(nbits);
+	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
+	if (mem == NULL) {
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->map = rte_bitmap_init(len, mem, alloc_size);
+	if (bm->map == NULL) {
+		rte_free(mem);
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap == NULL)
+		return;
+
+	rte_free(bitmap->map);
+	rte_free(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++)
+		rte_bitmap_set(bitmap->map, i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	rte_bitmap_reset(bitmap->map);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set() - set a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
+				  unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_set(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+					unsigned int bit,
+					unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_set(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear() - clear a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
+				    unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_clear(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+					  unsigned int bit,
+					  unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_clear(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+						 unsigned int len)
+{
+	unsigned int i, j = 0;
+
+	if (bitmap  == NULL || bitmap->map == NULL || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			if (++j == len)
+				return i - j + 1;
+		} else {
+			j = 0;
+		}
+	}
+
+	/* No set bit range of length len? */
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			return i;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+	int weight = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			weight++;
+	}
+	return weight;
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				      bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+	int max_len = 0, len = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			len++;
+		} else {
+			if (len > max_len)
+				max_len = len;
+			len = 0;
+		}
+	}
+
+	if (len > max_len)
+		max_len = len;
+
+	return max_len;
+}
+
+/**
+ * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
+ * @dest: pointer to dlb2_bitmap structure, which will contain the results of
+ *	  the 'or' of src1 and src2.
+ * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
+ * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
+ *
+ * This function 'or's two bitmaps together and stores the result in a third
+ * bitmap. The source and destination bitmaps can be the same.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - One of the bitmaps is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
+				 struct dlb2_bitmap *src1,
+				 struct dlb2_bitmap *src2)
+{
+	unsigned int i, min;
+	int numset = 0;
+
+	if (dest  == NULL || dest->map == NULL ||
+	    src1  == NULL || src1->map == NULL ||
+	    src2  == NULL || src2->map == NULL)
+		return -EINVAL;
+
+	min = dest->len;
+	min = (min > src1->len) ? src1->len : min;
+	min = (min > src2->len) ? src2->len : min;
+
+	for (i = 0; i != min; i++) {
+		if  (rte_bitmap_get(src1->map, i) ||
+		     rte_bitmap_get(src2->map, i)) {
+			rte_bitmap_set(dest->map, i);
+			numset++;
+		} else {
+			rte_bitmap_clear(dest->map, i);
+		}
+	}
+
+	return numset;
+}
+
+#endif /*  __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
new file mode 100644
index 0000000..5531739
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_LIST_H
+#define __DLB2_OSDEP_LIST_H
+
+#include <rte_tailq.h>
+
+struct dlb2_list_entry {
+	TAILQ_ENTRY(dlb2_list_entry) node;
+};
+
+/* Dummy - just a struct definition */
+TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
+
+/* =================
+ * TAILQ Supplements
+ * =================
+ */
+
+#ifndef TAILQ_FOREACH_ENTRY
+#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)		\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter)						\
+		&& (ptr = container_of(iter, typeof(*(ptr)), name)); \
+	    (iter) = TAILQ_NEXT((iter), node))
+#endif
+
+#ifndef TAILQ_FOREACH_ENTRY_SAFE
+#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)	\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter) &&						\
+		(ptr = container_of(iter, typeof(*(ptr)), name)) &&\
+		((tvar) = TAILQ_NEXT((iter), node), 1);	\
+	    (iter) = (tvar))
+#endif
+
+/***********************/
+/*** List operations ***/
+/***********************/
+
+/**
+ * dlb2_list_init_head() - initialize the head of a list
+ * @head: list head
+ */
+static inline void dlb2_list_init_head(struct dlb2_list_head *head)
+{
+	TAILQ_INIT(head);
+}
+
+/**
+ * dlb2_list_add() - add an entry to a list
+ * @head: list head
+ * @entry: new list entry
+ */
+static inline void
+dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
+{
+	TAILQ_INSERT_TAIL(head, entry, node);
+}
+
+/**
+ * dlb2_list_del() - delete an entry from a list
+ * @entry: list entry
+ * @head: list head
+ */
+static inline void dlb2_list_del(struct dlb2_list_head *head,
+				 struct dlb2_list_entry *entry)
+{
+	TAILQ_REMOVE(head, entry, node);
+}
+
+/**
+ * dlb2_list_empty() - check if a list is empty
+ * @head: list head
+ *
+ * Return:
+ * Returns 1 if empty, 0 if not.
+ */
+static inline int dlb2_list_empty(struct dlb2_list_head *head)
+{
+	return TAILQ_EMPTY(head);
+}
+
+/**
+ * dlb2_list_splice() - splice a list
+ * @src_head: list to be added
+ * @ head: where src_head will be inserted
+ */
+static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
+				    struct dlb2_list_head *head)
+{
+	TAILQ_CONCAT(head, src_head, node);
+}
+
+/**
+ * DLB2_LIST_HEAD() - retrieve the head of the list
+ * @head: list head
+ * @type: type of the list variable
+ * @name: name of the list field within the containing struct
+ */
+#define DLB2_LIST_HEAD(head, type, name)                       \
+	(TAILQ_FIRST(&head) ?					\
+		container_of(TAILQ_FIRST(&head), type, name) :	\
+		NULL)
+
+/**
+ * DLB2_LIST_FOR_EACH() - iterate over a list
+ * @head: list head
+ * @ptr: pointer to struct containing a struct list
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ */
+#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
+	TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
+
+/**
+ * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
+ * an element is removed from the list while processing it.
+ * @ptr: pointer to struct containing a struct list
+ * @ptr_tmp: pointer to struct containing a struct list (temporary)
+ * @head: list head
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ * @iter_tmp: iterator variable (temporary)
+ */
+#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
+	TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
+
+#endif /*  __DLB2_OSDEP_LIST_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
new file mode 100644
index 0000000..0a48f7e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_TYPES_H
+#define __DLB2_OSDEP_TYPES_H
+
+#include <linux/types.h>
+
+#include <inttypes.h>
+#include <ctype.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Types for user mode PF PMD */
+typedef uint8_t         u8;
+typedef int8_t          s8;
+typedef uint16_t        u16;
+typedef int16_t         s16;
+typedef uint32_t        u32;
+typedef int32_t         s32;
+typedef uint64_t        u64;
+
+#define __iomem
+
+/* END types for user mode PF PMD */
+
+#endif /* __DLB2_OSDEP_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
new file mode 100644
index 0000000..43ecad4
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
@@ -0,0 +1,2527 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "dlb2_osdep_types.h"
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+	struct {
+		u32 isr_pend : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+	struct {
+		u32 vf0_reset_in_progress : 1;
+		u32 vf1_reset_in_progress : 1;
+		u32 vf2_reset_in_progress : 1;
+		u32 vf3_reset_in_progress : 1;
+		u32 vf4_reset_in_progress : 1;
+		u32 vf5_reset_in_progress : 1;
+		u32 vf6_reset_in_progress : 1;
+		u32 vf7_reset_in_progress : 1;
+		u32 vf8_reset_in_progress : 1;
+		u32 vf9_reset_in_progress : 1;
+		u32 vf10_reset_in_progress : 1;
+		u32 vf11_reset_in_progress : 1;
+		u32 vf12_reset_in_progress : 1;
+		u32 vf13_reset_in_progress : 1;
+		u32 vf14_reset_in_progress : 1;
+		u32 vf15_reset_in_progress : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+	struct {
+		u32 vec_mask : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+	struct {
+		u32 func_vf_bar_dis : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+	struct {
+		u32 total_vas : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+	struct {
+		u32 total_dir_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+	struct {
+		u32 total_ldb_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+	struct {
+		u32 total_dir_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+	struct {
+		u32 total_ldb_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+	struct {
+		u32 total_dir_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+	struct {
+		u32 total_ldb_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 meas : 1;
+		u32 debug : 7;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 cq_int_rearm : 1;
+		u32 dsi_error : 1;
+		u32 rsvd0 : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 rsvd0 : 3;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+	struct {
+		u32 qid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+	struct {
+		u32 vqid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+	struct {
+		u32 qid : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 debug : 8;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 isz : 1;
+		u32 dsi_error : 1;
+		u32 dlbrsvd : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 vf_synd0_parity : 1;
+		u32 vf_synd1_parity : 1;
+		u32 vf_synd2_parity : 1;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+	struct {
+		u32 sn_cfg_v : 1;
+		u32 fid_cfg_v : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS  0
+#define DLB2_CQ_ISR_MODE_MSI  1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI  3
+union dlb2_sys_ldb_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+	struct {
+		u32 keep_pf_ppid : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+	struct {
+		u32 illegal_hcw : 1;
+		u32 illegal_pp : 1;
+		u32 illegal_pasid : 1;
+		u32 illegal_qid : 1;
+		u32 disabled_qid : 1;
+		u32 illegal_ldb_qid_cfg : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+	struct {
+		u32 msix_0_ack : 1;
+		u32 msix_1_ack : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+	struct {
+		u32 msix_0_passthru : 1;
+		u32 msix_1_passthru : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED     0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+	struct {
+		u32 mode : 1;
+		u32 poll_mode : 1;
+		u32 poll_mask : 1;
+		u32 poll_lock : 1;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+	struct {
+		u32 cq : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 alarm : 1;
+		u32 cwd : 1;
+		u32 vf_pf_mb : 1;
+		u32 rsvd0 : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+	struct {
+		u32 qid_fid_limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+	struct {
+		u32 compress_code : 3;
+		u32 rsvd0 : 29;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+	(DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+	struct {
+		u32 count : 15;
+		u32 rsvd0 : 17;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+	struct {
+		u32 sn : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+	struct {
+		u32 mode : 3;
+		u32 slot : 4;
+		u32 rsvz0 : 1;
+		u32 grp : 1;
+		u32 rsvz1 : 1;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+	struct {
+		u32 en : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+	struct {
+		u32 write_pointer : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+	struct {
+		u32 base : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+	struct {
+		u32 limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+	struct {
+		u32 pop_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+	struct {
+		u32 push_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+	struct {
+		u32 depth : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+	struct {
+		u32 write_pointer : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 dlb_cor_alarm_enable : 1;
+		u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+		u32 cfg_64bytes_qe_dir_cq_mode : 1;
+		u32 pad_write_ldb : 1;
+		u32 pad_write_dir : 1;
+		u32 pad_first_write_ldb : 1;
+		u32 pad_first_write_dir : 1;
+		u32 rsvz0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+	struct {
+		u32 egress_credit_status_empty : 1;
+		u32 egress_credit_status_afull : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+		u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+		u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+		u32 chp_rop_pipe_credit_status_empty : 1;
+		u32 chp_rop_pipe_credit_status_afull : 1;
+		u32 qed_to_cq_pipe_credit_status_empty : 1;
+		u32 qed_to_cq_pipe_credit_status_afull : 1;
+		u32 egress_lsp_token_credit_status_empty : 1;
+		u32 egress_lsp_token_credit_status_afull : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 rsvz0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+	struct {
+		u32 sn_mode_0 : 3;
+		u32 rszv0 : 5;
+		u32 sn_mode_1 : 3;
+		u32 rszv1 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+	struct {
+		u32 unit_single_step_mode : 1;
+		u32 rr_en : 1;
+		u32 rszv0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+	struct {
+		u32 prio : 24;
+		u32 v : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+	struct {
+		u32 qid_p0 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p1 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p2 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p3 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+	struct {
+		u32 qid_p4 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p5 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p6 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p7 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+	struct {
+		u32 token_depth_select : 4;
+		u32 disable_wb_opt : 1;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+	struct {
+		u32 token_count : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+	struct {
+		u32 thresh : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+	(DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+	(DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+	struct {
+		u32 depth : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+	struct {
+		u32 cq : 8;
+		u32 qidix : 3;
+		u32 value : 1;
+		u32 nalb_haswork_v : 1;
+		u32 rlist_haswork_v : 1;
+		u32 slist_haswork_v : 1;
+		u32 inflight_ok_v : 1;
+		u32 aqed_nfull_v : 1;
+		u32 rsvz0 : 15;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+	struct {
+		u32 transfer : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+	struct {
+		u32 bw_range : 9;
+		u32 rsvz0 : 22;
+		u32 no_extra_credit : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+	struct {
+		u32 disab_atq_empty_arb : 1;
+		u32 inc_tok_unit_idle : 1;
+		u32 disab_rlist_pri : 1;
+		u32 inc_cmp_unit_idle : 1;
+		u32 rsvz0 : 2;
+		u32 dir_single_op : 1;
+		u32 dir_half_bw : 1;
+		u32 dir_single_out : 1;
+		u32 dir_disab_multi : 1;
+		u32 atq_single_op : 1;
+		u32 atq_half_bw : 1;
+		u32 atq_single_out : 1;
+		u32 atq_disab_multi : 1;
+		u32 dirrpl_single_op : 1;
+		u32 dirrpl_half_bw : 1;
+		u32 dirrpl_single_out : 1;
+		u32 lbrpl_single_op : 1;
+		u32 lbrpl_half_bw : 1;
+		u32 lbrpl_single_out : 1;
+		u32 ldb_single_op : 1;
+		u32 ldb_half_bw : 1;
+		u32 ldb_disab_multi : 1;
+		u32 atm_single_sch : 1;
+		u32 atm_single_cmp : 1;
+		u32 ldb_ce_tog_arb : 1;
+		u32 rsvz1 : 1;
+		u32 smon0_valid_sel : 2;
+		u32 smon0_value_sel : 1;
+		u32 smon0_compare_sel : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+	struct {
+		u32 chp_pf_reset_done : 1;
+		u32 rop_pf_reset_done : 1;
+		u32 lsp_pf_reset_done : 1;
+		u32 nalb_pf_reset_done : 1;
+		u32 ap_pf_reset_done : 1;
+		u32 dp_pf_reset_done : 1;
+		u32 qed_pf_reset_done : 1;
+		u32 dqed_pf_reset_done : 1;
+		u32 aqed_pf_reset_done : 1;
+		u32 sys_pf_reset_done : 1;
+		u32 pf_reset_active : 1;
+		u32 flrsm_state : 7;
+		u32 rsvd0 : 13;
+		u32 dlb_proc_reset_done : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+	struct {
+		u32 chp_pipeidle : 1;
+		u32 rop_pipeidle : 1;
+		u32 lsp_pipeidle : 1;
+		u32 nalb_pipeidle : 1;
+		u32 ap_pipeidle : 1;
+		u32 dp_pipeidle : 1;
+		u32 qed_pipeidle : 1;
+		u32 dqed_pipeidle : 1;
+		u32 aqed_pipeidle : 1;
+		u32 sys_pipeidle : 1;
+		u32 chp_unit_idle : 1;
+		u32 rop_unit_idle : 1;
+		u32 lsp_unit_idle : 1;
+		u32 nalb_unit_idle : 1;
+		u32 ap_unit_idle : 1;
+		u32 dp_unit_idle : 1;
+		u32 qed_unit_idle : 1;
+		u32 dqed_unit_idle : 1;
+		u32 aqed_unit_idle : 1;
+		u32 sys_unit_idle : 1;
+		u32 rsvd1 : 4;
+		u32 mstr_cfg_ring_idle : 1;
+		u32 mstr_cfg_mstr_idle : 1;
+		u32 mstr_flr_clkreq_b : 1;
+		u32 mstr_proc_idle : 1;
+		u32 mstr_proc_idle_masked : 1;
+		u32 rsvd0 : 2;
+		u32 dlb_func_idle : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+	struct {
+		u32 prochot : 1;
+		u32 pgcb_dlb_idle : 1;
+		u32 pgcb_dlb_pg_rdy_ack_b : 1;
+		u32 pmsm_pgcb_req_b : 1;
+		u32 pgbc_pmc_pg_req_b : 1;
+		u32 pmc_pgcb_pg_ack_b : 1;
+		u32 pmc_pgcb_fet_en_b : 1;
+		u32 pgcb_fet_en_b : 1;
+		u32 rsvz0 : 1;
+		u32 rsvz1 : 1;
+		u32 fuse_force_on : 1;
+		u32 fuse_proc_disable : 1;
+		u32 rsvz2 : 1;
+		u32 rsvz3 : 1;
+		u32 pm_fsm_d0tod3_ok : 1;
+		u32 pm_fsm_d3tod0_ok : 1;
+		u32 dlb_in_d3 : 1;
+		u32 rsvz4 : 7;
+		u32 pmsm : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+	struct {
+		u32 disable : 1;
+		u32 rsvz0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+	struct {
+		u32 isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+	struct {
+		u32 pf_isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+	struct {
+		u32 isr_pend : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+	struct {
+		u32 reset_in_progress : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+	struct {
+		u32 vf_msi_isr : 32;
+	} field;
+	u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
new file mode 100644
index 0000000..6de8b95
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_mbox.h"
+#include "dlb2_osdep.h"
+#include "dlb2_osdep_bitmap.h"
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+	int i;
+
+	dlb2_list_init_head(&domain->used_ldb_queues);
+	dlb2_list_init_head(&domain->used_dir_pq_pairs);
+	dlb2_list_init_head(&domain->avail_ldb_queues);
+	dlb2_list_init_head(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+	int i;
+
+	dlb2_list_init_head(&rsrc->avail_domains);
+	dlb2_list_init_head(&rsrc->used_domains);
+	dlb2_list_init_head(&rsrc->avail_ldb_queues);
+	dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
+	arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
+	arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
+	arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb2_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	struct dlb2_list_entry *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. This is application
+	 * dependent, but the driver interleaves port IDs as much as possible
+	 * to reduce the likelihood of this. This initial allocation maximizes
+	 * the average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	/* Zero-out resource tracking data structures */
+	memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+	memset(&hw->pf, 0, sizeof(hw->pf));
+
+	dlb2_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+		dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+		dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_domains, list);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_ldb_queues, list);
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+		struct dlb2_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
+			      &port->func_list);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+				DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+	if (ret)
+		goto unwind;
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+					DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+		if (ret)
+			goto unwind;
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb2_resource_free(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+	r0.field.disable = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
new file mode 100644
index 0000000..503fdf3
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
@@ -0,0 +1,1913 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_osdep_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_start_domain() - start a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: start domain arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int dlb2_hw_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp,
+			 bool vdev_request,
+			 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_request,
+		    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding unmap procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_finish_map_qid_procedures() - finish any pending map procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding map procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ *					interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+			  int port_id,
+			  bool is_ldb,
+			  bool vdev_request,
+			  unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+					 u32 *ldb_interrupts,
+					 u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+				 u32 *ldb_interrupts,
+				 u32 *dir_interrupts);
+
+/**
+ * dlb2_read_vf_intr_status() - read the VF interrupt status register
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function can be called from a VF's interrupt handler to determine
+ * which interrupts have fired. The first 31 bits correspond to CQ interrupt
+ * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
+ *
+ * Return:
+ * Returns a bit vector indicating which interrupt vectors are active.
+ */
+u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_intr_status() - ack VF interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function ACKs a VF's interrupts. Its interrupts argument should be the
+ * value returned by dlb2_read_vf_intr_status().
+ */
+void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function clears the VF's MSI interrupt pending register. Its interrupts
+ * argument should be contain the MSI vectors to ACK. For example, if MSI MME
+ * is in mode 0, then one bit 0 should ever be set.
+ */
+void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When done processing the PF mailbox request, this function unsets
+ * the PF's mailbox ISR register.
+ */
+void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
+
+/**
+ * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
+ *					vdevs
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the vdev->PF ISR fires, this function can be called to determine which
+ * vdev(s) are requesting service. This bitvector must be passed to
+ * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
+ * vdevs.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested service.
+ */
+u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ *
+ * When done processing all VF mailbox requests, this function unsets the VF's
+ * mailbox ISR register.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
+ *				    FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the VF FLR ISR fires, this function can be called to determine which
+ * VF(s) are requesting FLRs. This bitvector must passed to
+ * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
+ */
+u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF FLR requests, this function unsets the VF's FLR
+ * ISR register.
+ */
+void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF requests, this function communicates to the
+ * hardware that processing is complete.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
+			     u32 mbox_bitvec,
+			     u32 flr_bitvec);
+
+/**
+ * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the watchdog timer interrupt cause registers to
+ * determine which port(s) had a watchdog timeout, and notifies the
+ * application(s) that own the port(s).
+ */
+void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads and logs the alarm syndrome, then acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_request,
+			      unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of resources in use by the PF or a VF. It
+ * fills in the fields that args points to, except the following:
+ * - max_contiguous_atomic_inflights
+ * - max_contiguous_hist_list_entries
+ * - max_contiguous_ldb_credits
+ * - max_contiguous_dir_credits
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
+				   struct dlb2_get_num_resources_args *arg,
+				   bool vdev_request,
+				   unsigned int vdev_id);
+
+/**
+ * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
+ *				       the PF
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function sends a VF->PF mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the PF has processed
+ * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
+ * when the PF has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
+ *				 request
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * processing a VF->PF mailbox request. It should only be called after sending
+ * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vf_flr_complete() - check the status of a VF FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * executing the VF FLR. It should only be called after setting the VF's FLR
+ * bit.
+ */
+bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
+ *					vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function sends a PF->vdev mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the vdev has
+ * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
+ * determine when the vdev has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
+ *			       request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function returns a boolean indicating whether the vdev has finished
+ * processing a PF->vdev mailbox request. It should only be called after
+ * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
+			     unsigned int vf_id,
+			     void *data,
+			     int len);
+
+/**
+ * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
+			       unsigned int vf_id,
+			       void *data,
+			       int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_reset_vdev() - reset the hardware owned by a virtual device
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function resets the hardware owned by a vdev, by resetting the vdev's
+ * domains one by one.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function returns whether or not the vdev's resource assignments are
+ * locked. If locked, no resources can be added to or subtracted from the
+ * group.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_lock_vdev() - lock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sets a flag indicating that the vdev is using its resources.
+ * When the vdev is locked, its resource assignment cannot be changed.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_unlock_vdev() - unlock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function unlocks the vdev's resource assignment, allowing it to be
+ * modified.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of scheduling domains to assign to this vdev
+ *
+ * This function assigns num scheduling domains to the specified vdev. If the
+ * vdev already has domains assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB queues to assign to this vdev
+ *
+ * This function assigns num LDB queues to the specified vdev. If the vdev
+ * already has LDB queues assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports to the specified vdev. If the vdev
+ * already has LDB ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @cos: class-of-service ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports from class-of-service cos to the
+ * specified vdev. If the vdev already has LDB ports from this class-of-service
+ * assigned, this existing assignment is adjusted accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
+				   u32 id,
+				   u32 cos,
+				   u32 num);
+
+/**
+ * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR ports to assign to this vdev
+ *
+ * This function assigns num DIR ports to the specified vdev. If the vdev
+ * already has DIR ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB credit credits to assign to this vdev
+ *
+ * This function assigns num LDB credit to the specified vdev. If the vdev
+ * already has LDB credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR credits to assign to this vdev
+ *
+ * This function assigns num DIR credit to the specified vdev. If the vdev
+ * already has DIR credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of history list entries to assign to this vdev
+ *
+ * This function assigns num history list entries to the specified vdev. If the
+ * vdev already has history list entries assigned, this existing assignment is
+ * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
+ * this function may fail if a sufficiently large contiguous chunk is not
+ * available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of atomic inflights to assign to this vdev
+ *
+ * This function assigns num atomic inflights to the specified vdev. If the vdev
+ * already has atomic inflights assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of entries, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function takes any resources currently assigned to the vdev and
+ * reassigns them to the PF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_notify_vf() - send an alarm to a VF
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID
+ * @notification: notification
+ *
+ * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
+ *
+ * Return:
+ * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
+ */
+int dlb2_notify_vf(struct dlb2_hw *hw,
+		   unsigned int vf_id,
+		   u32 notification);
+
+/**
+ * dlb2_vdev_in_use() - query whether a virtual device is in use
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sends a mailbox request to the vdev to query whether the vdev
+ * is in use.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
+ * an internal error occurs.
+ */
+int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+enum dlb2_virt_mode {
+	DLB2_VIRT_NONE,
+	DLB2_VIRT_SRIOV,
+	DLB2_VIRT_SIOV,
+
+	/* NUM_DLB2_VIRT_MODES must be last */
+	NUM_DLB2_VIRT_MODES,
+};
+
+/**
+ * dlb2_hw_set_virt_mode() - set the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: either none, SR-IOV, or Scalable IOV.
+ *
+ * This function sets the virtualization mode of the device. This controls
+ * whether the device uses a software or hardware mailbox.
+ *
+ * This should be called by the PF driver when either SR-IOV or Scalable IOV is
+ * selected as the virtualization mechanism, and by the VF/VDEV driver during
+ * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
+ *
+ * Errors:
+ * EINVAL - Invalid mode.
+ */
+int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
+
+/**
+ * dlb2_hw_get_virt_mode() - get the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function gets the virtualization mode of the device.
+ */
+enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_register_sw_mbox() - register a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
+ * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
+ * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
+ * @inject_arg: user argument for pf2vdev_inject callback.
+ *
+ * When Scalable IOV is enabled, the VDCM must register a software mailbox for
+ * every virtual device during vdev creation.
+ *
+ * This function notifies the driver to use a software mailbox using the
+ * provided pointers, instead of the device's hardware mailbox. When the driver
+ * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
+ * go to the software mailbox instead of the hardware one. This is used in
+ * Scalable IOV virtualization.
+ */
+void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
+			      unsigned int vdev_id,
+			      u32 *vdev2pf_mbox,
+			      u32 *pf2vdev_mbox,
+			      void (*pf2vdev_inject)(void *),
+			      void *inject_arg);
+
+/**
+ * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function notifies the driver to stop using a previously registered
+ * software mailbox.
+ */
+void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ * @addr_lo: least-significant 32 bits of address.
+ * @data: 32 data bits.
+ *
+ * This sets up the CQ's IMS entry with the provided address and data values.
+ * This function should only be called if the device is configured for Scalable
+ * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
+ * not needed.
+ */
+void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb,
+				u32 addr_lo,
+				u32 data);
+
+/**
+ * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ *
+ * This clears the CQ's IMS entry, reverting it to its reset state.
+ */
+void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb);
+
+/**
+ * dlb2_hw_register_pasid() - register a vdev's PASID
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @pasid: the vdev's PASID.
+ *
+ * This function stores the user-supplied PASID, and uses it when configuring
+ * the vdev's CQs.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+int dlb2_hw_register_pasid(struct dlb2_hw *hw,
+			   unsigned int vdev_id,
+			   unsigned int pasid);
+
+/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vf_request,
+				unsigned int vf_id);
+
+/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ *	to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ *	port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ *	    cause the total bandwidth across all classes of service to exceed
+ *	    100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+enum dlb2_wd_tmo {
+	/* 40s watchdog timeout */
+	DLB2_WD_TMO_40S,
+	/* 10s watchdog timeout */
+	DLB2_WD_TMO_10S,
+	/* 1s watchdog timeout */
+	DLB2_WD_TMO_1S,
+
+	/* Must be last */
+	NUM_DLB2_WD_TMOS,
+};
+
+/**
+ * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
+ *	caller-specified timeout.
+ * @hw: dlb2_hw handle for a particular device.
+ * @tmo: watchdog timeout.
+ *
+ * This function should be called during device initialization and after reset.
+ * The watchdog timer interrupt must also be enabled per-CQ, using either
+ * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid timeout.
+ */
+int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
+
+/**
+ * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid directed port ID.
+ */
+int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid load-balanced port ID.
+ */
+int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *	ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * load-balanced CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_ldb_cq_interrupt().
+ */
+void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical directed port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * directed CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_dir_cq_interrupt().
+ */
+void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+#endif /* __DLB2_RESOURCE_H */
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
new file mode 100644
index 0000000..bd8590d
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -0,0 +1,615 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "base/dlb2_resource.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_regs.h"
+#include "dlb2_main.h"
+#include "../dlb2_user.h"
+#include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
+#include "../dlb2_inline_fns.h"
+
+#define PF_ID_ZERO 0	/* PF ONLY! */
+#define NO_OWNER_VF 0	/* PF ONLY! */
+#define NOT_VF_REQ false /* PF ONLY! */
+
+#define DLB2_PCI_CFG_SPACE_SIZE 256
+#define DLB2_PCI_CAP_POINTER 0x34
+#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
+#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
+#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
+#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
+#define DLB2_PCI_EXT_CAP_ID_ERR 1
+#define DLB2_PCI_ERR_UNCOR_MASK 8
+#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
+
+#define DLB2_PCI_EXP_DEVCTL 8
+#define DLB2_PCI_LNKCTL 16
+#define DLB2_PCI_SLTCTL 24
+#define DLB2_PCI_RTCTL 28
+#define DLB2_PCI_EXP_DEVCTL2 40
+#define DLB2_PCI_LNKCTL2 48
+#define DLB2_PCI_SLTCTL2 56
+#define DLB2_PCI_CMD 4
+#define DLB2_PCI_X_CMD 2
+#define DLB2_PCI_EXP_DEVSTA 10
+#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
+#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
+
+#define DLB2_PCI_CAP_ID_EXP       0x10
+#define DLB2_PCI_CAP_ID_MSIX      0x11
+#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
+#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
+#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
+
+#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
+#define DLB2_PCI_PRI_ALLOC_REQ           0xC
+#define DLB2_PCI_PRI_CTRL                0x4
+#define DLB2_PCI_MSIX_FLAGS              0x2
+#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
+#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
+#define DLB2_PCI_ERR_ROOT_STATUS         0x30
+#define DLB2_PCI_ERR_COR_STATUS          0x10
+#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
+#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
+#define DLB2_PCI_ACS_CAP                 0x4
+#define DLB2_PCI_ACS_CTRL                0x6
+#define DLB2_PCI_ACS_SV                  0x1
+#define DLB2_PCI_ACS_RR                  0x4
+#define DLB2_PCI_ACS_CR                  0x8
+#define DLB2_PCI_ACS_UF                  0x10
+#define DLB2_PCI_ACS_EC                  0x20
+
+static int
+dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint32_t hdr;
+	size_t sz;
+	int pos;
+
+	pos = DLB2_PCI_CFG_SPACE_SIZE;
+	sz = sizeof(hdr);
+
+	while (pos > 0xFF) {
+		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
+			return -1;
+
+		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
+			return pos;
+
+		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint8_t pos;
+	int ret;
+	uint16_t hdr;
+
+	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
+	pos &= 0xFC;
+
+	if (ret != 1)
+		return -1;
+
+	while (pos > 0x3F) {
+		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
+		if (ret != 2)
+			return -1;
+
+		if (DLB2_PCI_CAP_ID(hdr) == id)
+			return pos;
+
+		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
+			return -1;
+
+		pos = DLB2_PCI_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+	int i;
+
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIR64B))
+		dlb2_dev->enqueue_four = dlb2_movdir64b;
+	else
+		dlb2_dev->enqueue_four = dlb2_movntdq;
+
+	/* Initialize software state */
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++)
+		dlb2_list_init_head(&dlb2_dev->ldb_port_pages[i].list);
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++)
+		dlb2_list_init_head(&dlb2_dev->dir_port_pages[i].list);
+
+	rte_spinlock_init(&dlb2_dev->resource_mutex);
+	rte_spinlock_init(&dlb2_dev->measurement_lock);
+
+	return 0;
+}
+
+static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
+{
+	u32 retries = 0;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
+		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+		union dlb2_cfg_mstr_cfg_pm_status pm_st;
+		u32 addr;
+
+		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+			break;
+
+		rte_delay_ms(1);
+	};
+
+	if (retries == DLB2_READY_RETRY_LIMIT) {
+		printf("[%s()] wait for device ready timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+struct dlb2_dev *
+dlb2_probe(struct rte_pci_device *pdev)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret = 0;
+
+	DLB2_INFO(dlb2_dev, "probe\n");
+
+	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
+			      RTE_CACHE_LINE_SIZE);
+
+	if (dlb2_dev == NULL) {
+		ret = -ENOMEM;
+		goto dlb2_dev_malloc_fail;
+	}
+
+	/* PCI Bus driver has already mapped bar space into process.
+	 * Save off our IO register and FUNC addresses.
+	 */
+
+	/* BAR 0 */
+	if (pdev->mem_resource[0].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
+	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.func_kva,
+		  (void *)dlb2_dev->hw.func_phys_addr,
+		  (void *)(pdev->mem_resource[0].len));
+
+	/* BAR 2 */
+	if (pdev->mem_resource[2].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
+	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.csr_kva,
+		  (void *)dlb2_dev->hw.csr_phys_addr,
+		  (void *)(pdev->mem_resource[2].len));
+
+	dlb2_dev->pdev = pdev;
+
+	/* PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb2_pf_enable_pm(dlb2_dev);
+
+	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb2_pf_reset(dlb2_dev);
+	if (ret)
+		goto dlb2_reset_fail;
+
+	ret = dlb2_pf_init_driver_state(dlb2_dev);
+	if (ret)
+		goto init_driver_state_fail;
+
+	ret = dlb2_resource_init(&dlb2_dev->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	return dlb2_dev;
+
+resource_init_fail:
+	dlb2_resource_free(&dlb2_dev->hw);
+init_driver_state_fail:
+dlb2_reset_fail:
+pci_mmap_bad_addr:
+wait_for_device_ready_fail:
+	rte_free(dlb2_dev);
+dlb2_dev_malloc_fail:
+	rte_errno = ret;
+	return NULL;
+}
+
+int
+dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
+{
+	int ret = 0;
+	int i = 0;
+	uint32_t dword[16];
+	uint16_t cmd;
+	off_t off;
+
+	uint16_t dev_ctl_word;
+	uint16_t dev_ctl2_word;
+	uint16_t lnk_word;
+	uint16_t lnk_word2;
+	uint16_t slt_word;
+	uint16_t slt_word2;
+	uint16_t rt_ctl_word;
+	uint32_t pri_reqs_dword;
+	uint16_t pri_ctrl_word;
+
+	int pcie_cap_offset;
+	int pri_cap_offset;
+	int msix_cap_offset;
+	int err_cap_offset;
+	int acs_cap_offset;
+	int wait_count;
+
+	uint16_t devsta_busy_word;
+	uint16_t devctl_word;
+
+	struct rte_pci_device *pdev = dlb2_dev->pdev;
+
+	/* Save PCI config state */
+
+	for (i = 0; i < 16; i++) {
+		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
+			return ret;
+	}
+
+	pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
+
+	if (pcie_cap_offset < 0) {
+		printf("[%s()] failed to find the pcie capability\n",
+		       __func__);
+		return pcie_cap_offset;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
+		dev_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
+		lnk_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
+		slt_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_RTCTL;
+	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
+		rt_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
+		dev_ctl2_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
+		lnk_word2 = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
+		slt_word2 = 0;
+
+	off = DLB2_PCI_EXT_CAP_ID_PRI;
+	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (pri_cap_offset >= 0) {
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
+			pri_reqs_dword = 0;
+	}
+
+	/* clear the PCI command register before issuing the FLR */
+
+	off = DLB2_PCI_CMD;
+	cmd = 0;
+	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+		printf("[%s()] failed to write the pci command\n",
+		       __func__);
+		return ret;
+	}
+
+	/* issue the FLR */
+	for (wait_count = 0; wait_count < 4; wait_count++) {
+		int sleep_time;
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
+		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to read the pci device status\n",
+			       __func__);
+			return ret;
+		}
+
+		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
+			break;
+
+		sleep_time = (1 << (wait_count)) * 100;
+		rte_delay_ms(sleep_time);
+	}
+
+	if (wait_count == 4) {
+		printf("[%s()] wait for pci pending transactions timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to read the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
+
+	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to write the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	rte_delay_ms(100);
+
+	/* Restore PCI config state */
+
+	if (pcie_cap_offset >= 0) {
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie device control at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_RTCTL;
+		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	if (pri_cap_offset >= 0) {
+		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
+
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
+		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ERR;
+	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (err_cap_offset >= 0) {
+		uint32_t tmp;
+
+		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	for (i = 16; i > 0; i--) {
+		off = (i - 1) * 4;
+		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_CMD;
+	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
+		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+			printf("[%s()] failed to write the pci command\n",
+			       __func__);
+			return ret;
+		}
+	}
+
+	msix_cap_offset = dlb2_pci_find_capability(pdev,
+						   DLB2_PCI_CAP_ID_MSIX);
+	if (msix_cap_offset >= 0) {
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
+			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ACS;
+	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (acs_cap_offset >= 0) {
+		uint16_t acs_cap, acs_ctrl, acs_mask;
+		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
+		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
+			acs_cap = 0;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
+		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
+		acs_ctrl |= (acs_cap & acs_mask);
+
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
+		acs_mask |= DLB2_PCI_ACS_EC;
+		acs_ctrl &= ~acs_mask;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
new file mode 100644
index 0000000..ec96f11
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -0,0 +1,106 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+#endif
+
+#include "base/dlb2_hw_types.h"
+#include "../dlb2_user.h"
+
+#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
+
+struct dlb2_dev;
+
+struct dlb2_port_memory {
+	struct dlb2_list_head list;
+	void *cq_base;
+	bool valid;
+};
+
+struct dlb2_dev {
+	struct rte_pci_device *pdev;
+	struct dlb2_hw hw;
+	/* struct list_head list; */
+	struct device *dlb2_device;
+	struct dlb2_port_memory ldb_port_pages[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_port_memory dir_port_pages[DLB2_MAX_NUM_DIR_PORTS];
+	/* The enqueue_four function enqueues four HCWs (one cache-line worth)
+	 * to the DLB2, using whichever mechanism is supported by the platform
+	 * on which this driver is running.
+	 */
+	void (*enqueue_four)(void *qe4, void *pp_addr);
+
+	bool domain_reset_failed;
+	/* The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	rte_spinlock_t resource_mutex;
+	rte_spinlock_t measurement_lock;
+	bool worker_launched;
+	u8 revision;
+};
+
+struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
+int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
+int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+
+#endif /* __DLB2_MAIN_H */
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
new file mode 100644
index 0000000..7d64309
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -0,0 +1,244 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_devargs.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_pci.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
+#include "../dlb2_inline_fns.h"
+#include "dlb2_main.h"
+#include "base/dlb2_hw_types.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_resource.h"
+
+static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
+
+static void
+dlb2_pf_low_level_io_init(void)
+{
+	int i;
+	/* Addresses will be initialized at port create */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		/* First directed ports */
+		dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
+
+		/* Now load balanced ports */
+		dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
+	}
+}
+
+static int
+dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
+{
+	RTE_SET_USED(handle);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+static int
+dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
+			   uint8_t *revision)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	*revision = dlb2_dev->revision;
+
+	return 0;
+}
+
+static void
+dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+	dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_num_resources_args *rsrcs)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
+}
+
+static int
+dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
+			 enum dlb2_cq_poll_modes *mode)
+{
+	RTE_SET_USED(handle);
+
+	*mode = DLB2_CQ_POLL_MODE_SPARSE;
+
+	return 0;
+}
+
+static void
+dlb2_pf_iface_fn_ptrs_init(void)
+{
+	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
+	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
+	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
+	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
+	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+}
+
+/* PCI DEV HOOKS */
+static int
+dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
+{
+	int ret = 0;
+	struct rte_pci_device *pci_dev;
+	struct dlb2_devargs dlb2_args = {
+		.socket_id = rte_socket_id(),
+		.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+		.num_dir_credits_override = -1,
+		.qid_depth_thresholds = { {0} },
+		.cos_id = DLB2_COS_DEFAULT
+	};
+	struct dlb2_eventdev *dlb2;
+
+	DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
+		     eventdev->data->dev_id, eventdev->data->socket_id);
+
+	dlb2_pf_iface_fn_ptrs_init();
+
+	pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
+
+		/* Probe the DLB2 PF layer */
+		dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
+
+		if (dlb2->qm_instance.pf_dev == NULL) {
+			DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
+				     rte_errno);
+			ret = -rte_errno;
+			goto dlb2_probe_failed;
+		}
+
+		/* Were we invoked with runtime parameters? */
+		if (pci_dev->device.devargs) {
+			ret = dlb2_parse_params(pci_dev->device.devargs->args,
+						pci_dev->device.devargs->name,
+						&dlb2_args);
+			if (ret) {
+				DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
+					     ret, rte_errno);
+				goto dlb2_probe_failed;
+			}
+		}
+
+		ret = dlb2_primary_eventdev_probe(eventdev,
+						  event_dlb2_pf_name,
+						  &dlb2_args);
+	} else {
+		ret = dlb2_secondary_eventdev_probe(eventdev,
+						    event_dlb2_pf_name);
+	}
+	if (ret)
+		goto dlb2_probe_failed;
+
+	DLB2_LOG_INFO("DLB2 PF Probe success\n");
+
+	return 0;
+
+dlb2_probe_failed:
+
+	DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
+
+	return ret;
+}
+
+#define EVENTDEV_INTEL_VENDOR_ID 0x8086
+
+static const struct rte_pci_id pci_id_dlb2_map[] = {
+	{
+		RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
+			       PCI_DEVICE_ID_INTEL_DLB2_PF)
+	},
+	{
+		.vendor_id = 0,
+	},
+};
+
+static int
+event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
+		     struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
+					     sizeof(struct dlb2_eventdev),
+					     dlb2_eventdev_pci_init,
+					     event_dlb2_pf_name);
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+}
+
+static int
+event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_remove(pci_dev, NULL);
+
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+
+}
+
+static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
+	.id_table = pci_id_dlb2_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe = event_dlb2_pci_probe,
+	.remove = event_dlb2_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 06/22] event/dlb2: add probe

[Jerin Jacob]

On Sat, Oct 17, 2020 at 11:52 PM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> The DLB2 hardware is a PCI device. This commit adds
> support for probe and other initialization. The
> dlb2_iface.[ch] files implement a flexible interface
> that supports both the PF PMD and the bifurcated PMD.
> The bifurcated PMD will be released in a future
> patch set. Note that the flexible interface is only
> used for configuration, and is not used in the data
> path. The shared code is added in pf/base.
> This PMD supports command line parameters, and those
> are parsed at probe-time.
>
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>



There is a build issue clang10.

[for-main]dell[dpdk-next-eventdev] $ clang -v
clang version 10.0.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/10.2.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/8.4.0
Found candidate GCC installation:
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.2.0
Found candidate GCC installation:
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/8.4.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/8.4.0
Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0
Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/8.4.0
Selected GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.2.0
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64

meson  -Dexamples=l3fwd --buildtype=debugoptimized --werror
--default-library=static /export/dpdk-next-eventdev/devtools/..
./build-clang-static



ccache clang -Idrivers/libtmp_rte_pmd_dlb2_event.a.p -Idrivers
-I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
-Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
-I../config -Ilib/librte_eal/include -I../lib/librte_e
al/include -Ilib/librte_eal/linux/include
-I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
-I../lib/librte_eal/x86/include -Ilib/librte_eal/common
-I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
-Ilib/librte_kv
args -I../lib/librte_kvargs -Ilib/librte_metrics
-I../lib/librte_metrics -Ilib/librte_telemetry
-I../lib/librte_telemetry -Ilib/librte_ring -I../lib/librte_ring
-Ilib/librte_ethdev -I../lib/librte_ethdev -Ilib/librte_net
-I../lib/librte_net
 -Ilib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
-I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
-Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_timer
-I../lib/librte_timer -Ilib/librte_cryptodev -I../lib/li
brte_cryptodev -Ilib/librte_pci -I../lib/librte_pci -Idrivers/bus/pci
-I../drivers/bus/pci -I../drivers/bus/pci/linux -Xclang
-fcolor-diagnostics -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
-Werror -O2 -g -include rte_config.h -Wextra
-Wcast-qual -Wdeprecated -Wformat-nonliteral -Wformat-security
-Wmissing-declarations -Wmissing-prototypes -Wnested-externs
-Wold-style-definition -Wpointer-arith -Wsign-compare
-Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-pa
cked-member -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC
-march=native -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_pf_dlb2_pf.c.o -MF
drivers/libtmp_rte_pmd_dlb2_event.a.p/ev
ent_dlb2_pf_dlb2_pf.c.o.d -o
drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_pf_dlb2_pf.c.o -c
../drivers/event/dlb2/pf/dlb2_pf.c
In file included from ../drivers/event/dlb2/pf/dlb2_pf.c:36:
../drivers/event/dlb2/pf/../dlb2_inline_fns.h:45:2: error: use of
unknown builtin '__builtin_ia32_movntdq'
[-Wimplicit-function-declaration]
        __builtin_ia32_movntdq((__v2di *)pp_addr + 0, (__v2di)src_data0);
        ^
../drivers/event/dlb2/pf/../dlb2_inline_fns.h:45:2: note: did you mean
'__builtin_ia32_movntq'?
/usr/lib/clang/10.0.1/include/xmmintrin.h:2122:3: note:
'__builtin_ia32_movntq' declared here
  __builtin_ia32_movntq(__p, __a);
  ^
In file included from ../drivers/event/dlb2/pf/dlb2_pf.c:36:
../drivers/event/dlb2/pf/../dlb2_inline_fns.h:61:2: error: use of
unknown builtin '__builtin_ia32_movntdq'
[-Wimplicit-function-declaration]
        __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
        ^
2 errors generated.
[2124/2479] Compiling C object
drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_dlb2.c.o
FAILED: drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_dlb2.c.o
ccache clang -Idrivers/libtmp_rte_pmd_dlb2_event.a.p -Idrivers
-I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
-Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
-I../config -Ilib/librte_eal/include -I../lib/librte_e
al/include -Ilib/librte_eal/linux/include
-I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
-I../lib/librte_eal/x86/include -Ilib/librte_eal/common
-I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
-Ilib/librte_kv
args -I../lib/librte_kvargs -Ilib/librte_metrics
-I../lib/librte_metrics -Ilib/librte_telemetry
-I../lib/librte_telemetry -Ilib/librte_ring -I../lib/librte_ring
-Ilib/librte_ethdev -I../lib/librte_ethdev -Ilib/librte_net
-I../lib/librte_net
 -Ilib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
-I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
-Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_timer
-I../lib/librte_timer -Ilib/librte_cryptodev -I../lib/li
brte_cryptodev -Ilib/librte_pci -I../lib/librte_pci -Idrivers/bus/pci
-I../drivers/bus/pci -I../drivers/bus/pci/linux -Xclang
-fcolor-diagnostics -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
-Werror -O2 -g -include rte_config.h -Wextra
-Wcast-qual -Wdeprecated -Wformat-nonliteral -Wformat-security
-Wmissing-declarations -Wmissing-prototypes -Wnested-externs
-Wold-style-definition -Wpointer-arith -Wsign-compare
-Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-pa
cked-member -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC
-march=native -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -MQ
drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_dlb2.c.o -MF
drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dl
b2_dlb2.c.o.d -o
drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_dlb2.c.o -c
../drivers/event/dlb2/dlb2.c
In file included from ../drivers/event/dlb2/dlb2.c:35:
../drivers/event/dlb2/dlb2_inline_fns.h:45:2: error: use of unknown
builtin '__builtin_ia32_movntdq' [-Wimplicit-function-declaration]
        __builtin_ia32_movntdq((__v2di *)pp_addr + 0, (__v2di)src_data0);
        ^
../drivers/event/dlb2/dlb2_inline_fns.h:45:2: note: did you mean
'__builtin_ia32_movntq'?
/usr/lib/clang/10.0.1/include/xmmintrin.h:2122:3: note:
'__builtin_ia32_movntq' declared here
  __builtin_ia32_movntq(__p, __a);
  ^
In file included from ../drivers/event/dlb2/dlb2.c:35:
../drivers/event/dlb2/dlb2_inline_fns.h:61:2: error: use of unknown
builtin '__builtin_ia32_movntdq' [-Wimplicit-function-declaration]
        __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
        ^
2 errors generated.
[2125/2479] Generating rte_common_sfc_efx.sym_chk with a meson_exe.py
custom command
ninja: build stopped: subcommand failed.




> ---
>  drivers/event/dlb2/dlb2.c                      |  532 +++++
>  drivers/event/dlb2/dlb2_iface.c                |   28 +
>  drivers/event/dlb2/dlb2_iface.h                |   29 +
>  drivers/event/dlb2/meson.build                 |    6 +-
>  drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
>  drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
>  drivers/event/dlb2/pf/base/dlb2_osdep.h        |  247 +++
>  drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
>  drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
>  drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
>  drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
>  drivers/event/dlb2/pf/base/dlb2_resource.c     |  274 +++
>  drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
>  drivers/event/dlb2/pf/dlb2_main.c              |  615 ++++++
>  drivers/event/dlb2/pf/dlb2_main.h              |  106 +
>  drivers/event/dlb2/pf/dlb2_pf.c                |  244 +++
>  16 files changed, 8085 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/event/dlb2/dlb2.c
>  create mode 100644 drivers/event/dlb2/dlb2_iface.c
>  create mode 100644 drivers/event/dlb2/dlb2_iface.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
>  create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
>  create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
>  create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
>
> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> new file mode 100644
> index 0000000..26985b9
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2.c
> @@ -0,0 +1,532 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <assert.h>
> +#include <errno.h>
> +#include <nmmintrin.h>
> +#include <pthread.h>
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <sys/mman.h>
> +#include <sys/fcntl.h>
> +
> +#include <rte_common.h>
> +#include <rte_config.h>
> +#include <rte_cycles.h>
> +#include <rte_debug.h>
> +#include <rte_dev.h>
> +#include <rte_errno.h>
> +#include <rte_eventdev.h>
> +#include <rte_eventdev_pmd.h>
> +#include <rte_io.h>
> +#include <rte_kvargs.h>
> +#include <rte_log.h>
> +#include <rte_malloc.h>
> +#include <rte_mbuf.h>
> +#include <rte_prefetch.h>
> +#include <rte_ring.h>
> +#include <rte_string_fns.h>
> +
> +#include "dlb2_priv.h"
> +#include "dlb2_iface.h"
> +#include "dlb2_inline_fns.h"
> +
> +#if !defined RTE_ARCH_X86_64
> +#error "This implementation only supports RTE_ARCH_X86_64 architecture."
> +#endif
> +
> +/*
> + * Resources exposed to eventdev. Some values overridden at runtime using
> + * values returned by the DLB kernel driver.
> + */
> +#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
> +#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
> +#endif
> +static struct rte_event_dev_info evdev_dlb2_default_info = {
> +       .driver_name = "", /* probe will set */
> +       .min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
> +       .max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
> +#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
> +       .max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
> +#else
> +       .max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
> +#endif
> +       .max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
> +       .max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
> +       .max_event_priority_levels = DLB2_QID_PRIORITIES,
> +       .max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
> +       .max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
> +       .max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
> +       .max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
> +       .max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
> +       .max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
> +       .event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
> +                         RTE_EVENT_DEV_CAP_EVENT_QOS |
> +                         RTE_EVENT_DEV_CAP_BURST_MODE |
> +                         RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
> +                         RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
> +                         RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
> +};
> +
> +struct process_local_port_data
> +dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
> +
> +/* override defaults with value(s) provided on command line */
> +static void
> +dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
> +                                int *qid_depth_thresholds)
> +{
> +       int q;
> +
> +       for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
> +               if (qid_depth_thresholds[q] != 0)
> +                       dlb2->ev_queues[q].depth_threshold =
> +                               qid_depth_thresholds[q];
> +       }
> +}
> +
> +static int
> +dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
> +{
> +       struct dlb2_hw_dev *handle = &dlb2->qm_instance;
> +       struct dlb2_hw_resource_info *dlb2_info = &handle->info;
> +       int ret;
> +
> +       /* Query driver resources provisioned for this device */
> +
> +       ret = dlb2_iface_get_num_resources(handle,
> +                                          &dlb2->hw_rsrc_query_results);
> +       if (ret) {
> +               DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
> +               return ret;
> +       }
> +
> +       /* Complete filling in device resource info returned to evdev app,
> +        * overriding any default values.
> +        * The capabilities (CAPs) were set at compile time.
> +        */
> +
> +       evdev_dlb2_default_info.max_event_queues =
> +               dlb2->hw_rsrc_query_results.num_ldb_queues;
> +
> +       evdev_dlb2_default_info.max_event_ports =
> +               dlb2->hw_rsrc_query_results.num_ldb_ports;
> +
> +       evdev_dlb2_default_info.max_num_events =
> +               dlb2->hw_rsrc_query_results.num_ldb_credits;
> +
> +       /* Save off values used when creating the scheduling domain. */
> +
> +       handle->info.num_sched_domains =
> +               dlb2->hw_rsrc_query_results.num_sched_domains;
> +
> +       handle->info.hw_rsrc_max.nb_events_limit =
> +               dlb2->hw_rsrc_query_results.num_ldb_credits;
> +
> +       handle->info.hw_rsrc_max.num_queues =
> +               dlb2->hw_rsrc_query_results.num_ldb_queues +
> +               dlb2->hw_rsrc_query_results.num_dir_ports;
> +
> +       handle->info.hw_rsrc_max.num_ldb_queues =
> +               dlb2->hw_rsrc_query_results.num_ldb_queues;
> +
> +       handle->info.hw_rsrc_max.num_ldb_ports =
> +               dlb2->hw_rsrc_query_results.num_ldb_ports;
> +
> +       handle->info.hw_rsrc_max.num_dir_ports =
> +               dlb2->hw_rsrc_query_results.num_dir_ports;
> +
> +       handle->info.hw_rsrc_max.reorder_window_size =
> +               dlb2->hw_rsrc_query_results.num_hist_list_entries;
> +
> +       rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
> +
> +       return 0;
> +}
> +
> +#define RTE_BASE_10 10
> +
> +static int
> +dlb2_string_to_int(int *result, const char *str)
> +{
> +       long ret;
> +       char *endptr;
> +
> +       if (str == NULL || result == NULL)
> +               return -EINVAL;
> +
> +       errno = 0;
> +       ret = strtol(str, &endptr, RTE_BASE_10);
> +       if (errno)
> +               return -errno;
> +
> +       /* long int and int may be different width for some architectures */
> +       if (ret < INT_MIN || ret > INT_MAX || endptr == str)
> +               return -EINVAL;
> +
> +       *result = ret;
> +       return 0;
> +}
> +
> +static int
> +set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
> +{
> +       int *socket_id = opaque;
> +       int ret;
> +
> +       ret = dlb2_string_to_int(socket_id, value);
> +       if (ret < 0)
> +               return ret;
> +
> +       if (*socket_id > RTE_MAX_NUMA_NODES)
> +               return -EINVAL;
> +       return 0;
> +}
> +
> +static int
> +set_max_num_events(const char *key __rte_unused,
> +                  const char *value,
> +                  void *opaque)
> +{
> +       int *max_num_events = opaque;
> +       int ret;
> +
> +       if (value == NULL || opaque == NULL) {
> +               DLB2_LOG_ERR("NULL pointer\n");
> +               return -EINVAL;
> +       }
> +
> +       ret = dlb2_string_to_int(max_num_events, value);
> +       if (ret < 0)
> +               return ret;
> +
> +       if (*max_num_events < 0 || *max_num_events >
> +                       DLB2_MAX_NUM_LDB_CREDITS) {
> +               DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
> +                            DLB2_MAX_NUM_LDB_CREDITS);
> +               return -EINVAL;
> +       }
> +
> +       return 0;
> +}
> +
> +static int
> +set_num_dir_credits(const char *key __rte_unused,
> +                   const char *value,
> +                   void *opaque)
> +{
> +       int *num_dir_credits = opaque;
> +       int ret;
> +
> +       if (value == NULL || opaque == NULL) {
> +               DLB2_LOG_ERR("NULL pointer\n");
> +               return -EINVAL;
> +       }
> +
> +       ret = dlb2_string_to_int(num_dir_credits, value);
> +       if (ret < 0)
> +               return ret;
> +
> +       if (*num_dir_credits < 0 ||
> +           *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
> +               DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
> +                            DLB2_MAX_NUM_DIR_CREDITS);
> +               return -EINVAL;
> +       }
> +
> +       return 0;
> +}
> +
> +static int
> +set_dev_id(const char *key __rte_unused,
> +          const char *value,
> +          void *opaque)
> +{
> +       int *dev_id = opaque;
> +       int ret;
> +
> +       if (value == NULL || opaque == NULL) {
> +               DLB2_LOG_ERR("NULL pointer\n");
> +               return -EINVAL;
> +       }
> +
> +       ret = dlb2_string_to_int(dev_id, value);
> +       if (ret < 0)
> +               return ret;
> +
> +       return 0;
> +}
> +
> +static int
> +set_cos(const char *key __rte_unused,
> +       const char *value,
> +       void *opaque)
> +{
> +       enum dlb2_cos *cos_id = opaque;
> +       int x = 0;
> +       int ret;
> +
> +       if (value == NULL || opaque == NULL) {
> +               DLB2_LOG_ERR("NULL pointer\n");
> +               return -EINVAL;
> +       }
> +
> +       ret = dlb2_string_to_int(&x, value);
> +       if (ret < 0)
> +               return ret;
> +
> +       if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
> +               DLB2_LOG_ERR(
> +                       "COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
> +                       x);
> +               return -EINVAL;
> +       }
> +
> +       *cos_id = x;
> +
> +       return 0;
> +}
> +
> +
> +static int
> +set_qid_depth_thresh(const char *key __rte_unused,
> +                    const char *value,
> +                    void *opaque)
> +{
> +       struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
> +       int first, last, thresh, i;
> +
> +       if (value == NULL || opaque == NULL) {
> +               DLB2_LOG_ERR("NULL pointer\n");
> +               return -EINVAL;
> +       }
> +
> +       /* command line override may take one of the following 3 forms:
> +        * qid_depth_thresh=all:<threshold_value> ... all queues
> +        * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
> +        * qid_depth_thresh=qid:<threshold_value> ... just one queue
> +        */
> +       if (sscanf(value, "all:%d", &thresh) == 1) {
> +               first = 0;
> +               last = DLB2_MAX_NUM_QUEUES - 1;
> +       } else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
> +               /* we have everything we need */
> +       } else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
> +               last = first;
> +       } else {
> +               DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val, qid-qid:val, or qid:val\n");
> +               return -EINVAL;
> +       }
> +
> +       if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
> +               DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
> +               return -EINVAL;
> +       }
> +
> +       if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
> +               DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
> +                            DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
> +               return -EINVAL;
> +       }
> +
> +       for (i = first; i <= last; i++)
> +               qid_thresh->val[i] = thresh; /* indexed by qid */
> +
> +       return 0;
> +}
> +
> +static void
> +dlb2_entry_points_init(struct rte_eventdev *dev)
> +{
> +       RTE_SET_USED(dev);
> +
> +       /* Eventdev PMD entry points */
> +}
> +
> +int
> +dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
> +                           const char *name,
> +                           struct dlb2_devargs *dlb2_args)
> +{
> +       struct dlb2_eventdev *dlb2;
> +       int err;
> +
> +       dlb2 = dev->data->dev_private;
> +
> +       dlb2->event_dev = dev; /* backlink */
> +
> +       evdev_dlb2_default_info.driver_name = name;
> +
> +       dlb2->max_num_events_override = dlb2_args->max_num_events;
> +       dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
> +       dlb2->qm_instance.cos_id = dlb2_args->cos_id;
> +
> +       err = dlb2_iface_open(&dlb2->qm_instance, name);
> +       if (err < 0) {
> +               DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
> +                            err);
> +               return err;
> +       }
> +
> +       err = dlb2_iface_get_device_version(&dlb2->qm_instance,
> +                                           &dlb2->revision);
> +       if (err < 0) {
> +               DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
> +                            err);
> +               return err;
> +       }
> +
> +       err = dlb2_hw_query_resources(dlb2);
> +       if (err) {
> +               DLB2_LOG_ERR("get resources err=%d for %s\n",
> +                            err, name);
> +               return err;
> +       }
> +
> +       dlb2_iface_hardware_init(&dlb2->qm_instance);
> +
> +       err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
> +       if (err < 0) {
> +               DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
> +                            err);
> +               return err;
> +       }
> +
> +       rte_spinlock_init(&dlb2->qm_instance.resource_lock);
> +
> +       dlb2_iface_low_level_io_init();
> +
> +       dlb2_entry_points_init(dev);
> +
> +       dlb2_init_queue_depth_thresholds(dlb2,
> +                                        dlb2_args->qid_depth_thresholds.val);
> +
> +       return 0;
> +}
> +
> +int
> +dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
> +                             const char *name)
> +{
> +       struct dlb2_eventdev *dlb2;
> +       int err;
> +
> +       dlb2 = dev->data->dev_private;
> +
> +       evdev_dlb2_default_info.driver_name = name;
> +
> +       err = dlb2_iface_open(&dlb2->qm_instance, name);
> +       if (err < 0) {
> +               DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
> +                            err);
> +               return err;
> +       }
> +
> +       err = dlb2_hw_query_resources(dlb2);
> +       if (err) {
> +               DLB2_LOG_ERR("get resources err=%d for %s\n",
> +                            err, name);
> +               return err;
> +       }
> +
> +       dlb2_iface_low_level_io_init();
> +
> +       dlb2_entry_points_init(dev);
> +
> +       return 0;
> +}
> +
> +int
> +dlb2_parse_params(const char *params,
> +                 const char *name,
> +                 struct dlb2_devargs *dlb2_args)
> +{
> +       int ret = 0;
> +       static const char * const args[] = { NUMA_NODE_ARG,
> +                                            DLB2_MAX_NUM_EVENTS,
> +                                            DLB2_NUM_DIR_CREDITS,
> +                                            DEV_ID_ARG,
> +                                            DLB2_QID_DEPTH_THRESH_ARG,
> +                                            DLB2_COS_ARG,
> +                                            NULL };
> +
> +       if (params != NULL && params[0] != '\0') {
> +               struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
> +
> +               if (kvlist == NULL) {
> +                       RTE_LOG(INFO, PMD,
> +                               "Ignoring unsupported parameters when creating device '%s'\n",
> +                               name);
> +               } else {
> +                       int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
> +                                                    set_numa_node,
> +                                                    &dlb2_args->socket_id);
> +                       if (ret != 0) {
> +                               DLB2_LOG_ERR("%s: Error parsing numa node parameter",
> +                                            name);
> +                               rte_kvargs_free(kvlist);
> +                               return ret;
> +                       }
> +
> +                       ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
> +                                                set_max_num_events,
> +                                                &dlb2_args->max_num_events);
> +                       if (ret != 0) {
> +                               DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
> +                                            name);
> +                               rte_kvargs_free(kvlist);
> +                               return ret;
> +                       }
> +
> +                       ret = rte_kvargs_process(kvlist,
> +                                       DLB2_NUM_DIR_CREDITS,
> +                                       set_num_dir_credits,
> +                                       &dlb2_args->num_dir_credits_override);
> +                       if (ret != 0) {
> +                               DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
> +                                            name);
> +                               rte_kvargs_free(kvlist);
> +                               return ret;
> +                       }
> +
> +                       ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
> +                                                set_dev_id,
> +                                                &dlb2_args->dev_id);
> +                       if (ret != 0) {
> +                               DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
> +                                            name);
> +                               rte_kvargs_free(kvlist);
> +                               return ret;
> +                       }
> +
> +                       ret = rte_kvargs_process(
> +                                       kvlist,
> +                                       DLB2_QID_DEPTH_THRESH_ARG,
> +                                       set_qid_depth_thresh,
> +                                       &dlb2_args->qid_depth_thresholds);
> +                       if (ret != 0) {
> +                               DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
> +                                            name);
> +                               rte_kvargs_free(kvlist);
> +                               return ret;
> +                       }
> +
> +                       ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
> +                                                set_cos,
> +                                                &dlb2_args->cos_id);
> +                       if (ret != 0) {
> +                               DLB2_LOG_ERR("%s: Error parsing cos parameter",
> +                                            name);
> +                               rte_kvargs_free(kvlist);
> +                               return ret;
> +                       }
> +
> +                       rte_kvargs_free(kvlist);
> +               }
> +       }
> +       return ret;
> +}
> +RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
> diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
> new file mode 100644
> index 0000000..0d93faf
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2_iface.c
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <stdint.h>
> +
> +#include "dlb2_priv.h"
> +
> +/* DLB2 PMD Internal interface function pointers.
> + * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
> + * serviced by DLB kernel module.
> + * If PCI (PF PMD),  these will be implemented locally in user mode.
> + */
> +
> +void (*dlb2_iface_low_level_io_init)(void);
> +
> +int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
> +
> +int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
> +                                    uint8_t *revision);
> +
> +void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
> +
> +int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
> +                                  enum dlb2_cq_poll_modes *mode);
> +
> +int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
> +                                   struct dlb2_get_num_resources_args *rsrcs);
> diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
> new file mode 100644
> index 0000000..4fb416e
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2_iface.h
> @@ -0,0 +1,29 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef _DLB2_IFACE_H_
> +#define _DLB2_IFACE_H_
> +
> +/* DLB2 PMD Internal interface function pointers.
> + * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
> + * serviced by DLB kernel module.
> + * If PCI (PF PMD),  these will be implemented locally in user mode.
> + */
> +
> +extern void (*dlb2_iface_low_level_io_init)(void);
> +
> +extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
> +
> +extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
> +                                           uint8_t *revision);
> +
> +extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
> +
> +extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
> +                                         enum dlb2_cq_poll_modes *mode);
> +
> +extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
> +                               struct dlb2_get_num_resources_args *rsrcs);
> +
> +#endif /* _DLB2_IFACE_H_ */
> diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
> index 54ba2c8..99b71f9 100644
> --- a/drivers/event/dlb2/meson.build
> +++ b/drivers/event/dlb2/meson.build
> @@ -1,7 +1,11 @@
>  # SPDX-License-Identifier: BSD-3-Clause
>  # Copyright(c) 2019-2020 Intel Corporation
>
> -sources = files(
> +sources = files('dlb2.c',
> +               'dlb2_iface.c',
> +               'pf/dlb2_main.c',
> +               'pf/dlb2_pf.c',
> +               'pf/base/dlb2_resource.c'
>  )
>
>  deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
> diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
> new file mode 100644
> index 0000000..428a5e8
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
> @@ -0,0 +1,367 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_HW_TYPES_H
> +#define __DLB2_HW_TYPES_H
> +
> +#include "dlb2_user.h"
> +
> +#include "dlb2_osdep_list.h"
> +#include "dlb2_osdep_types.h"
> +
> +#define DLB2_MAX_NUM_VDEVS                     16
> +#define DLB2_MAX_NUM_DOMAINS                   32
> +#define DLB2_MAX_NUM_LDB_QUEUES                        32 /* LDB == load-balanced */
> +#define DLB2_MAX_NUM_DIR_QUEUES                        64 /* DIR == directed */
> +#define DLB2_MAX_NUM_LDB_PORTS                 64
> +#define DLB2_MAX_NUM_DIR_PORTS                 64
> +#define DLB2_MAX_NUM_LDB_CREDITS               (8 * 1024)
> +#define DLB2_MAX_NUM_DIR_CREDITS               (2 * 1024)
> +#define DLB2_MAX_NUM_HIST_LIST_ENTRIES         2048
> +#define DLB2_MAX_NUM_AQED_ENTRIES              2048
> +#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ           8
> +#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS    2
> +#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES     5
> +#define DLB2_QID_PRIORITIES                    8
> +#define DLB2_NUM_ARB_WEIGHTS                   8
> +#define DLB2_MAX_WEIGHT                                255
> +#define DLB2_NUM_COS_DOMAINS                   4
> +#define DLB2_MAX_CQ_COMP_CHECK_LOOPS           409600
> +#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS         (32 * 64 * 1024 * (800 / 30))
> +#ifdef FPGA
> +#define DLB2_HZ                                        2000000
> +#else
> +#define DLB2_HZ                                        800000000
> +#endif
> +
> +#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
> +#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
> +
> +/* Interrupt related macros */
> +#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
> +#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
> +#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
> +       (DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
> +        DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
> +#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
> +       (DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
> +#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
> +       DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
> +#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
> +       DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
> +
> +/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
> +#define DLB2_INT_NON_CQ 0
> +
> +#define DLB2_ALARM_HW_SOURCE_SYS 0
> +#define DLB2_ALARM_HW_SOURCE_DLB 1
> +
> +#define DLB2_ALARM_HW_UNIT_CHP 4
> +
> +#define DLB2_ALARM_SYS_AID_ILLEGAL_QID         3
> +#define DLB2_ALARM_SYS_AID_DISABLED_QID                4
> +#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW         5
> +#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ      1
> +#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
> +
> +#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
> +#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
> +#define DLB2_VF_BASE_CQ_VECTOR_ID           0
> +#define DLB2_VF_LAST_CQ_VECTOR_ID           30
> +#define DLB2_VF_MBOX_VECTOR_ID              31
> +#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
> +       (DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
> +        DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
> +
> +#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
> +                                            DLB2_MAX_NUM_DIR_PORTS + 1)
> +
> +/*
> + * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
> + * the PF driver.
> + */
> +#define DLB2_DRV_LDB_PP_BASE   0x2300000
> +#define DLB2_DRV_LDB_PP_STRIDE 0x1000
> +#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
> +                               DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
> +#define DLB2_DRV_DIR_PP_BASE   0x2200000
> +#define DLB2_DRV_DIR_PP_STRIDE 0x1000
> +#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
> +                               DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
> +#define DLB2_LDB_PP_BASE       0x2100000
> +#define DLB2_LDB_PP_STRIDE     0x1000
> +#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
> +                               DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
> +#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
> +#define DLB2_DIR_PP_BASE       0x2000000
> +#define DLB2_DIR_PP_STRIDE     0x1000
> +#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
> +                               DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
> +#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
> +
> +struct dlb2_resource_id {
> +       u32 phys_id;
> +       u32 virt_id;
> +       u8 vdev_owned;
> +       u8 vdev_id;
> +};
> +
> +struct dlb2_freelist {
> +       u32 base;
> +       u32 bound;
> +       u32 offset;
> +};
> +
> +static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
> +{
> +       return list->bound - list->base - list->offset;
> +}
> +
> +struct dlb2_hcw {
> +       u64 data;
> +       /* Word 3 */
> +       u16 opaque;
> +       u8 qid;
> +       u8 sched_type:2;
> +       u8 priority:3;
> +       u8 msg_type:3;
> +       /* Word 4 */
> +       u16 lock_id;
> +       u8 ts_flag:1;
> +       u8 rsvd1:2;
> +       u8 no_dec:1;
> +       u8 cmp_id:4;
> +       u8 cq_token:1;
> +       u8 qe_comp:1;
> +       u8 qe_frag:1;
> +       u8 qe_valid:1;
> +       u8 int_arm:1;
> +       u8 error:1;
> +       u8 rsvd:2;
> +};
> +
> +struct dlb2_ldb_queue {
> +       struct dlb2_list_entry domain_list;
> +       struct dlb2_list_entry func_list;
> +       struct dlb2_resource_id id;
> +       struct dlb2_resource_id domain_id;
> +       u32 num_qid_inflights;
> +       u32 aqed_limit;
> +       u32 sn_group; /* sn == sequence number */
> +       u32 sn_slot;
> +       u32 num_mappings;
> +       u8 sn_cfg_valid;
> +       u8 num_pending_additions;
> +       u8 owned;
> +       u8 configured;
> +};
> +
> +/*
> + * Directed ports and queues are paired by nature, so the driver tracks them
> + * with a single data structure.
> + */
> +struct dlb2_dir_pq_pair {
> +       struct dlb2_list_entry domain_list;
> +       struct dlb2_list_entry func_list;
> +       struct dlb2_resource_id id;
> +       struct dlb2_resource_id domain_id;
> +       u32 ref_cnt;
> +       u8 init_tkn_cnt;
> +       u8 queue_configured;
> +       u8 port_configured;
> +       u8 owned;
> +       u8 enabled;
> +};
> +
> +enum dlb2_qid_map_state {
> +       /* The slot doesn't contain a valid queue mapping */
> +       DLB2_QUEUE_UNMAPPED,
> +       /* The slot contains a valid queue mapping */
> +       DLB2_QUEUE_MAPPED,
> +       /* The driver is mapping a queue into this slot */
> +       DLB2_QUEUE_MAP_IN_PROG,
> +       /* The driver is unmapping a queue from this slot */
> +       DLB2_QUEUE_UNMAP_IN_PROG,
> +       /*
> +        * The driver is unmapping a queue from this slot, and once complete
> +        * will replace it with another mapping.
> +        */
> +       DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
> +};
> +
> +struct dlb2_ldb_port_qid_map {
> +       enum dlb2_qid_map_state state;
> +       u16 qid;
> +       u16 pending_qid;
> +       u8 priority;
> +       u8 pending_priority;
> +};
> +
> +struct dlb2_ldb_port {
> +       struct dlb2_list_entry domain_list;
> +       struct dlb2_list_entry func_list;
> +       struct dlb2_resource_id id;
> +       struct dlb2_resource_id domain_id;
> +       /* The qid_map represents the hardware QID mapping state. */
> +       struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
> +       u32 hist_list_entry_base;
> +       u32 hist_list_entry_limit;
> +       u32 ref_cnt;
> +       u8 init_tkn_cnt;
> +       u8 num_pending_removals;
> +       u8 num_mappings;
> +       u8 owned;
> +       u8 enabled;
> +       u8 configured;
> +};
> +
> +struct dlb2_sn_group {
> +       u32 mode;
> +       u32 sequence_numbers_per_queue;
> +       u32 slot_use_bitmap;
> +       u32 id;
> +};
> +
> +static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
> +{
> +       u32 mask[] = {
> +               0x0000ffff,  /* 64 SNs per queue */
> +               0x000000ff,  /* 128 SNs per queue */
> +               0x0000000f,  /* 256 SNs per queue */
> +               0x00000003,  /* 512 SNs per queue */
> +               0x00000001}; /* 1024 SNs per queue */
> +
> +       return group->slot_use_bitmap == mask[group->mode];
> +}
> +
> +static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
> +{
> +       u32 bound[6] = {16, 8, 4, 2, 1};
> +       u32 i;
> +
> +       for (i = 0; i < bound[group->mode]; i++) {
> +               if (!(group->slot_use_bitmap & (1 << i))) {
> +                       group->slot_use_bitmap |= 1 << i;
> +                       return i;
> +               }
> +       }
> +
> +       return -1;
> +}
> +
> +static inline void
> +dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
> +{
> +       group->slot_use_bitmap &= ~(1 << slot);
> +}
> +
> +static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
> +{
> +       int i, cnt = 0;
> +
> +       for (i = 0; i < 32; i++)
> +               cnt += !!(group->slot_use_bitmap & (1 << i));
> +
> +       return cnt;
> +}
> +
> +struct dlb2_hw_domain {
> +       struct dlb2_function_resources *parent_func;
> +       struct dlb2_list_entry func_list;
> +       struct dlb2_list_head used_ldb_queues;
> +       struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
> +       struct dlb2_list_head used_dir_pq_pairs;
> +       struct dlb2_list_head avail_ldb_queues;
> +       struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
> +       struct dlb2_list_head avail_dir_pq_pairs;
> +       u32 total_hist_list_entries;
> +       u32 avail_hist_list_entries;
> +       u32 hist_list_entry_base;
> +       u32 hist_list_entry_offset;
> +       u32 num_ldb_credits;
> +       u32 num_dir_credits;
> +       u32 num_avail_aqed_entries;
> +       u32 num_used_aqed_entries;
> +       struct dlb2_resource_id id;
> +       int num_pending_removals;
> +       int num_pending_additions;
> +       u8 configured;
> +       u8 started;
> +};
> +
> +struct dlb2_bitmap;
> +
> +struct dlb2_function_resources {
> +       struct dlb2_list_head avail_domains;
> +       struct dlb2_list_head used_domains;
> +       struct dlb2_list_head avail_ldb_queues;
> +       struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
> +       struct dlb2_list_head avail_dir_pq_pairs;
> +       struct dlb2_bitmap *avail_hist_list_entries;
> +       u32 num_avail_domains;
> +       u32 num_avail_ldb_queues;
> +       u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
> +       u32 num_avail_dir_pq_pairs;
> +       u32 num_avail_qed_entries;
> +       u32 num_avail_dqed_entries;
> +       u32 num_avail_aqed_entries;
> +       u8 locked; /* (VDEV only) */
> +};
> +
> +/*
> + * After initialization, each resource in dlb2_hw_resources is located in one
> + * of the following lists:
> + * -- The PF's available resources list. These are unconfigured resources owned
> + *     by the PF and not allocated to a dlb2 scheduling domain.
> + * -- A VDEV's available resources list. These are VDEV-owned unconfigured
> + *     resources not allocated to a dlb2 scheduling domain.
> + * -- A domain's available resources list. These are domain-owned unconfigured
> + *     resources.
> + * -- A domain's used resources list. These are are domain-owned configured
> + *     resources.
> + *
> + * A resource moves to a new list when a VDEV or domain is created or destroyed,
> + * or when the resource is configured.
> + */
> +struct dlb2_hw_resources {
> +       struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
> +       struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
> +       struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
> +       struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
> +};
> +
> +struct dlb2_mbox {
> +       u32 *mbox;
> +       u32 *isr_in_progress;
> +};
> +
> +struct dlb2_sw_mbox {
> +       struct dlb2_mbox vdev_to_pf;
> +       struct dlb2_mbox pf_to_vdev;
> +       void (*pf_to_vdev_inject)(void *arg);
> +       void *pf_to_vdev_inject_arg;
> +};
> +
> +struct dlb2_hw {
> +       /* BAR 0 address */
> +       void  *csr_kva;
> +       unsigned long csr_phys_addr;
> +       /* BAR 2 address */
> +       void  *func_kva;
> +       unsigned long func_phys_addr;
> +
> +       /* Resource tracking */
> +       struct dlb2_hw_resources rsrcs;
> +       struct dlb2_function_resources pf;
> +       struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
> +       struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
> +       u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
> +
> +       /* Virtualization */
> +       int virt_mode;
> +       struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
> +       unsigned int pasid[DLB2_MAX_NUM_VDEVS];
> +};
> +
> +#endif /* __DLB2_HW_TYPES_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
> new file mode 100644
> index 0000000..ce462c0
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
> @@ -0,0 +1,596 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_BASE_DLB2_MBOX_H
> +#define __DLB2_BASE_DLB2_MBOX_H
> +
> +#include "dlb2_osdep_types.h"
> +#include "dlb2_regs.h"
> +
> +#define DLB2_MBOX_INTERFACE_VERSION 1
> +
> +/*
> + * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
> + * to send requests of its own (e.g. notifying a VF of an impending FLR).
> + * To avoid communication race conditions, e.g. the PF sends a response and then
> + * sends a request before the VF reads the response, the PF->VF mailbox is
> + * divided into two sections:
> + * - Bytes 0-47: PF responses
> + * - Bytes 48-63: PF requests
> + *
> + * Partitioning the PF->VF mailbox allows responses and requests to occupy the
> + * mailbox simultaneously.
> + */
> +#define DLB2_PF2VF_RESP_BYTES    48
> +#define DLB2_PF2VF_RESP_BASE     0
> +#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
> +
> +#define DLB2_PF2VF_REQ_BYTES     16
> +#define DLB2_PF2VF_REQ_BASE      (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
> +#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
> +
> +/*
> + * Similarly, the VF->PF mailbox is divided into two sections:
> + * - Bytes 0-239: VF requests
> + * -- (Bytes 0-3 are unused due to a hardware errata)
> + * - Bytes 240-255: VF responses
> + */
> +#define DLB2_VF2PF_REQ_BYTES    236
> +#define DLB2_VF2PF_REQ_BASE     4
> +#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
> +
> +#define DLB2_VF2PF_RESP_BYTES    16
> +#define DLB2_VF2PF_RESP_BASE     (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
> +#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
> +
> +/* VF-initiated commands */
> +enum dlb2_mbox_cmd_type {
> +       DLB2_MBOX_CMD_REGISTER,
> +       DLB2_MBOX_CMD_UNREGISTER,
> +       DLB2_MBOX_CMD_GET_NUM_RESOURCES,
> +       DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
> +       DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
> +       DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
> +       DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
> +       DLB2_MBOX_CMD_CREATE_LDB_PORT,
> +       DLB2_MBOX_CMD_CREATE_DIR_PORT,
> +       DLB2_MBOX_CMD_ENABLE_LDB_PORT,
> +       DLB2_MBOX_CMD_DISABLE_LDB_PORT,
> +       DLB2_MBOX_CMD_ENABLE_DIR_PORT,
> +       DLB2_MBOX_CMD_DISABLE_DIR_PORT,
> +       DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
> +       DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
> +       DLB2_MBOX_CMD_MAP_QID,
> +       DLB2_MBOX_CMD_UNMAP_QID,
> +       DLB2_MBOX_CMD_START_DOMAIN,
> +       DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
> +       DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
> +       DLB2_MBOX_CMD_ARM_CQ_INTR,
> +       DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
> +       DLB2_MBOX_CMD_GET_SN_ALLOCATION,
> +       DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
> +       DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
> +       DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
> +       DLB2_MBOX_CMD_GET_COS_BW,
> +       DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
> +       DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
> +
> +       /* NUM_QE_CMD_TYPES must be last */
> +       NUM_DLB2_MBOX_CMD_TYPES,
> +};
> +
> +static const char dlb2_mbox_cmd_type_strings[][128] = {
> +       "DLB2_MBOX_CMD_REGISTER",
> +       "DLB2_MBOX_CMD_UNREGISTER",
> +       "DLB2_MBOX_CMD_GET_NUM_RESOURCES",
> +       "DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
> +       "DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
> +       "DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
> +       "DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
> +       "DLB2_MBOX_CMD_CREATE_LDB_PORT",
> +       "DLB2_MBOX_CMD_CREATE_DIR_PORT",
> +       "DLB2_MBOX_CMD_ENABLE_LDB_PORT",
> +       "DLB2_MBOX_CMD_DISABLE_LDB_PORT",
> +       "DLB2_MBOX_CMD_ENABLE_DIR_PORT",
> +       "DLB2_MBOX_CMD_DISABLE_DIR_PORT",
> +       "DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
> +       "DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
> +       "DLB2_MBOX_CMD_MAP_QID",
> +       "DLB2_MBOX_CMD_UNMAP_QID",
> +       "DLB2_MBOX_CMD_START_DOMAIN",
> +       "DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
> +       "DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
> +       "DLB2_MBOX_CMD_ARM_CQ_INTR",
> +       "DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
> +       "DLB2_MBOX_CMD_GET_SN_ALLOCATION",
> +       "DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
> +       "DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
> +       "DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
> +       "DLB2_MBOX_CMD_GET_COS_BW",
> +       "DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
> +       "DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
> +};
> +
> +/* PF-initiated commands */
> +enum dlb2_mbox_vf_cmd_type {
> +       DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
> +       DLB2_MBOX_VF_CMD_NOTIFICATION,
> +       DLB2_MBOX_VF_CMD_IN_USE,
> +
> +       /* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
> +       NUM_DLB2_MBOX_VF_CMD_TYPES,
> +};
> +
> +static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
> +       "DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
> +       "DLB2_MBOX_VF_CMD_NOTIFICATION",
> +       "DLB2_MBOX_VF_CMD_IN_USE",
> +};
> +
> +#define DLB2_MBOX_CMD_TYPE(hdr) \
> +       (((struct dlb2_mbox_req_hdr *)hdr)->type)
> +#define DLB2_MBOX_CMD_STRING(hdr) \
> +       dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
> +
> +enum dlb2_mbox_status_type {
> +       DLB2_MBOX_ST_SUCCESS,
> +       DLB2_MBOX_ST_INVALID_CMD_TYPE,
> +       DLB2_MBOX_ST_VERSION_MISMATCH,
> +       DLB2_MBOX_ST_INVALID_OWNER_VF,
> +};
> +
> +static const char dlb2_mbox_status_type_strings[][128] = {
> +       "DLB2_MBOX_ST_SUCCESS",
> +       "DLB2_MBOX_ST_INVALID_CMD_TYPE",
> +       "DLB2_MBOX_ST_VERSION_MISMATCH",
> +       "DLB2_MBOX_ST_INVALID_OWNER_VF",
> +};
> +
> +#define DLB2_MBOX_ST_TYPE(hdr) \
> +       (((struct dlb2_mbox_resp_hdr *)hdr)->status)
> +#define DLB2_MBOX_ST_STRING(hdr) \
> +       dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
> +
> +/* This structure is always the first field in a request structure */
> +struct dlb2_mbox_req_hdr {
> +       u32 type;
> +};
> +
> +/* This structure is always the first field in a response structure */
> +struct dlb2_mbox_resp_hdr {
> +       u32 status;
> +};
> +
> +struct dlb2_mbox_register_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u16 min_interface_version;
> +       u16 max_interface_version;
> +};
> +
> +struct dlb2_mbox_register_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 interface_version;
> +       u8 pf_id;
> +       u8 vf_id;
> +       u8 is_auxiliary_vf;
> +       u8 primary_vf_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_unregister_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_unregister_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_get_num_resources_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_get_num_resources_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u16 num_sched_domains;
> +       u16 num_ldb_queues;
> +       u16 num_ldb_ports;
> +       u16 num_cos_ldb_ports[4];
> +       u16 num_dir_ports;
> +       u32 num_atomic_inflights;
> +       u32 num_hist_list_entries;
> +       u32 max_contiguous_hist_list_entries;
> +       u16 num_ldb_credits;
> +       u16 num_dir_credits;
> +};
> +
> +struct dlb2_mbox_create_sched_domain_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 num_ldb_queues;
> +       u32 num_ldb_ports;
> +       u32 num_cos_ldb_ports[4];
> +       u32 num_dir_ports;
> +       u32 num_atomic_inflights;
> +       u32 num_hist_list_entries;
> +       u32 num_ldb_credits;
> +       u32 num_dir_credits;
> +       u8 cos_strict;
> +       u8 padding0[3];
> +       u32 padding1;
> +};
> +
> +struct dlb2_mbox_create_sched_domain_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_reset_sched_domain_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_reset_sched_domain_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +};
> +
> +struct dlb2_mbox_create_ldb_queue_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 num_sequence_numbers;
> +       u32 num_qid_inflights;
> +       u32 num_atomic_inflights;
> +       u32 lock_id_comp_level;
> +       u32 depth_threshold;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_create_ldb_queue_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_create_dir_queue_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 depth_threshold;
> +};
> +
> +struct dlb2_mbox_create_dir_queue_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_create_ldb_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u16 cq_depth;
> +       u16 cq_history_list_size;
> +       u8 cos_id;
> +       u8 cos_strict;
> +       u16 padding1;
> +       u64 cq_base_address;
> +};
> +
> +struct dlb2_mbox_create_ldb_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_create_dir_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u64 cq_base_address;
> +       u16 cq_depth;
> +       u16 padding0;
> +       s32 queue_id;
> +};
> +
> +struct dlb2_mbox_create_dir_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_enable_ldb_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_enable_ldb_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_disable_ldb_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_disable_ldb_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_enable_dir_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_enable_dir_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_disable_dir_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_disable_dir_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       s32 owned;
> +};
> +
> +struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       s32 owned;
> +};
> +
> +struct dlb2_mbox_map_qid_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 qid;
> +       u32 priority;
> +       u32 padding0;
> +};
> +
> +struct dlb2_mbox_map_qid_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_unmap_qid_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 qid;
> +};
> +
> +struct dlb2_mbox_unmap_qid_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_start_domain_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +};
> +
> +struct dlb2_mbox_start_domain_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u16 port_id;
> +       u16 thresh;
> +       u16 vector;
> +       u16 owner_vf;
> +       u16 reserved[2];
> +};
> +
> +struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_enable_dir_port_intr_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u16 port_id;
> +       u16 thresh;
> +       u16 vector;
> +       u16 owner_vf;
> +       u16 reserved[2];
> +};
> +
> +struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_arm_cq_intr_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 is_ldb;
> +};
> +
> +struct dlb2_mbox_arm_cq_intr_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding0;
> +};
> +
> +/*
> + * The alert_id and aux_alert_data follows the format of the alerts defined in
> + * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
> + * the aux_alert_data value varies depending on the alert.
> + */
> +struct dlb2_mbox_vf_alert_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 alert_id;
> +       u32 aux_alert_data;
> +};
> +
> +enum dlb2_mbox_vf_notification_type {
> +       DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
> +       DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
> +
> +       /* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
> +       NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
> +};
> +
> +struct dlb2_mbox_vf_notification_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 notification;
> +};
> +
> +struct dlb2_mbox_vf_in_use_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_vf_in_use_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 in_use;
> +};
> +
> +struct dlb2_mbox_get_sn_allocation_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 group_id;
> +};
> +
> +struct dlb2_mbox_get_sn_allocation_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 num;
> +};
> +
> +struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 queue_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 depth;
> +};
> +
> +struct dlb2_mbox_get_dir_queue_depth_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 queue_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 depth;
> +};
> +
> +struct dlb2_mbox_pending_port_unmaps_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_pending_port_unmaps_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 num;
> +};
> +
> +struct dlb2_mbox_get_cos_bw_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 cos_id;
> +};
> +
> +struct dlb2_mbox_get_cos_bw_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 num;
> +};
> +
> +struct dlb2_mbox_get_sn_occupancy_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 group_id;
> +};
> +
> +struct dlb2_mbox_get_sn_occupancy_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 num;
> +};
> +
> +struct dlb2_mbox_query_cq_poll_mode_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 mode;
> +};
> +
> +#endif /* __DLB2_BASE_DLB2_MBOX_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
> new file mode 100644
> index 0000000..43f2125
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
> @@ -0,0 +1,247 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_OSDEP_H
> +#define __DLB2_OSDEP_H
> +
> +#include <string.h>
> +#include <time.h>
> +#include <unistd.h>
> +#include <pthread.h>
> +
> +#include <rte_string_fns.h>
> +#include <rte_cycles.h>
> +#include <rte_io.h>
> +#include <rte_log.h>
> +#include <rte_spinlock.h>
> +#include "../dlb2_main.h"
> +#include "dlb2_resource.h"
> +#include "../../dlb2_log.h"
> +#include "../../dlb2_user.h"
> +
> +
> +#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
> +#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
> +
> +/* Read/write register 'reg' in the CSR BAR space */
> +#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
> +#define DLB2_CSR_RD(hw, reg) \
> +       DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
> +#define DLB2_CSR_WR(hw, reg, value) \
> +       DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
> +
> +/* Read/write register 'reg' in the func BAR space */
> +#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
> +#define DLB2_FUNC_RD(hw, reg) \
> +       DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
> +#define DLB2_FUNC_WR(hw, reg, value) \
> +       DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
> +
> +/* Map to PMDs logging interface */
> +#define DLB2_ERR(dev, fmt, args...) \
> +       DLB2_LOG_ERR(fmt, ## args)
> +
> +#define DLB2_INFO(dev, fmt, args...) \
> +       DLB2_LOG_INFO(fmt, ## args)
> +
> +#define DLB2_DEBUG(dev, fmt, args...) \
> +       DLB2_LOG_DBG(fmt, ## args)
> +
> +/**
> + * os_udelay() - busy-wait for a number of microseconds
> + * @usecs: delay duration.
> + */
> +static inline void os_udelay(int usecs)
> +{
> +       rte_delay_us(usecs);
> +}
> +
> +/**
> + * os_msleep() - sleep for a number of milliseconds
> + * @usecs: delay duration.
> + */
> +static inline void os_msleep(int msecs)
> +{
> +       rte_delay_ms(msecs);
> +}
> +
> +#define DLB2_PP_BASE(__is_ldb) \
> +       ((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
> +
> +/**
> + * os_map_producer_port() - map a producer port into the caller's address space
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: port ID
> + * @is_ldb: true for load-balanced port, false for a directed port
> + *
> + * This function maps the requested producer port memory into the caller's
> + * address space.
> + *
> + * Return:
> + * Returns the base address at which the PP memory was mapped, else NULL.
> + */
> +static inline void *os_map_producer_port(struct dlb2_hw *hw,
> +                                        u8 port_id,
> +                                        bool is_ldb)
> +{
> +       uint64_t addr;
> +       uint64_t pp_dma_base;
> +
> +       pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
> +       addr = (pp_dma_base + (PAGE_SIZE * port_id));
> +
> +       return (void *)(uintptr_t)addr;
> +}
> +
> +/**
> + * os_unmap_producer_port() - unmap a producer port
> + * @addr: mapped producer port address
> + *
> + * This function undoes os_map_producer_port() by unmapping the producer port
> + * memory from the caller's address space.
> + *
> + * Return:
> + * Returns the base address at which the PP memory was mapped, else NULL.
> + */
> +static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
> +{
> +       RTE_SET_USED(hw);
> +       RTE_SET_USED(addr);
> +}
> +
> +/**
> + * os_fence_hcw() - fence an HCW to ensure it arrives at the device
> + * @hw: dlb2_hw handle for a particular device.
> + * @pp_addr: producer port address
> + */
> +static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
> +{
> +       RTE_SET_USED(hw);
> +
> +       /* To ensure outstanding HCWs reach the device, read the PP address. IA
> +        * memory ordering prevents reads from passing older writes, and the
> +        * mfence also ensures this.
> +        */
> +       rte_mb();
> +
> +       *(volatile u64 *)pp_addr;
> +}
> +
> +/**
> + * os_enqueue_four_hcws() - enqueue four HCWs to DLB
> + * @hw: dlb2_hw handle for a particular device.
> + * @hcw: pointer to the 64B-aligned contiguous HCW memory
> + * @addr: producer port address
> + */
> +static inline void os_enqueue_four_hcws(struct dlb2_hw *hw,
> +                                       struct dlb2_hcw *hcw,
> +                                       void *addr)
> +{
> +       struct dlb2_dev *dlb2_dev;
> +
> +       dlb2_dev = container_of(hw, struct dlb2_dev, hw);
> +
> +       dlb2_dev->enqueue_four(hcw, addr);
> +}
> +
> +/**
> + * DLB2_HW_ERR() - log an error message
> + * @dlb2: dlb2_hw handle for a particular device.
> + * @...: variable string args.
> + */
> +#define DLB2_HW_ERR(dlb2, ...) do {    \
> +       RTE_SET_USED(dlb2);             \
> +       DLB2_ERR(dlb2, __VA_ARGS__);    \
> +} while (0)
> +
> +/**
> + * DLB2_HW_DBG() - log an info message
> + * @dlb2: dlb2_hw handle for a particular device.
> + * @...: variable string args.
> + */
> +#define DLB2_HW_DBG(dlb2, ...) do {    \
> +       RTE_SET_USED(dlb2);             \
> +       DLB2_DEBUG(dlb2, __VA_ARGS__);  \
> +} while (0)
> +
> +/* The callback runs until it completes all outstanding QID->CQ
> + * map and unmap requests. To prevent deadlock, this function gives other
> + * threads a chance to grab the resource mutex and configure hardware.
> + */
> +static void *dlb2_complete_queue_map_unmap(void *__args)
> +{
> +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
> +       int ret;
> +
> +       while (1) {
> +               rte_spinlock_lock(&dlb2_dev->resource_mutex);
> +
> +               ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
> +               ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
> +
> +               if (ret != 0) {
> +                       rte_spinlock_unlock(&dlb2_dev->resource_mutex);
> +                       /* Relinquish the CPU so the application can process
> +                        * its CQs, so this function doesn't deadlock.
> +                        */
> +                       sched_yield();
> +               } else {
> +                       break;
> +               }
> +       }
> +
> +       dlb2_dev->worker_launched = false;
> +
> +       rte_spinlock_unlock(&dlb2_dev->resource_mutex);
> +
> +       return NULL;
> +}
> +
> +
> +/**
> + * os_schedule_work() - launch a thread to process pending map and unmap work
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function launches a kernel thread that will run until all pending
> + * map and unmap procedures are complete.
> + */
> +static inline void os_schedule_work(struct dlb2_hw *hw)
> +{
> +       struct dlb2_dev *dlb2_dev;
> +       pthread_t complete_queue_map_unmap_thread;
> +       int ret;
> +
> +       dlb2_dev = container_of(hw, struct dlb2_dev, hw);
> +
> +       ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
> +                                    "dlb_queue_unmap_waiter",
> +                                    NULL,
> +                                    dlb2_complete_queue_map_unmap,
> +                                    dlb2_dev);
> +       if (ret)
> +               DLB2_ERR(dlb2_dev,
> +                        "Could not create queue complete map/unmap thread, err=%d\n",
> +                        ret);
> +       else
> +               dlb2_dev->worker_launched = true;
> +}
> +
> +/**
> + * os_worker_active() - query whether the map/unmap worker thread is active
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function returns a boolean indicating whether a thread (launched by
> + * os_schedule_work()) is active. This function is used to determine
> + * whether or not to launch a worker thread.
> + */
> +static inline bool os_worker_active(struct dlb2_hw *hw)
> +{
> +       struct dlb2_dev *dlb2_dev;
> +
> +       dlb2_dev = container_of(hw, struct dlb2_dev, hw);
> +
> +       return dlb2_dev->worker_launched;
> +}
> +
> +#endif /*  __DLB2_OSDEP_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
> new file mode 100644
> index 0000000..423233b
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
> @@ -0,0 +1,440 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_OSDEP_BITMAP_H
> +#define __DLB2_OSDEP_BITMAP_H
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <unistd.h>
> +#include <rte_bitmap.h>
> +#include <rte_string_fns.h>
> +#include <rte_malloc.h>
> +#include <rte_errno.h>
> +#include "../dlb2_main.h"
> +
> +/*************************/
> +/*** Bitmap operations ***/
> +/*************************/
> +struct dlb2_bitmap {
> +       struct rte_bitmap *map;
> +       unsigned int len;
> +};
> +
> +/**
> + * dlb2_bitmap_alloc() - alloc a bitmap data structure
> + * @bitmap: pointer to dlb2_bitmap structure pointer.
> + * @len: number of entries in the bitmap.
> + *
> + * This function allocates a bitmap and initializes it with length @len. All
> + * entries are initially zero.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or len is 0.
> + * ENOMEM - could not allocate memory for the bitmap data structure.
> + */
> +static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
> +                                   unsigned int len)
> +{
> +       struct dlb2_bitmap *bm;
> +       void *mem;
> +       uint32_t alloc_size;
> +       uint32_t nbits = (uint32_t)len;
> +
> +       if (bitmap == NULL || nbits == 0)
> +               return -EINVAL;
> +
> +       /* Allocate DLB2 bitmap control struct */
> +       bm = rte_malloc("DLB2_PF",
> +                       sizeof(struct dlb2_bitmap),
> +                       RTE_CACHE_LINE_SIZE);
> +
> +       if (bm == NULL)
> +               return -ENOMEM;
> +
> +       /* Allocate bitmap memory */
> +       alloc_size = rte_bitmap_get_memory_footprint(nbits);
> +       mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
> +       if (mem == NULL) {
> +               rte_free(bm);
> +               return -ENOMEM;
> +       }
> +
> +       bm->map = rte_bitmap_init(len, mem, alloc_size);
> +       if (bm->map == NULL) {
> +               rte_free(mem);
> +               rte_free(bm);
> +               return -ENOMEM;
> +       }
> +
> +       bm->len = len;
> +
> +       *bitmap = bm;
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_free() - free a previously allocated bitmap data structure
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
> + */
> +static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
> +{
> +       if (bitmap == NULL)
> +               return;
> +
> +       rte_free(bitmap->map);
> +       rte_free(bitmap);
> +}
> +
> +/**
> + * dlb2_bitmap_fill() - fill a bitmap with all 1s
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function sets all bitmap values to 1.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
> +{
> +       unsigned int i;
> +
> +       if (bitmap  == NULL || bitmap->map == NULL)
> +               return -EINVAL;
> +
> +       for (i = 0; i != bitmap->len; i++)
> +               rte_bitmap_set(bitmap->map, i);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_fill() - fill a bitmap with all 0s
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function sets all bitmap values to 0.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
> +{
> +       if (bitmap  == NULL || bitmap->map == NULL)
> +               return -EINVAL;
> +
> +       rte_bitmap_reset(bitmap->map);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_set() - set a bitmap entry
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: bit index.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
> + *         bitmap length.
> + */
> +static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
> +                                 unsigned int bit)
> +{
> +       if (bitmap  == NULL || bitmap->map == NULL)
> +               return -EINVAL;
> +
> +       if (bitmap->len <= bit)
> +               return -EINVAL;
> +
> +       rte_bitmap_set(bitmap->map, bit);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_set_range() - set a range of bitmap entries
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: starting bit index.
> + * @len: length of the range.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
> + *         length.
> + */
> +static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
> +                                       unsigned int bit,
> +                                       unsigned int len)
> +{
> +       unsigned int i;
> +
> +       if (bitmap  == NULL || bitmap->map == NULL)
> +               return -EINVAL;
> +
> +       if (bitmap->len <= bit)
> +               return -EINVAL;
> +
> +       for (i = 0; i != len; i++)
> +               rte_bitmap_set(bitmap->map, bit + i);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_clear() - clear a bitmap entry
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: bit index.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
> + *         bitmap length.
> + */
> +static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
> +                                   unsigned int bit)
> +{
> +       if (bitmap  == NULL || bitmap->map == NULL)
> +               return -EINVAL;
> +
> +       if (bitmap->len <= bit)
> +               return -EINVAL;
> +
> +       rte_bitmap_clear(bitmap->map, bit);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_clear_range() - clear a range of bitmap entries
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: starting bit index.
> + * @len: length of the range.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
> + *         length.
> + */
> +static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
> +                                         unsigned int bit,
> +                                         unsigned int len)
> +{
> +       unsigned int i;
> +
> +       if (bitmap  == NULL || bitmap->map == NULL)
> +               return -EINVAL;
> +
> +       if (bitmap->len <= bit)
> +               return -EINVAL;
> +
> +       for (i = 0; i != len; i++)
> +               rte_bitmap_clear(bitmap->map, bit + i);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_find_set_bit_range() - find an range of set bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @len: length of the range.
> + *
> + * This function looks for a range of set bits of length @len.
> + *
> + * Return:
> + * Returns the base bit index upon success, < 0 otherwise.
> + *
> + * Errors:
> + * ENOENT - unable to find a length *len* range of set bits.
> + * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
> + */
> +static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
> +                                                unsigned int len)
> +{
> +       unsigned int i, j = 0;
> +
> +       if (bitmap  == NULL || bitmap->map == NULL || len == 0)
> +               return -EINVAL;
> +
> +       if (bitmap->len < len)
> +               return -ENOENT;
> +
> +       for (i = 0; i != bitmap->len; i++) {
> +               if  (rte_bitmap_get(bitmap->map, i)) {
> +                       if (++j == len)
> +                               return i - j + 1;
> +               } else {
> +                       j = 0;
> +               }
> +       }
> +
> +       /* No set bit range of length len? */
> +       return -ENOENT;
> +}
> +
> +/**
> + * dlb2_bitmap_find_set_bit() - find an range of set bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function looks for a single set bit.
> + *
> + * Return:
> + * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
> + */
> +static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
> +{
> +       unsigned int i;
> +
> +       if (bitmap == NULL)
> +               return -EINVAL;
> +
> +       if (bitmap->map == NULL)
> +               return -EINVAL;
> +
> +       for (i = 0; i != bitmap->len; i++) {
> +               if  (rte_bitmap_get(bitmap->map, i))
> +                       return i;
> +       }
> +
> +       return -ENOENT;
> +}
> +
> +/**
> + * dlb2_bitmap_count() - returns the number of set bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function looks for a single set bit.
> + *
> + * Return:
> + * Returns the number of set bits upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
> +{
> +       int weight = 0;
> +       unsigned int i;
> +
> +       if (bitmap == NULL)
> +               return -EINVAL;
> +
> +       if (bitmap->map == NULL)
> +               return -EINVAL;
> +
> +       for (i = 0; i != bitmap->len; i++) {
> +               if  (rte_bitmap_get(bitmap->map, i))
> +                       weight++;
> +       }
> +       return weight;
> +}
> +
> +/**
> + * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
> + *                                   bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * Return:
> + * Returns the bitmap's longest contiguous range of of set bits upon success,
> + * <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
> +{
> +       int max_len = 0, len = 0;
> +       unsigned int i;
> +
> +       if (bitmap == NULL)
> +               return -EINVAL;
> +
> +       if (bitmap->map == NULL)
> +               return -EINVAL;
> +
> +       for (i = 0; i != bitmap->len; i++) {
> +               if  (rte_bitmap_get(bitmap->map, i)) {
> +                       len++;
> +               } else {
> +                       if (len > max_len)
> +                               max_len = len;
> +                       len = 0;
> +               }
> +       }
> +
> +       if (len > max_len)
> +               max_len = len;
> +
> +       return max_len;
> +}
> +
> +/**
> + * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
> + * @dest: pointer to dlb2_bitmap structure, which will contain the results of
> + *       the 'or' of src1 and src2.
> + * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
> + * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
> + *
> + * This function 'or's two bitmaps together and stores the result in a third
> + * bitmap. The source and destination bitmaps can be the same.
> + *
> + * Return:
> + * Returns the number of set bits upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - One of the bitmaps is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
> +                                struct dlb2_bitmap *src1,
> +                                struct dlb2_bitmap *src2)
> +{
> +       unsigned int i, min;
> +       int numset = 0;
> +
> +       if (dest  == NULL || dest->map == NULL ||
> +           src1  == NULL || src1->map == NULL ||
> +           src2  == NULL || src2->map == NULL)
> +               return -EINVAL;
> +
> +       min = dest->len;
> +       min = (min > src1->len) ? src1->len : min;
> +       min = (min > src2->len) ? src2->len : min;
> +
> +       for (i = 0; i != min; i++) {
> +               if  (rte_bitmap_get(src1->map, i) ||
> +                    rte_bitmap_get(src2->map, i)) {
> +                       rte_bitmap_set(dest->map, i);
> +                       numset++;
> +               } else {
> +                       rte_bitmap_clear(dest->map, i);
> +               }
> +       }
> +
> +       return numset;
> +}
> +
> +#endif /*  __DLB2_OSDEP_BITMAP_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
> new file mode 100644
> index 0000000..5531739
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
> @@ -0,0 +1,131 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_OSDEP_LIST_H
> +#define __DLB2_OSDEP_LIST_H
> +
> +#include <rte_tailq.h>
> +
> +struct dlb2_list_entry {
> +       TAILQ_ENTRY(dlb2_list_entry) node;
> +};
> +
> +/* Dummy - just a struct definition */
> +TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
> +
> +/* =================
> + * TAILQ Supplements
> + * =================
> + */
> +
> +#ifndef TAILQ_FOREACH_ENTRY
> +#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)             \
> +       for ((iter) = TAILQ_FIRST(&head);                       \
> +           (iter)                                              \
> +               && (ptr = container_of(iter, typeof(*(ptr)), name)); \
> +           (iter) = TAILQ_NEXT((iter), node))
> +#endif
> +
> +#ifndef TAILQ_FOREACH_ENTRY_SAFE
> +#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)  \
> +       for ((iter) = TAILQ_FIRST(&head);                       \
> +           (iter) &&                                           \
> +               (ptr = container_of(iter, typeof(*(ptr)), name)) &&\
> +               ((tvar) = TAILQ_NEXT((iter), node), 1); \
> +           (iter) = (tvar))
> +#endif
> +
> +/***********************/
> +/*** List operations ***/
> +/***********************/
> +
> +/**
> + * dlb2_list_init_head() - initialize the head of a list
> + * @head: list head
> + */
> +static inline void dlb2_list_init_head(struct dlb2_list_head *head)
> +{
> +       TAILQ_INIT(head);
> +}
> +
> +/**
> + * dlb2_list_add() - add an entry to a list
> + * @head: list head
> + * @entry: new list entry
> + */
> +static inline void
> +dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
> +{
> +       TAILQ_INSERT_TAIL(head, entry, node);
> +}
> +
> +/**
> + * dlb2_list_del() - delete an entry from a list
> + * @entry: list entry
> + * @head: list head
> + */
> +static inline void dlb2_list_del(struct dlb2_list_head *head,
> +                                struct dlb2_list_entry *entry)
> +{
> +       TAILQ_REMOVE(head, entry, node);
> +}
> +
> +/**
> + * dlb2_list_empty() - check if a list is empty
> + * @head: list head
> + *
> + * Return:
> + * Returns 1 if empty, 0 if not.
> + */
> +static inline int dlb2_list_empty(struct dlb2_list_head *head)
> +{
> +       return TAILQ_EMPTY(head);
> +}
> +
> +/**
> + * dlb2_list_splice() - splice a list
> + * @src_head: list to be added
> + * @ head: where src_head will be inserted
> + */
> +static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
> +                                   struct dlb2_list_head *head)
> +{
> +       TAILQ_CONCAT(head, src_head, node);
> +}
> +
> +/**
> + * DLB2_LIST_HEAD() - retrieve the head of the list
> + * @head: list head
> + * @type: type of the list variable
> + * @name: name of the list field within the containing struct
> + */
> +#define DLB2_LIST_HEAD(head, type, name)                       \
> +       (TAILQ_FIRST(&head) ?                                   \
> +               container_of(TAILQ_FIRST(&head), type, name) :  \
> +               NULL)
> +
> +/**
> + * DLB2_LIST_FOR_EACH() - iterate over a list
> + * @head: list head
> + * @ptr: pointer to struct containing a struct list
> + * @name: name of the list field within the containing struct
> + * @iter: iterator variable
> + */
> +#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
> +       TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
> +
> +/**
> + * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
> + * an element is removed from the list while processing it.
> + * @ptr: pointer to struct containing a struct list
> + * @ptr_tmp: pointer to struct containing a struct list (temporary)
> + * @head: list head
> + * @name: name of the list field within the containing struct
> + * @iter: iterator variable
> + * @iter_tmp: iterator variable (temporary)
> + */
> +#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
> +       TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
> +
> +#endif /*  __DLB2_OSDEP_LIST_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
> new file mode 100644
> index 0000000..0a48f7e
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
> @@ -0,0 +1,31 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_OSDEP_TYPES_H
> +#define __DLB2_OSDEP_TYPES_H
> +
> +#include <linux/types.h>
> +
> +#include <inttypes.h>
> +#include <ctype.h>
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <string.h>
> +#include <unistd.h>
> +#include <errno.h>
> +
> +/* Types for user mode PF PMD */
> +typedef uint8_t         u8;
> +typedef int8_t          s8;
> +typedef uint16_t        u16;
> +typedef int16_t         s16;
> +typedef uint32_t        u32;
> +typedef int32_t         s32;
> +typedef uint64_t        u64;
> +
> +#define __iomem
> +
> +/* END types for user mode PF PMD */
> +
> +#endif /* __DLB2_OSDEP_TYPES_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
> new file mode 100644
> index 0000000..43ecad4
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
> @@ -0,0 +1,2527 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_REGS_H
> +#define __DLB2_REGS_H
> +
> +#include "dlb2_osdep_types.h"
> +
> +#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
> +#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
> +       (0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
> +union dlb2_func_pf_vf2pf_mailbox {
> +       struct {
> +               u32 msg : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
> +       (0x1f00 + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
> +union dlb2_func_pf_vf2pf_mailbox_isr {
> +       struct {
> +               u32 vf0_isr : 1;
> +               u32 vf1_isr : 1;
> +               u32 vf2_isr : 1;
> +               u32 vf3_isr : 1;
> +               u32 vf4_isr : 1;
> +               u32 vf5_isr : 1;
> +               u32 vf6_isr : 1;
> +               u32 vf7_isr : 1;
> +               u32 vf8_isr : 1;
> +               u32 vf9_isr : 1;
> +               u32 vf10_isr : 1;
> +               u32 vf11_isr : 1;
> +               u32 vf12_isr : 1;
> +               u32 vf13_isr : 1;
> +               u32 vf14_isr : 1;
> +               u32 vf15_isr : 1;
> +               u32 rsvd0 : 16;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
> +       (0x1f04 + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
> +union dlb2_func_pf_vf2pf_flr_isr {
> +       struct {
> +               u32 vf0_isr : 1;
> +               u32 vf1_isr : 1;
> +               u32 vf2_isr : 1;
> +               u32 vf3_isr : 1;
> +               u32 vf4_isr : 1;
> +               u32 vf5_isr : 1;
> +               u32 vf6_isr : 1;
> +               u32 vf7_isr : 1;
> +               u32 vf8_isr : 1;
> +               u32 vf9_isr : 1;
> +               u32 vf10_isr : 1;
> +               u32 vf11_isr : 1;
> +               u32 vf12_isr : 1;
> +               u32 vf13_isr : 1;
> +               u32 vf14_isr : 1;
> +               u32 vf15_isr : 1;
> +               u32 rsvd0 : 16;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
> +       (0x1f10 + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
> +union dlb2_func_pf_vf2pf_isr_pend {
> +       struct {
> +               u32 isr_pend : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
> +#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
> +       (0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
> +union dlb2_func_pf_pf2vf_mailbox {
> +       struct {
> +               u32 msg : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
> +       (0x2f00 + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
> +union dlb2_func_pf_pf2vf_mailbox_isr {
> +       struct {
> +               u32 vf0_isr : 1;
> +               u32 vf1_isr : 1;
> +               u32 vf2_isr : 1;
> +               u32 vf3_isr : 1;
> +               u32 vf4_isr : 1;
> +               u32 vf5_isr : 1;
> +               u32 vf6_isr : 1;
> +               u32 vf7_isr : 1;
> +               u32 vf8_isr : 1;
> +               u32 vf9_isr : 1;
> +               u32 vf10_isr : 1;
> +               u32 vf11_isr : 1;
> +               u32 vf12_isr : 1;
> +               u32 vf13_isr : 1;
> +               u32 vf14_isr : 1;
> +               u32 vf15_isr : 1;
> +               u32 rsvd0 : 16;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
> +       (0x3000 + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
> +union dlb2_func_pf_vf_reset_in_progress {
> +       struct {
> +               u32 vf0_reset_in_progress : 1;
> +               u32 vf1_reset_in_progress : 1;
> +               u32 vf2_reset_in_progress : 1;
> +               u32 vf3_reset_in_progress : 1;
> +               u32 vf4_reset_in_progress : 1;
> +               u32 vf5_reset_in_progress : 1;
> +               u32 vf6_reset_in_progress : 1;
> +               u32 vf7_reset_in_progress : 1;
> +               u32 vf8_reset_in_progress : 1;
> +               u32 vf9_reset_in_progress : 1;
> +               u32 vf10_reset_in_progress : 1;
> +               u32 vf11_reset_in_progress : 1;
> +               u32 vf12_reset_in_progress : 1;
> +               u32 vf13_reset_in_progress : 1;
> +               u32 vf14_reset_in_progress : 1;
> +               u32 vf15_reset_in_progress : 1;
> +               u32 rsvd0 : 16;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
> +       (0x100000c + (x) * 0x10)
> +#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
> +union dlb2_msix_mem_vector_ctrl {
> +       struct {
> +               u32 vec_mask : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
> +       (0x20 + (x) * 0x4)
> +#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
> +union dlb2_iosf_func_vf_bar_dsbl {
> +       struct {
> +               u32 func_vf_bar_dis : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_VAS 0x1000011c
> +#define DLB2_SYS_TOTAL_VAS_RST 0x20
> +union dlb2_sys_total_vas {
> +       struct {
> +               u32 total_vas : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
> +#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
> +union dlb2_sys_total_dir_ports {
> +       struct {
> +               u32 total_dir_ports : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
> +#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
> +union dlb2_sys_total_ldb_ports {
> +       struct {
> +               u32 total_ldb_ports : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
> +#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
> +union dlb2_sys_total_dir_qid {
> +       struct {
> +               u32 total_dir_qid : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
> +#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
> +union dlb2_sys_total_ldb_qid {
> +       struct {
> +               u32 total_ldb_qid : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
> +#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
> +union dlb2_sys_total_dir_crds {
> +       struct {
> +               u32 total_dir_credits : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
> +#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
> +union dlb2_sys_total_ldb_crds {
> +       struct {
> +               u32 total_ldb_credits : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
> +#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
> +union dlb2_sys_alarm_pf_synd2 {
> +       struct {
> +               u32 lock_id : 16;
> +               u32 meas : 1;
> +               u32 debug : 7;
> +               u32 cq_pop : 1;
> +               u32 qe_uhl : 1;
> +               u32 qe_orsp : 1;
> +               u32 qe_valid : 1;
> +               u32 cq_int_rearm : 1;
> +               u32 dsi_error : 1;
> +               u32 rsvd0 : 2;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
> +#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
> +union dlb2_sys_alarm_pf_synd1 {
> +       struct {
> +               u32 dsi : 16;
> +               u32 qid : 8;
> +               u32 qtype : 2;
> +               u32 qpri : 3;
> +               u32 msg_type : 3;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
> +#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
> +union dlb2_sys_alarm_pf_synd0 {
> +       struct {
> +               u32 syndrome : 8;
> +               u32 rtype : 2;
> +               u32 rsvd0 : 3;
> +               u32 is_ldb : 1;
> +               u32 cls : 2;
> +               u32 aid : 6;
> +               u32 unit : 4;
> +               u32 source : 4;
> +               u32 more : 1;
> +               u32 valid : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_LDB_VPP_V(x) \
> +       (0x10000f00 + (x) * 0x1000)
> +#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
> +union dlb2_sys_vf_ldb_vpp_v {
> +       struct {
> +               u32 vpp_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_LDB_VPP2PP(x) \
> +       (0x10000f04 + (x) * 0x1000)
> +#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
> +union dlb2_sys_vf_ldb_vpp2pp {
> +       struct {
> +               u32 pp : 6;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_DIR_VPP_V(x) \
> +       (0x10000f08 + (x) * 0x1000)
> +#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
> +union dlb2_sys_vf_dir_vpp_v {
> +       struct {
> +               u32 vpp_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_DIR_VPP2PP(x) \
> +       (0x10000f0c + (x) * 0x1000)
> +#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
> +union dlb2_sys_vf_dir_vpp2pp {
> +       struct {
> +               u32 pp : 6;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_LDB_VQID_V(x) \
> +       (0x10000f10 + (x) * 0x1000)
> +#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
> +union dlb2_sys_vf_ldb_vqid_v {
> +       struct {
> +               u32 vqid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_LDB_VQID2QID(x) \
> +       (0x10000f14 + (x) * 0x1000)
> +#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
> +union dlb2_sys_vf_ldb_vqid2qid {
> +       struct {
> +               u32 qid : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_QID2VQID(x) \
> +       (0x10000f18 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_QID2VQID_RST 0x0
> +union dlb2_sys_ldb_qid2vqid {
> +       struct {
> +               u32 vqid : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_DIR_VQID_V(x) \
> +       (0x10000f1c + (x) * 0x1000)
> +#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
> +union dlb2_sys_vf_dir_vqid_v {
> +       struct {
> +               u32 vqid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_DIR_VQID2QID(x) \
> +       (0x10000f20 + (x) * 0x1000)
> +#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
> +union dlb2_sys_vf_dir_vqid2qid {
> +       struct {
> +               u32 qid : 6;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_VASQID_V(x) \
> +       (0x10000f24 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_VASQID_V_RST 0x0
> +union dlb2_sys_ldb_vasqid_v {
> +       struct {
> +               u32 vasqid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_VASQID_V(x) \
> +       (0x10000f28 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_VASQID_V_RST 0x0
> +union dlb2_sys_dir_vasqid_v {
> +       struct {
> +               u32 vasqid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_VF_SYND2(x) \
> +       (0x10000f48 + (x) * 0x1000)
> +#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
> +union dlb2_sys_alarm_vf_synd2 {
> +       struct {
> +               u32 lock_id : 16;
> +               u32 debug : 8;
> +               u32 cq_pop : 1;
> +               u32 qe_uhl : 1;
> +               u32 qe_orsp : 1;
> +               u32 qe_valid : 1;
> +               u32 isz : 1;
> +               u32 dsi_error : 1;
> +               u32 dlbrsvd : 2;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_VF_SYND1(x) \
> +       (0x10000f44 + (x) * 0x1000)
> +#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
> +union dlb2_sys_alarm_vf_synd1 {
> +       struct {
> +               u32 dsi : 16;
> +               u32 qid : 8;
> +               u32 qtype : 2;
> +               u32 qpri : 3;
> +               u32 msg_type : 3;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_VF_SYND0(x) \
> +       (0x10000f40 + (x) * 0x1000)
> +#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
> +union dlb2_sys_alarm_vf_synd0 {
> +       struct {
> +               u32 syndrome : 8;
> +               u32 rtype : 2;
> +               u32 vf_synd0_parity : 1;
> +               u32 vf_synd1_parity : 1;
> +               u32 vf_synd2_parity : 1;
> +               u32 is_ldb : 1;
> +               u32 cls : 2;
> +               u32 aid : 6;
> +               u32 unit : 4;
> +               u32 source : 4;
> +               u32 more : 1;
> +               u32 valid : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_QID_CFG_V(x) \
> +       (0x10000f58 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
> +union dlb2_sys_ldb_qid_cfg_v {
> +       struct {
> +               u32 sn_cfg_v : 1;
> +               u32 fid_cfg_v : 1;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_QID_ITS(x) \
> +       (0x10000f54 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_QID_ITS_RST 0x0
> +union dlb2_sys_ldb_qid_its {
> +       struct {
> +               u32 qid_its : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_QID_V(x) \
> +       (0x10000f50 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_QID_V_RST 0x0
> +union dlb2_sys_ldb_qid_v {
> +       struct {
> +               u32 qid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_QID_ITS(x) \
> +       (0x10000f64 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_QID_ITS_RST 0x0
> +union dlb2_sys_dir_qid_its {
> +       struct {
> +               u32 qid_its : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_QID_V(x) \
> +       (0x10000f60 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_QID_V_RST 0x0
> +union dlb2_sys_dir_qid_v {
> +       struct {
> +               u32 qid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
> +       (0x10000fa8 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
> +union dlb2_sys_ldb_cq_ai_data {
> +       struct {
> +               u32 cq_ai_data : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
> +       (0x10000fa4 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
> +union dlb2_sys_ldb_cq_ai_addr {
> +       struct {
> +               u32 rsvd1 : 2;
> +               u32 cq_ai_addr : 18;
> +               u32 rsvd0 : 12;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_PASID(x) \
> +       (0x10000fa0 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
> +union dlb2_sys_ldb_cq_pasid {
> +       struct {
> +               u32 pasid : 20;
> +               u32 exe_req : 1;
> +               u32 priv_req : 1;
> +               u32 fmt2 : 1;
> +               u32 rsvd0 : 9;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_AT(x) \
> +       (0x10000f9c + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_AT_RST 0x0
> +union dlb2_sys_ldb_cq_at {
> +       struct {
> +               u32 cq_at : 2;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_ISR(x) \
> +       (0x10000f98 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
> +/* CQ Interrupt Modes */
> +#define DLB2_CQ_ISR_MODE_DIS  0
> +#define DLB2_CQ_ISR_MODE_MSI  1
> +#define DLB2_CQ_ISR_MODE_MSIX 2
> +#define DLB2_CQ_ISR_MODE_ADI  3
> +union dlb2_sys_ldb_cq_isr {
> +       struct {
> +               u32 vector : 6;
> +               u32 vf : 4;
> +               u32 en_code : 2;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
> +       (0x10000f94 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
> +union dlb2_sys_ldb_cq2vf_pf_ro {
> +       struct {
> +               u32 vf : 4;
> +               u32 is_pf : 1;
> +               u32 ro : 1;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_PP_V(x) \
> +       (0x10000f90 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_PP_V_RST 0x0
> +union dlb2_sys_ldb_pp_v {
> +       struct {
> +               u32 pp_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_PP2VDEV(x) \
> +       (0x10000f8c + (x) * 0x1000)
> +#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
> +union dlb2_sys_ldb_pp2vdev {
> +       struct {
> +               u32 vdev : 4;
> +               u32 rsvd0 : 28;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_PP2VAS(x) \
> +       (0x10000f88 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_PP2VAS_RST 0x0
> +union dlb2_sys_ldb_pp2vas {
> +       struct {
> +               u32 vas : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
> +       (0x10000f84 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
> +union dlb2_sys_ldb_cq_addr_u {
> +       struct {
> +               u32 addr_u : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
> +       (0x10000f80 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
> +union dlb2_sys_ldb_cq_addr_l {
> +       struct {
> +               u32 rsvd0 : 6;
> +               u32 addr_l : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_FMT(x) \
> +       (0x10000fec + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
> +union dlb2_sys_dir_cq_fmt {
> +       struct {
> +               u32 keep_pf_ppid : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
> +       (0x10000fe8 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
> +union dlb2_sys_dir_cq_ai_data {
> +       struct {
> +               u32 cq_ai_data : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
> +       (0x10000fe4 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
> +union dlb2_sys_dir_cq_ai_addr {
> +       struct {
> +               u32 rsvd1 : 2;
> +               u32 cq_ai_addr : 18;
> +               u32 rsvd0 : 12;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_PASID(x) \
> +       (0x10000fe0 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
> +union dlb2_sys_dir_cq_pasid {
> +       struct {
> +               u32 pasid : 20;
> +               u32 exe_req : 1;
> +               u32 priv_req : 1;
> +               u32 fmt2 : 1;
> +               u32 rsvd0 : 9;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_AT(x) \
> +       (0x10000fdc + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_AT_RST 0x0
> +union dlb2_sys_dir_cq_at {
> +       struct {
> +               u32 cq_at : 2;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_ISR(x) \
> +       (0x10000fd8 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
> +union dlb2_sys_dir_cq_isr {
> +       struct {
> +               u32 vector : 6;
> +               u32 vf : 4;
> +               u32 en_code : 2;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
> +       (0x10000fd4 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
> +union dlb2_sys_dir_cq2vf_pf_ro {
> +       struct {
> +               u32 vf : 4;
> +               u32 is_pf : 1;
> +               u32 ro : 1;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_PP_V(x) \
> +       (0x10000fd0 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_PP_V_RST 0x0
> +union dlb2_sys_dir_pp_v {
> +       struct {
> +               u32 pp_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_PP2VDEV(x) \
> +       (0x10000fcc + (x) * 0x1000)
> +#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
> +union dlb2_sys_dir_pp2vdev {
> +       struct {
> +               u32 vdev : 4;
> +               u32 rsvd0 : 28;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_PP2VAS(x) \
> +       (0x10000fc8 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_PP2VAS_RST 0x0
> +union dlb2_sys_dir_pp2vas {
> +       struct {
> +               u32 vas : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
> +       (0x10000fc4 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
> +union dlb2_sys_dir_cq_addr_u {
> +       struct {
> +               u32 addr_u : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
> +       (0x10000fc0 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
> +union dlb2_sys_dir_cq_addr_l {
> +       struct {
> +               u32 rsvd0 : 6;
> +               u32 addr_l : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
> +#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
> +union dlb2_sys_ingress_alarm_enbl {
> +       struct {
> +               u32 illegal_hcw : 1;
> +               u32 illegal_pp : 1;
> +               u32 illegal_pasid : 1;
> +               u32 illegal_qid : 1;
> +               u32 disabled_qid : 1;
> +               u32 illegal_ldb_qid_cfg : 1;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_MSIX_ACK 0x10000400
> +#define DLB2_SYS_MSIX_ACK_RST 0x0
> +union dlb2_sys_msix_ack {
> +       struct {
> +               u32 msix_0_ack : 1;
> +               u32 msix_1_ack : 1;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
> +#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
> +union dlb2_sys_msix_passthru {
> +       struct {
> +               u32 msix_0_passthru : 1;
> +               u32 msix_1_passthru : 1;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_MSIX_MODE 0x10000408
> +#define DLB2_SYS_MSIX_MODE_RST 0x0
> +/* MSI-X Modes */
> +#define DLB2_MSIX_MODE_PACKED     0
> +#define DLB2_MSIX_MODE_COMPRESSED 1
> +union dlb2_sys_msix_mode {
> +       struct {
> +               u32 mode : 1;
> +               u32 poll_mode : 1;
> +               u32 poll_mask : 1;
> +               u32 poll_lock : 1;
> +               u32 rsvd0 : 28;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
> +#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
> +union dlb2_sys_dir_cq_31_0_occ_int_sts {
> +       struct {
> +               u32 cq_0_occ_int : 1;
> +               u32 cq_1_occ_int : 1;
> +               u32 cq_2_occ_int : 1;
> +               u32 cq_3_occ_int : 1;
> +               u32 cq_4_occ_int : 1;
> +               u32 cq_5_occ_int : 1;
> +               u32 cq_6_occ_int : 1;
> +               u32 cq_7_occ_int : 1;
> +               u32 cq_8_occ_int : 1;
> +               u32 cq_9_occ_int : 1;
> +               u32 cq_10_occ_int : 1;
> +               u32 cq_11_occ_int : 1;
> +               u32 cq_12_occ_int : 1;
> +               u32 cq_13_occ_int : 1;
> +               u32 cq_14_occ_int : 1;
> +               u32 cq_15_occ_int : 1;
> +               u32 cq_16_occ_int : 1;
> +               u32 cq_17_occ_int : 1;
> +               u32 cq_18_occ_int : 1;
> +               u32 cq_19_occ_int : 1;
> +               u32 cq_20_occ_int : 1;
> +               u32 cq_21_occ_int : 1;
> +               u32 cq_22_occ_int : 1;
> +               u32 cq_23_occ_int : 1;
> +               u32 cq_24_occ_int : 1;
> +               u32 cq_25_occ_int : 1;
> +               u32 cq_26_occ_int : 1;
> +               u32 cq_27_occ_int : 1;
> +               u32 cq_28_occ_int : 1;
> +               u32 cq_29_occ_int : 1;
> +               u32 cq_30_occ_int : 1;
> +               u32 cq_31_occ_int : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
> +#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
> +union dlb2_sys_dir_cq_63_32_occ_int_sts {
> +       struct {
> +               u32 cq_32_occ_int : 1;
> +               u32 cq_33_occ_int : 1;
> +               u32 cq_34_occ_int : 1;
> +               u32 cq_35_occ_int : 1;
> +               u32 cq_36_occ_int : 1;
> +               u32 cq_37_occ_int : 1;
> +               u32 cq_38_occ_int : 1;
> +               u32 cq_39_occ_int : 1;
> +               u32 cq_40_occ_int : 1;
> +               u32 cq_41_occ_int : 1;
> +               u32 cq_42_occ_int : 1;
> +               u32 cq_43_occ_int : 1;
> +               u32 cq_44_occ_int : 1;
> +               u32 cq_45_occ_int : 1;
> +               u32 cq_46_occ_int : 1;
> +               u32 cq_47_occ_int : 1;
> +               u32 cq_48_occ_int : 1;
> +               u32 cq_49_occ_int : 1;
> +               u32 cq_50_occ_int : 1;
> +               u32 cq_51_occ_int : 1;
> +               u32 cq_52_occ_int : 1;
> +               u32 cq_53_occ_int : 1;
> +               u32 cq_54_occ_int : 1;
> +               u32 cq_55_occ_int : 1;
> +               u32 cq_56_occ_int : 1;
> +               u32 cq_57_occ_int : 1;
> +               u32 cq_58_occ_int : 1;
> +               u32 cq_59_occ_int : 1;
> +               u32 cq_60_occ_int : 1;
> +               u32 cq_61_occ_int : 1;
> +               u32 cq_62_occ_int : 1;
> +               u32 cq_63_occ_int : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
> +#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
> +union dlb2_sys_ldb_cq_31_0_occ_int_sts {
> +       struct {
> +               u32 cq_0_occ_int : 1;
> +               u32 cq_1_occ_int : 1;
> +               u32 cq_2_occ_int : 1;
> +               u32 cq_3_occ_int : 1;
> +               u32 cq_4_occ_int : 1;
> +               u32 cq_5_occ_int : 1;
> +               u32 cq_6_occ_int : 1;
> +               u32 cq_7_occ_int : 1;
> +               u32 cq_8_occ_int : 1;
> +               u32 cq_9_occ_int : 1;
> +               u32 cq_10_occ_int : 1;
> +               u32 cq_11_occ_int : 1;
> +               u32 cq_12_occ_int : 1;
> +               u32 cq_13_occ_int : 1;
> +               u32 cq_14_occ_int : 1;
> +               u32 cq_15_occ_int : 1;
> +               u32 cq_16_occ_int : 1;
> +               u32 cq_17_occ_int : 1;
> +               u32 cq_18_occ_int : 1;
> +               u32 cq_19_occ_int : 1;
> +               u32 cq_20_occ_int : 1;
> +               u32 cq_21_occ_int : 1;
> +               u32 cq_22_occ_int : 1;
> +               u32 cq_23_occ_int : 1;
> +               u32 cq_24_occ_int : 1;
> +               u32 cq_25_occ_int : 1;
> +               u32 cq_26_occ_int : 1;
> +               u32 cq_27_occ_int : 1;
> +               u32 cq_28_occ_int : 1;
> +               u32 cq_29_occ_int : 1;
> +               u32 cq_30_occ_int : 1;
> +               u32 cq_31_occ_int : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
> +#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
> +union dlb2_sys_ldb_cq_63_32_occ_int_sts {
> +       struct {
> +               u32 cq_32_occ_int : 1;
> +               u32 cq_33_occ_int : 1;
> +               u32 cq_34_occ_int : 1;
> +               u32 cq_35_occ_int : 1;
> +               u32 cq_36_occ_int : 1;
> +               u32 cq_37_occ_int : 1;
> +               u32 cq_38_occ_int : 1;
> +               u32 cq_39_occ_int : 1;
> +               u32 cq_40_occ_int : 1;
> +               u32 cq_41_occ_int : 1;
> +               u32 cq_42_occ_int : 1;
> +               u32 cq_43_occ_int : 1;
> +               u32 cq_44_occ_int : 1;
> +               u32 cq_45_occ_int : 1;
> +               u32 cq_46_occ_int : 1;
> +               u32 cq_47_occ_int : 1;
> +               u32 cq_48_occ_int : 1;
> +               u32 cq_49_occ_int : 1;
> +               u32 cq_50_occ_int : 1;
> +               u32 cq_51_occ_int : 1;
> +               u32 cq_52_occ_int : 1;
> +               u32 cq_53_occ_int : 1;
> +               u32 cq_54_occ_int : 1;
> +               u32 cq_55_occ_int : 1;
> +               u32 cq_56_occ_int : 1;
> +               u32 cq_57_occ_int : 1;
> +               u32 cq_58_occ_int : 1;
> +               u32 cq_59_occ_int : 1;
> +               u32 cq_60_occ_int : 1;
> +               u32 cq_61_occ_int : 1;
> +               u32 cq_62_occ_int : 1;
> +               u32 cq_63_occ_int : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
> +#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
> +union dlb2_sys_dir_cq_opt_clr {
> +       struct {
> +               u32 cq : 6;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
> +#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
> +union dlb2_sys_alarm_hw_synd {
> +       struct {
> +               u32 syndrome : 8;
> +               u32 rtype : 2;
> +               u32 alarm : 1;
> +               u32 cwd : 1;
> +               u32 vf_pf_mb : 1;
> +               u32 rsvd0 : 1;
> +               u32 cls : 2;
> +               u32 aid : 6;
> +               u32 unit : 4;
> +               u32 source : 4;
> +               u32 more : 1;
> +               u32 valid : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
> +       (0x20000000 + (x) * 0x1000)
> +#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
> +union dlb2_aqed_pipe_qid_fid_lim {
> +       struct {
> +               u32 qid_fid_limit : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
> +       (0x20080000 + (x) * 0x1000)
> +#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
> +union dlb2_aqed_pipe_qid_hid_width {
> +       struct {
> +               u32 compress_code : 3;
> +               u32 rsvd0 : 29;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
> +#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
> +union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_ATM_QID2CQIDIX_00(x) \
> +       (0x30080000 + (x) * 0x1000)
> +#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
> +#define DLB2_ATM_QID2CQIDIX(x, y) \
> +       (DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
> +#define DLB2_ATM_QID2CQIDIX_NUM 16
> +union dlb2_atm_qid2cqidix_00 {
> +       struct {
> +               u32 cq_p0 : 8;
> +               u32 cq_p1 : 8;
> +               u32 cq_p2 : 8;
> +               u32 cq_p3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
> +#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
> +union dlb2_atm_cfg_arb_weights_rdy_bin {
> +       struct {
> +               u32 bin0 : 8;
> +               u32 bin1 : 8;
> +               u32 bin2 : 8;
> +               u32 bin3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
> +#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
> +union dlb2_atm_cfg_arb_weights_sched_bin {
> +       struct {
> +               u32 bin0 : 8;
> +               u32 bin1 : 8;
> +               u32 bin2 : 8;
> +               u32 bin3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
> +       (0x40000000 + (x) * 0x1000)
> +#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
> +union dlb2_chp_cfg_dir_vas_crd {
> +       struct {
> +               u32 count : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
> +       (0x40080000 + (x) * 0x1000)
> +#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
> +union dlb2_chp_cfg_ldb_vas_crd {
> +       struct {
> +               u32 count : 15;
> +               u32 rsvd0 : 17;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_ORD_QID_SN(x) \
> +       (0x40100000 + (x) * 0x1000)
> +#define DLB2_CHP_ORD_QID_SN_RST 0x0
> +union dlb2_chp_ord_qid_sn {
> +       struct {
> +               u32 sn : 10;
> +               u32 rsvd0 : 22;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_ORD_QID_SN_MAP(x) \
> +       (0x40180000 + (x) * 0x1000)
> +#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
> +union dlb2_chp_ord_qid_sn_map {
> +       struct {
> +               u32 mode : 3;
> +               u32 slot : 4;
> +               u32 rsvz0 : 1;
> +               u32 grp : 1;
> +               u32 rsvz1 : 1;
> +               u32 rsvd0 : 22;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_SN_CHK_ENBL(x) \
> +       (0x40200000 + (x) * 0x1000)
> +#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
> +union dlb2_chp_sn_chk_enbl {
> +       struct {
> +               u32 en : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_DEPTH(x) \
> +       (0x40280000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
> +union dlb2_chp_dir_cq_depth {
> +       struct {
> +               u32 depth : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
> +       (0x40300000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
> +union dlb2_chp_dir_cq_int_depth_thrsh {
> +       struct {
> +               u32 depth_threshold : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
> +       (0x40380000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
> +union dlb2_chp_dir_cq_int_enb {
> +       struct {
> +               u32 en_tim : 1;
> +               u32 en_depth : 1;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
> +       (0x40480000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
> +union dlb2_chp_dir_cq_tmr_thrsh {
> +       struct {
> +               u32 thrsh_0 : 1;
> +               u32 thrsh_13_1 : 13;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
> +       (0x40500000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
> +union dlb2_chp_dir_cq_tkn_depth_sel {
> +       struct {
> +               u32 token_depth_select : 4;
> +               u32 rsvd0 : 28;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
> +       (0x40580000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
> +union dlb2_chp_dir_cq_wd_enb {
> +       struct {
> +               u32 wd_enable : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_WPTR(x) \
> +       (0x40600000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
> +union dlb2_chp_dir_cq_wptr {
> +       struct {
> +               u32 write_pointer : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ2VAS(x) \
> +       (0x40680000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
> +union dlb2_chp_dir_cq2vas {
> +       struct {
> +               u32 cq2vas : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_HIST_LIST_BASE(x) \
> +       (0x40700000 + (x) * 0x1000)
> +#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
> +union dlb2_chp_hist_list_base {
> +       struct {
> +               u32 base : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_HIST_LIST_LIM(x) \
> +       (0x40780000 + (x) * 0x1000)
> +#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
> +union dlb2_chp_hist_list_lim {
> +       struct {
> +               u32 limit : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
> +       (0x40800000 + (x) * 0x1000)
> +#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
> +union dlb2_chp_hist_list_pop_ptr {
> +       struct {
> +               u32 pop_ptr : 13;
> +               u32 generation : 1;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
> +       (0x40880000 + (x) * 0x1000)
> +#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
> +union dlb2_chp_hist_list_push_ptr {
> +       struct {
> +               u32 push_ptr : 13;
> +               u32 generation : 1;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_DEPTH(x) \
> +       (0x40900000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
> +union dlb2_chp_ldb_cq_depth {
> +       struct {
> +               u32 depth : 11;
> +               u32 rsvd0 : 21;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
> +       (0x40980000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
> +union dlb2_chp_ldb_cq_int_depth_thrsh {
> +       struct {
> +               u32 depth_threshold : 11;
> +               u32 rsvd0 : 21;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
> +       (0x40a00000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
> +union dlb2_chp_ldb_cq_int_enb {
> +       struct {
> +               u32 en_tim : 1;
> +               u32 en_depth : 1;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
> +       (0x40b00000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
> +union dlb2_chp_ldb_cq_tmr_thrsh {
> +       struct {
> +               u32 thrsh_0 : 1;
> +               u32 thrsh_13_1 : 13;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
> +       (0x40b80000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
> +union dlb2_chp_ldb_cq_tkn_depth_sel {
> +       struct {
> +               u32 token_depth_select : 4;
> +               u32 rsvd0 : 28;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
> +       (0x40c00000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
> +union dlb2_chp_ldb_cq_wd_enb {
> +       struct {
> +               u32 wd_enable : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_WPTR(x) \
> +       (0x40c80000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
> +union dlb2_chp_ldb_cq_wptr {
> +       struct {
> +               u32 write_pointer : 11;
> +               u32 rsvd0 : 21;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ2VAS(x) \
> +       (0x40d00000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
> +union dlb2_chp_ldb_cq2vas {
> +       struct {
> +               u32 cq2vas : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
> +#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
> +union dlb2_chp_cfg_chp_csr_ctrl {
> +       struct {
> +               u32 int_cor_alarm_dis : 1;
> +               u32 int_cor_synd_dis : 1;
> +               u32 int_uncr_alarm_dis : 1;
> +               u32 int_unc_synd_dis : 1;
> +               u32 int_inf0_alarm_dis : 1;
> +               u32 int_inf0_synd_dis : 1;
> +               u32 int_inf1_alarm_dis : 1;
> +               u32 int_inf1_synd_dis : 1;
> +               u32 int_inf2_alarm_dis : 1;
> +               u32 int_inf2_synd_dis : 1;
> +               u32 int_inf3_alarm_dis : 1;
> +               u32 int_inf3_synd_dis : 1;
> +               u32 int_inf4_alarm_dis : 1;
> +               u32 int_inf4_synd_dis : 1;
> +               u32 int_inf5_alarm_dis : 1;
> +               u32 int_inf5_synd_dis : 1;
> +               u32 dlb_cor_alarm_enable : 1;
> +               u32 cfg_64bytes_qe_ldb_cq_mode : 1;
> +               u32 cfg_64bytes_qe_dir_cq_mode : 1;
> +               u32 pad_write_ldb : 1;
> +               u32 pad_write_dir : 1;
> +               u32 pad_first_write_ldb : 1;
> +               u32 pad_first_write_dir : 1;
> +               u32 rsvz0 : 9;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
> +#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
> +union dlb2_chp_dir_cq_intr_armed0 {
> +       struct {
> +               u32 armed : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
> +#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
> +union dlb2_chp_dir_cq_intr_armed1 {
> +       struct {
> +               u32 armed : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
> +#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
> +union dlb2_chp_cfg_dir_cq_timer_ctl {
> +       struct {
> +               u32 sample_interval : 8;
> +               u32 enb : 1;
> +               u32 rsvz0 : 23;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
> +#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
> +union dlb2_chp_cfg_dir_wdto_0 {
> +       struct {
> +               u32 wdto : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
> +#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
> +union dlb2_chp_cfg_dir_wdto_1 {
> +       struct {
> +               u32 wdto : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
> +#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
> +union dlb2_chp_cfg_dir_wd_disable0 {
> +       struct {
> +               u32 wd_disable : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
> +#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
> +union dlb2_chp_cfg_dir_wd_disable1 {
> +       struct {
> +               u32 wd_disable : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
> +#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
> +union dlb2_chp_cfg_dir_wd_enb_interval {
> +       struct {
> +               u32 sample_interval : 28;
> +               u32 enb : 1;
> +               u32 rsvz0 : 3;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
> +#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
> +union dlb2_chp_cfg_dir_wd_threshold {
> +       struct {
> +               u32 wd_threshold : 8;
> +               u32 rsvz0 : 24;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
> +#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
> +union dlb2_chp_ldb_cq_intr_armed0 {
> +       struct {
> +               u32 armed : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
> +#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
> +union dlb2_chp_ldb_cq_intr_armed1 {
> +       struct {
> +               u32 armed : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
> +#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
> +union dlb2_chp_cfg_ldb_cq_timer_ctl {
> +       struct {
> +               u32 sample_interval : 8;
> +               u32 enb : 1;
> +               u32 rsvz0 : 23;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
> +#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
> +union dlb2_chp_cfg_ldb_wdto_0 {
> +       struct {
> +               u32 wdto : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
> +#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
> +union dlb2_chp_cfg_ldb_wdto_1 {
> +       struct {
> +               u32 wdto : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
> +#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
> +union dlb2_chp_cfg_ldb_wd_disable0 {
> +       struct {
> +               u32 wd_disable : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
> +#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
> +union dlb2_chp_cfg_ldb_wd_disable1 {
> +       struct {
> +               u32 wd_disable : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
> +#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
> +union dlb2_chp_cfg_ldb_wd_enb_interval {
> +       struct {
> +               u32 sample_interval : 28;
> +               u32 enb : 1;
> +               u32 rsvz0 : 3;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
> +#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
> +union dlb2_chp_cfg_ldb_wd_threshold {
> +       struct {
> +               u32 wd_threshold : 8;
> +               u32 rsvz0 : 24;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
> +#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
> +union dlb2_chp_ctrl_diag_02 {
> +       struct {
> +               u32 egress_credit_status_empty : 1;
> +               u32 egress_credit_status_afull : 1;
> +               u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
> +               u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
> +               u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
> +               u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
> +               u32 chp_lsp_tok_pipe_credit_status_empty : 1;
> +               u32 chp_lsp_tok_pipe_credit_status_afull : 1;
> +               u32 chp_rop_pipe_credit_status_empty : 1;
> +               u32 chp_rop_pipe_credit_status_afull : 1;
> +               u32 qed_to_cq_pipe_credit_status_empty : 1;
> +               u32 qed_to_cq_pipe_credit_status_afull : 1;
> +               u32 egress_lsp_token_credit_status_empty : 1;
> +               u32 egress_lsp_token_credit_status_afull : 1;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
> +union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
> +union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
> +union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
> +union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_DP_DIR_CSR_CTRL 0x54000010
> +#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
> +union dlb2_dp_dir_csr_ctrl {
> +       struct {
> +               u32 int_cor_alarm_dis : 1;
> +               u32 int_cor_synd_dis : 1;
> +               u32 int_uncr_alarm_dis : 1;
> +               u32 int_unc_synd_dis : 1;
> +               u32 int_inf0_alarm_dis : 1;
> +               u32 int_inf0_synd_dis : 1;
> +               u32 int_inf1_alarm_dis : 1;
> +               u32 int_inf1_synd_dis : 1;
> +               u32 int_inf2_alarm_dis : 1;
> +               u32 int_inf2_synd_dis : 1;
> +               u32 int_inf3_alarm_dis : 1;
> +               u32 int_inf3_synd_dis : 1;
> +               u32 int_inf4_alarm_dis : 1;
> +               u32 int_inf4_synd_dis : 1;
> +               u32 int_inf5_alarm_dis : 1;
> +               u32 int_inf5_synd_dis : 1;
> +               u32 rsvz0 : 16;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
> +       (0x96000000 + (x) * 0x4)
> +#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
> +union dlb2_ro_pipe_grp_0_slt_shft {
> +       struct {
> +               u32 change : 10;
> +               u32 rsvd0 : 22;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
> +       (0x96010000 + (x) * 0x4)
> +#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
> +union dlb2_ro_pipe_grp_1_slt_shft {
> +       struct {
> +               u32 change : 10;
> +               u32 rsvd0 : 22;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
> +#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
> +union dlb2_ro_pipe_grp_sn_mode {
> +       struct {
> +               u32 sn_mode_0 : 3;
> +               u32 rszv0 : 5;
> +               u32 sn_mode_1 : 3;
> +               u32 rszv1 : 21;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
> +#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
> +union dlb2_ro_pipe_cfg_ctrl_general_0 {
> +       struct {
> +               u32 unit_single_step_mode : 1;
> +               u32 rr_en : 1;
> +               u32 rszv0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ2PRIOV(x) \
> +       (0xa0000000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ2PRIOV_RST 0x0
> +union dlb2_lsp_cq2priov {
> +       struct {
> +               u32 prio : 24;
> +               u32 v : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ2QID0(x) \
> +       (0xa0080000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ2QID0_RST 0x0
> +union dlb2_lsp_cq2qid0 {
> +       struct {
> +               u32 qid_p0 : 7;
> +               u32 rsvd3 : 1;
> +               u32 qid_p1 : 7;
> +               u32 rsvd2 : 1;
> +               u32 qid_p2 : 7;
> +               u32 rsvd1 : 1;
> +               u32 qid_p3 : 7;
> +               u32 rsvd0 : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ2QID1(x) \
> +       (0xa0100000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ2QID1_RST 0x0
> +union dlb2_lsp_cq2qid1 {
> +       struct {
> +               u32 qid_p4 : 7;
> +               u32 rsvd3 : 1;
> +               u32 qid_p5 : 7;
> +               u32 rsvd2 : 1;
> +               u32 qid_p6 : 7;
> +               u32 rsvd1 : 1;
> +               u32 qid_p7 : 7;
> +               u32 rsvd0 : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_DIR_DSBL(x) \
> +       (0xa0180000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
> +union dlb2_lsp_cq_dir_dsbl {
> +       struct {
> +               u32 disabled : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
> +       (0xa0200000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
> +union dlb2_lsp_cq_dir_tkn_cnt {
> +       struct {
> +               u32 count : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
> +       (0xa0280000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
> +union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
> +       struct {
> +               u32 token_depth_select : 4;
> +               u32 disable_wb_opt : 1;
> +               u32 ignore_depth : 1;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
> +       (0xa0300000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
> +union dlb2_lsp_cq_dir_tot_sch_cntl {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
> +       (0xa0380000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
> +union dlb2_lsp_cq_dir_tot_sch_cnth {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_DSBL(x) \
> +       (0xa0400000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
> +union dlb2_lsp_cq_ldb_dsbl {
> +       struct {
> +               u32 disabled : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
> +       (0xa0480000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
> +union dlb2_lsp_cq_ldb_infl_cnt {
> +       struct {
> +               u32 count : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
> +       (0xa0500000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
> +union dlb2_lsp_cq_ldb_infl_lim {
> +       struct {
> +               u32 limit : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
> +       (0xa0580000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
> +union dlb2_lsp_cq_ldb_tkn_cnt {
> +       struct {
> +               u32 token_count : 11;
> +               u32 rsvd0 : 21;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
> +       (0xa0600000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
> +union dlb2_lsp_cq_ldb_tkn_depth_sel {
> +       struct {
> +               u32 token_depth_select : 4;
> +               u32 ignore_depth : 1;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
> +       (0xa0680000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
> +union dlb2_lsp_cq_ldb_tot_sch_cntl {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
> +       (0xa0700000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
> +union dlb2_lsp_cq_ldb_tot_sch_cnth {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
> +       (0xa0780000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
> +union dlb2_lsp_qid_dir_max_depth {
> +       struct {
> +               u32 depth : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
> +       (0xa0800000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
> +union dlb2_lsp_qid_dir_tot_enq_cntl {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
> +       (0xa0880000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
> +union dlb2_lsp_qid_dir_tot_enq_cnth {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
> +       (0xa0900000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
> +union dlb2_lsp_qid_dir_enqueue_cnt {
> +       struct {
> +               u32 count : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
> +       (0xa0980000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
> +union dlb2_lsp_qid_dir_depth_thrsh {
> +       struct {
> +               u32 thresh : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
> +       (0xa0a00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
> +union dlb2_lsp_qid_aqed_active_cnt {
> +       struct {
> +               u32 count : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
> +       (0xa0a80000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
> +union dlb2_lsp_qid_aqed_active_lim {
> +       struct {
> +               u32 limit : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
> +       (0xa0b00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
> +union dlb2_lsp_qid_atm_tot_enq_cntl {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
> +       (0xa0b80000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
> +union dlb2_lsp_qid_atm_tot_enq_cnth {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
> +       (0xa0c00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
> +union dlb2_lsp_qid_atq_enqueue_cnt {
> +       struct {
> +               u32 count : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
> +       (0xa0c80000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
> +union dlb2_lsp_qid_ldb_enqueue_cnt {
> +       struct {
> +               u32 count : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
> +       (0xa0d00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
> +union dlb2_lsp_qid_ldb_infl_cnt {
> +       struct {
> +               u32 count : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
> +       (0xa0d80000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
> +union dlb2_lsp_qid_ldb_infl_lim {
> +       struct {
> +               u32 limit : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID2CQIDIX_00(x) \
> +       (0xa0e00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
> +#define DLB2_LSP_QID2CQIDIX(x, y) \
> +       (DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
> +#define DLB2_LSP_QID2CQIDIX_NUM 16
> +union dlb2_lsp_qid2cqidix_00 {
> +       struct {
> +               u32 cq_p0 : 8;
> +               u32 cq_p1 : 8;
> +               u32 cq_p2 : 8;
> +               u32 cq_p3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID2CQIDIX2_00(x) \
> +       (0xa1600000 + (x) * 0x1000)
> +#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
> +#define DLB2_LSP_QID2CQIDIX2(x, y) \
> +       (DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
> +#define DLB2_LSP_QID2CQIDIX2_NUM 16
> +union dlb2_lsp_qid2cqidix2_00 {
> +       struct {
> +               u32 cq_p0 : 8;
> +               u32 cq_p1 : 8;
> +               u32 cq_p2 : 8;
> +               u32 cq_p3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
> +       (0xa1e00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
> +union dlb2_lsp_qid_ldb_replay_cnt {
> +       struct {
> +               u32 count : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
> +       (0xa1f00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
> +union dlb2_lsp_qid_naldb_max_depth {
> +       struct {
> +               u32 depth : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
> +       (0xa1f80000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
> +union dlb2_lsp_qid_naldb_tot_enq_cntl {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
> +       (0xa2000000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
> +union dlb2_lsp_qid_naldb_tot_enq_cnth {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
> +       (0xa2080000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
> +union dlb2_lsp_qid_atm_depth_thrsh {
> +       struct {
> +               u32 thresh : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
> +       (0xa2100000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
> +union dlb2_lsp_qid_naldb_depth_thrsh {
> +       struct {
> +               u32 thresh : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_ATM_ACTIVE(x) \
> +       (0xa2180000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
> +union dlb2_lsp_qid_atm_active {
> +       struct {
> +               u32 count : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
> +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
> +union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
> +       struct {
> +               u32 pri0_weight : 8;
> +               u32 pri1_weight : 8;
> +               u32 pri2_weight : 8;
> +               u32 pri3_weight : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
> +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
> +union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
> +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
> +union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
> +       struct {
> +               u32 pri0_weight : 8;
> +               u32 pri1_weight : 8;
> +               u32 pri2_weight : 8;
> +               u32 pri3_weight : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
> +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
> +union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
> +#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
> +union dlb2_lsp_ldb_sched_ctrl {
> +       struct {
> +               u32 cq : 8;
> +               u32 qidix : 3;
> +               u32 value : 1;
> +               u32 nalb_haswork_v : 1;
> +               u32 rlist_haswork_v : 1;
> +               u32 slist_haswork_v : 1;
> +               u32 inflight_ok_v : 1;
> +               u32 aqed_nfull_v : 1;
> +               u32 rsvz0 : 15;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
> +#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
> +union dlb2_lsp_dir_sch_cnt_l {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
> +#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
> +union dlb2_lsp_dir_sch_cnt_h {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
> +#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
> +union dlb2_lsp_ldb_sch_cnt_l {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
> +#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
> +union dlb2_lsp_ldb_sch_cnt_h {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
> +#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
> +union dlb2_lsp_cfg_shdw_ctrl {
> +       struct {
> +               u32 transfer : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
> +       (0xa4000074 + (x) * 4)
> +#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
> +union dlb2_lsp_cfg_shdw_range_cos {
> +       struct {
> +               u32 bw_range : 9;
> +               u32 rsvz0 : 22;
> +               u32 no_extra_credit : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
> +#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
> +union dlb2_lsp_cfg_ctrl_general_0 {
> +       struct {
> +               u32 disab_atq_empty_arb : 1;
> +               u32 inc_tok_unit_idle : 1;
> +               u32 disab_rlist_pri : 1;
> +               u32 inc_cmp_unit_idle : 1;
> +               u32 rsvz0 : 2;
> +               u32 dir_single_op : 1;
> +               u32 dir_half_bw : 1;
> +               u32 dir_single_out : 1;
> +               u32 dir_disab_multi : 1;
> +               u32 atq_single_op : 1;
> +               u32 atq_half_bw : 1;
> +               u32 atq_single_out : 1;
> +               u32 atq_disab_multi : 1;
> +               u32 dirrpl_single_op : 1;
> +               u32 dirrpl_half_bw : 1;
> +               u32 dirrpl_single_out : 1;
> +               u32 lbrpl_single_op : 1;
> +               u32 lbrpl_half_bw : 1;
> +               u32 lbrpl_single_out : 1;
> +               u32 ldb_single_op : 1;
> +               u32 ldb_half_bw : 1;
> +               u32 ldb_disab_multi : 1;
> +               u32 atm_single_sch : 1;
> +               u32 atm_single_cmp : 1;
> +               u32 ldb_ce_tog_arb : 1;
> +               u32 rsvz1 : 1;
> +               u32 smon0_valid_sel : 2;
> +               u32 smon0_value_sel : 1;
> +               u32 smon0_compare_sel : 2;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
> +#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
> +union dlb2_cfg_mstr_diag_reset_sts {
> +       struct {
> +               u32 chp_pf_reset_done : 1;
> +               u32 rop_pf_reset_done : 1;
> +               u32 lsp_pf_reset_done : 1;
> +               u32 nalb_pf_reset_done : 1;
> +               u32 ap_pf_reset_done : 1;
> +               u32 dp_pf_reset_done : 1;
> +               u32 qed_pf_reset_done : 1;
> +               u32 dqed_pf_reset_done : 1;
> +               u32 aqed_pf_reset_done : 1;
> +               u32 sys_pf_reset_done : 1;
> +               u32 pf_reset_active : 1;
> +               u32 flrsm_state : 7;
> +               u32 rsvd0 : 13;
> +               u32 dlb_proc_reset_done : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
> +#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
> +union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
> +       struct {
> +               u32 chp_pipeidle : 1;
> +               u32 rop_pipeidle : 1;
> +               u32 lsp_pipeidle : 1;
> +               u32 nalb_pipeidle : 1;
> +               u32 ap_pipeidle : 1;
> +               u32 dp_pipeidle : 1;
> +               u32 qed_pipeidle : 1;
> +               u32 dqed_pipeidle : 1;
> +               u32 aqed_pipeidle : 1;
> +               u32 sys_pipeidle : 1;
> +               u32 chp_unit_idle : 1;
> +               u32 rop_unit_idle : 1;
> +               u32 lsp_unit_idle : 1;
> +               u32 nalb_unit_idle : 1;
> +               u32 ap_unit_idle : 1;
> +               u32 dp_unit_idle : 1;
> +               u32 qed_unit_idle : 1;
> +               u32 dqed_unit_idle : 1;
> +               u32 aqed_unit_idle : 1;
> +               u32 sys_unit_idle : 1;
> +               u32 rsvd1 : 4;
> +               u32 mstr_cfg_ring_idle : 1;
> +               u32 mstr_cfg_mstr_idle : 1;
> +               u32 mstr_flr_clkreq_b : 1;
> +               u32 mstr_proc_idle : 1;
> +               u32 mstr_proc_idle_masked : 1;
> +               u32 rsvd0 : 2;
> +               u32 dlb_func_idle : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
> +#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
> +union dlb2_cfg_mstr_cfg_pm_status {
> +       struct {
> +               u32 prochot : 1;
> +               u32 pgcb_dlb_idle : 1;
> +               u32 pgcb_dlb_pg_rdy_ack_b : 1;
> +               u32 pmsm_pgcb_req_b : 1;
> +               u32 pgbc_pmc_pg_req_b : 1;
> +               u32 pmc_pgcb_pg_ack_b : 1;
> +               u32 pmc_pgcb_fet_en_b : 1;
> +               u32 pgcb_fet_en_b : 1;
> +               u32 rsvz0 : 1;
> +               u32 rsvz1 : 1;
> +               u32 fuse_force_on : 1;
> +               u32 fuse_proc_disable : 1;
> +               u32 rsvz2 : 1;
> +               u32 rsvz3 : 1;
> +               u32 pm_fsm_d0tod3_ok : 1;
> +               u32 pm_fsm_d3tod0_ok : 1;
> +               u32 dlb_in_d3 : 1;
> +               u32 rsvz4 : 7;
> +               u32 pmsm : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
> +#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
> +union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
> +       struct {
> +               u32 disable : 1;
> +               u32 rsvz0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
> +#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
> +       (0x1000 + (x) * 0x4)
> +#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
> +union dlb2_func_vf_vf2pf_mailbox {
> +       struct {
> +               u32 msg : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
> +#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
> +#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
> +union dlb2_func_vf_vf2pf_mailbox_isr {
> +       struct {
> +               u32 isr : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
> +#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
> +       (0x2000 + (x) * 0x4)
> +#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
> +union dlb2_func_vf_pf2vf_mailbox {
> +       struct {
> +               u32 msg : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
> +#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
> +union dlb2_func_vf_pf2vf_mailbox_isr {
> +       struct {
> +               u32 pf_isr : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
> +#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
> +union dlb2_func_vf_vf_msi_isr_pend {
> +       struct {
> +               u32 isr_pend : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
> +#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
> +union dlb2_func_vf_vf_reset_in_progress {
> +       struct {
> +               u32 reset_in_progress : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
> +#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
> +union dlb2_func_vf_vf_msi_isr {
> +       struct {
> +               u32 vf_msi_isr : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#endif /* __DLB2_REGS_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
> new file mode 100644
> index 0000000..6de8b95
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
> @@ -0,0 +1,274 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include "dlb2_user.h"
> +
> +#include "dlb2_hw_types.h"
> +#include "dlb2_mbox.h"
> +#include "dlb2_osdep.h"
> +#include "dlb2_osdep_bitmap.h"
> +#include "dlb2_osdep_types.h"
> +#include "dlb2_regs.h"
> +#include "dlb2_resource.h"
> +
> +static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
> +{
> +       int i;
> +
> +       dlb2_list_init_head(&domain->used_ldb_queues);
> +       dlb2_list_init_head(&domain->used_dir_pq_pairs);
> +       dlb2_list_init_head(&domain->avail_ldb_queues);
> +       dlb2_list_init_head(&domain->avail_dir_pq_pairs);
> +
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               dlb2_list_init_head(&domain->used_ldb_ports[i]);
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               dlb2_list_init_head(&domain->avail_ldb_ports[i]);
> +}
> +
> +static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
> +{
> +       int i;
> +
> +       dlb2_list_init_head(&rsrc->avail_domains);
> +       dlb2_list_init_head(&rsrc->used_domains);
> +       dlb2_list_init_head(&rsrc->avail_ldb_queues);
> +       dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
> +
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
> +}
> +
> +void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
> +{
> +       union dlb2_chp_cfg_chp_csr_ctrl r0;
> +
> +       r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
> +
> +       r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
> +
> +       DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
> +}
> +
> +int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
> +                             struct dlb2_get_num_resources_args *arg,
> +                             bool vdev_req,
> +                             unsigned int vdev_id)
> +{
> +       struct dlb2_function_resources *rsrcs;
> +       struct dlb2_bitmap *map;
> +       int i;
> +
> +       if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
> +               return -EINVAL;
> +
> +       if (vdev_req)
> +               rsrcs = &hw->vdev[vdev_id];
> +       else
> +               rsrcs = &hw->pf;
> +
> +       arg->num_sched_domains = rsrcs->num_avail_domains;
> +
> +       arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
> +
> +       arg->num_ldb_ports = 0;
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
> +
> +       arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
> +       arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
> +       arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
> +       arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
> +
> +       arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
> +
> +       arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
> +
> +       map = rsrcs->avail_hist_list_entries;
> +
> +       arg->num_hist_list_entries = dlb2_bitmap_count(map);
> +
> +       arg->max_contiguous_hist_list_entries =
> +               dlb2_bitmap_longest_set_range(map);
> +
> +       arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
> +
> +       arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
> +
> +       return 0;
> +}
> +
> +void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
> +{
> +       union dlb2_chp_cfg_chp_csr_ctrl r0;
> +
> +       r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
> +
> +       r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
> +
> +       DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
> +}
> +
> +void dlb2_resource_free(struct dlb2_hw *hw)
> +{
> +       int i;
> +
> +       if (hw->pf.avail_hist_list_entries)
> +               dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
> +
> +       for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
> +               if (hw->vdev[i].avail_hist_list_entries)
> +                       dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
> +       }
> +}
> +
> +int dlb2_resource_init(struct dlb2_hw *hw)
> +{
> +       struct dlb2_list_entry *list;
> +       unsigned int i;
> +       int ret;
> +
> +       /*
> +        * For optimal load-balancing, ports that map to one or more QIDs in
> +        * common should not be in numerical sequence. This is application
> +        * dependent, but the driver interleaves port IDs as much as possible
> +        * to reduce the likelihood of this. This initial allocation maximizes
> +        * the average distance between an ID and its immediate neighbors (i.e.
> +        * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
> +        * 3, etc.).
> +        */
> +       u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
> +               0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
> +               16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
> +               32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
> +               48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
> +       };
> +
> +       /* Zero-out resource tracking data structures */
> +       memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
> +       memset(&hw->pf, 0, sizeof(hw->pf));
> +
> +       dlb2_init_fn_rsrc_lists(&hw->pf);
> +
> +       for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
> +               memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
> +               dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
> +       }
> +
> +       for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
> +               memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
> +               dlb2_init_domain_rsrc_lists(&hw->domains[i]);
> +               hw->domains[i].parent_func = &hw->pf;
> +       }
> +
> +       /* Give all resources to the PF driver */
> +       hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
> +       for (i = 0; i < hw->pf.num_avail_domains; i++) {
> +               list = &hw->domains[i].func_list;
> +
> +               dlb2_list_add(&hw->pf.avail_domains, list);
> +       }
> +
> +       hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
> +       for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
> +               list = &hw->rsrcs.ldb_queues[i].func_list;
> +
> +               dlb2_list_add(&hw->pf.avail_ldb_queues, list);
> +       }
> +
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               hw->pf.num_avail_ldb_ports[i] =
> +                       DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
> +
> +       for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
> +               int cos_id = i >> DLB2_NUM_COS_DOMAINS;
> +               struct dlb2_ldb_port *port;
> +
> +               port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
> +
> +               dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
> +                             &port->func_list);
> +       }
> +
> +       hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
> +       for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
> +               list = &hw->rsrcs.dir_pq_pairs[i].func_list;
> +
> +               dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
> +       }
> +
> +       hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
> +       hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
> +       hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
> +
> +       ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
> +                               DLB2_MAX_NUM_HIST_LIST_ENTRIES);
> +       if (ret)
> +               goto unwind;
> +
> +       ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
> +       if (ret)
> +               goto unwind;
> +
> +       for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
> +               ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
> +                                       DLB2_MAX_NUM_HIST_LIST_ENTRIES);
> +               if (ret)
> +                       goto unwind;
> +
> +               ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
> +               if (ret)
> +                       goto unwind;
> +       }
> +
> +       /* Initialize the hardware resource IDs */
> +       for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
> +               hw->domains[i].id.phys_id = i;
> +               hw->domains[i].id.vdev_owned = false;
> +       }
> +
> +       for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
> +               hw->rsrcs.ldb_queues[i].id.phys_id = i;
> +               hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
> +       }
> +
> +       for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
> +               hw->rsrcs.ldb_ports[i].id.phys_id = i;
> +               hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
> +       }
> +
> +       for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
> +               hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
> +               hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
> +       }
> +
> +       for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
> +               hw->rsrcs.sn_groups[i].id = i;
> +               /* Default mode (0) is 64 sequence numbers per queue */
> +               hw->rsrcs.sn_groups[i].mode = 0;
> +               hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
> +               hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
> +       }
> +
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
> +
> +       return 0;
> +
> +unwind:
> +       dlb2_resource_free(hw);
> +
> +       return ret;
> +}
> +
> +void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
> +{
> +       union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
> +
> +       r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
> +
> +       r0.field.disable = 0;
> +
> +       DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
> +}
> diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
> new file mode 100644
> index 0000000..503fdf3
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
> @@ -0,0 +1,1913 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_RESOURCE_H
> +#define __DLB2_RESOURCE_H
> +
> +#include "dlb2_user.h"
> +
> +#include "dlb2_hw_types.h"
> +#include "dlb2_osdep_types.h"
> +
> +/**
> + * dlb2_resource_init() - initialize the device
> + * @hw: pointer to struct dlb2_hw.
> + *
> + * This function initializes the device's software state (pointed to by the hw
> + * argument) and programs global scheduling QoS registers. This function should
> + * be called during driver initialization.
> + *
> + * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
> + * device is reset.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + */
> +int dlb2_resource_init(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_resource_free() - free device state memory
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function frees software state pointed to by dlb2_hw. This function
> + * should be called when resetting the device or unloading the driver.
> + */
> +void dlb2_resource_free(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_resource_reset() - reset in-use resources to their initial state
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function resets in-use resources, and makes them available for use.
> + * All resources go back to their owning function, whether a PF or a VF.
> + */
> +void dlb2_resource_reset(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_create_sched_domain() - create a scheduling domain
> + * @hw: dlb2_hw handle for a particular device.
> + * @args: scheduling domain creation arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function creates a scheduling domain containing the resources specified
> + * in args. The individual resources (queues, ports, credits) can be configured
> + * after creating a scheduling domain.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the domain ID.
> + *
> + * resp->id contains a virtual ID if vdev_request is true.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, or the requested domain name
> + *         is already in use.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
> +                               struct dlb2_create_sched_domain_args *args,
> +                               struct dlb2_cmd_response *resp,
> +                               bool vdev_request,
> +                               unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_create_ldb_queue() - create a load-balanced queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: queue creation arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function creates a load-balanced queue.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the queue ID.
> + *
> + * resp->id contains a virtual ID if vdev_request is true.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, the domain is not configured,
> + *         the domain has already been started, or the requested queue name is
> + *         already in use.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_create_ldb_queue_args *args,
> +                            struct dlb2_cmd_response *resp,
> +                            bool vdev_request,
> +                            unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_create_dir_queue() - create a directed queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: queue creation arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function creates a directed queue.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the queue ID.
> + *
> + * resp->id contains a virtual ID if vdev_request is true.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, the domain is not configured,
> + *         or the domain has already been started.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_create_dir_queue_args *args,
> +                            struct dlb2_cmd_response *resp,
> +                            bool vdev_request,
> +                            unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_create_dir_port() - create a directed port
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port creation arguments.
> + * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function creates a directed port.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the port ID.
> + *
> + * resp->id contains a virtual ID if vdev_request is true.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
> + *         pointer address is not properly aligned, the domain is not
> + *         configured, or the domain has already been started.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_create_dir_port_args *args,
> +                           uintptr_t cq_dma_base,
> +                           struct dlb2_cmd_response *resp,
> +                           bool vdev_request,
> +                           unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_create_ldb_port() - create a load-balanced port
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port creation arguments.
> + * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function creates a load-balanced port.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the port ID.
> + *
> + * resp->id contains a virtual ID if vdev_request is true.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
> + *         pointer address is not properly aligned, the domain is not
> + *         configured, or the domain has already been started.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_create_ldb_port_args *args,
> +                           uintptr_t cq_dma_base,
> +                           struct dlb2_cmd_response *resp,
> +                           bool vdev_request,
> +                           unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_start_domain() - start a scheduling domain
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: start domain arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function starts a scheduling domain, which allows applications to send
> + * traffic through it. Once a domain is started, its resources can no longer be
> + * configured (besides QID remapping and port enable/disable).
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - the domain is not configured, or the domain is already started.
> + */
> +int dlb2_hw_start_domain(struct dlb2_hw *hw,
> +                        u32 domain_id,
> +                        struct dlb2_start_domain_args *args,
> +                        struct dlb2_cmd_response *resp,
> +                        bool vdev_request,
> +                        unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: map QID arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to schedule QEs from the specified queue
> + * to the specified port. Each load-balanced port can be mapped to up to 8
> + * queues; each load-balanced queue can potentially map to all the
> + * load-balanced ports.
> + *
> + * A successful return does not necessarily mean the mapping was configured. If
> + * this function is unable to immediately map the queue to the port, it will
> + * add the requested operation to a per-port list of pending map/unmap
> + * operations, and (if it's not already running) launch a kernel thread that
> + * periodically attempts to process all pending operations. In a sense, this is
> + * an asynchronous function.
> + *
> + * This asynchronicity creates two views of the state of hardware: the actual
> + * hardware state and the requested state (as if every request completed
> + * immediately). If there are any pending map/unmap operations, the requested
> + * state will differ from the actual state. All validation is performed with
> + * respect to the pending state; for instance, if there are 8 pending map
> + * operations for port X, a request for a 9th will fail because a load-balanced
> + * port can only map up to 8 queues.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
> + *         the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_map_qid(struct dlb2_hw *hw,
> +                   u32 domain_id,
> +                   struct dlb2_map_qid_args *args,
> +                   struct dlb2_cmd_response *resp,
> +                   bool vdev_request,
> +                   unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: unmap QID arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to stop scheduling QEs from the specified
> + * queue to the specified port.
> + *
> + * A successful return does not necessarily mean the mapping was removed. If
> + * this function is unable to immediately unmap the queue from the port, it
> + * will add the requested operation to a per-port list of pending map/unmap
> + * operations, and (if it's not already running) launch a kernel thread that
> + * periodically attempts to process all pending operations. See
> + * dlb2_hw_map_qid() for more details.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
> + *         the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
> +                     u32 domain_id,
> +                     struct dlb2_unmap_qid_args *args,
> +                     struct dlb2_cmd_response *resp,
> +                     bool vdev_request,
> +                     unsigned int vdev_id);
> +
> +/**
> + * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function attempts to finish any outstanding unmap procedures.
> + * This function should be called by the kernel thread responsible for
> + * finishing map/unmap procedures.
> + *
> + * Return:
> + * Returns the number of procedures that weren't completed.
> + */
> +unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_finish_map_qid_procedures() - finish any pending map procedures
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function attempts to finish any outstanding map procedures.
> + * This function should be called by the kernel thread responsible for
> + * finishing map/unmap procedures.
> + *
> + * Return:
> + * Returns the number of procedures that weren't completed.
> + */
> +unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port enable arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to schedule QEs to a load-balanced port.
> + * Ports are enabled by default.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid or the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_enable_ldb_port_args *args,
> +                           struct dlb2_cmd_response *resp,
> +                           bool vdev_request,
> +                           unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port disable arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to stop scheduling QEs to a load-balanced
> + * port. Ports are enabled by default.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid or the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_disable_ldb_port_args *args,
> +                            struct dlb2_cmd_response *resp,
> +                            bool vdev_request,
> +                            unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port enable arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to schedule QEs to a directed port.
> + * Ports are enabled by default.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid or the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_enable_dir_port_args *args,
> +                           struct dlb2_cmd_response *resp,
> +                           bool vdev_request,
> +                           unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port disable arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to stop scheduling QEs to a directed port.
> + * Ports are enabled by default.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid or the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_disable_dir_port_args *args,
> +                            struct dlb2_cmd_response *resp,
> +                            bool vdev_request,
> +                            unsigned int vdev_id);
> +
> +/**
> + * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
> + *                                     interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: load-balanced port ID.
> + * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
> + *         else a value up to 64.
> + * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
> + * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
> + *     virtual port ID to a physical port ID. Ignored if mode is not MSI.
> + * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
> + * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
> + *     greater than 0.
> + *
> + * This function configures the DLB registers for load-balanced CQ's
> + * interrupts. This doesn't enable the CQ's interrupt; that can be done with
> + * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid.
> + */
> +int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
> +                                   int port_id,
> +                                   int vector,
> +                                   int mode,
> +                                   unsigned int vf,
> +                                   unsigned int owner_vf,
> +                                   u16 threshold);
> +
> +/**
> + * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: load-balanced port ID.
> + * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
> + *         else a value up to 64.
> + * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
> + * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
> + *     virtual port ID to a physical port ID. Ignored if mode is not MSI.
> + * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
> + * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
> + *     greater than 0.
> + *
> + * This function configures the DLB registers for directed CQ's interrupts.
> + * This doesn't enable the CQ's interrupt; that can be done with
> + * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid.
> + */
> +int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
> +                                   int port_id,
> +                                   int vector,
> +                                   int mode,
> +                                   unsigned int vf,
> +                                   unsigned int owner_vf,
> +                                   u16 threshold);
> +
> +/**
> + * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + */
> +void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + */
> +void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
> + *
> + * This function configures the hardware to use either packed or compressed
> + * mode. This function should not be called if using MSI interrupts.
> + */
> +void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
> +
> +/**
> + * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + * @vector: interrupt vector.
> + *
> + * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
> + * acked in dlb2_ack_compressed_cq_intr().
> + */
> +void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
> +
> +/**
> + * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: port ID
> + * @is_ldb: true for load-balanced port, false for a directed port
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function arms the CQ's interrupt. The CQ must be configured prior to
> + * calling this function.
> + *
> + * The function does no parameter validation; that is the caller's
> + * responsibility.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return: returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - Invalid port ID.
> + */
> +int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
> +                         int port_id,
> +                         bool is_ldb,
> +                         bool vdev_request,
> +                         unsigned int vdev_id);
> +
> +/**
> + * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
> + * @hw: dlb2_hw handle for a particular device.
> + * @ldb_interrupts: 2-entry array of u32 bitmaps
> + * @dir_interrupts: 4-entry array of u32 bitmaps
> + *
> + * This function can be called from a compressed CQ interrupt handler to
> + * determine which CQ interrupts have fired. The caller should take appropriate
> + * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
> + * with dlb2_ack_compressed_cq_intr().
> + */
> +void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
> +                                        u32 *ldb_interrupts,
> +                                        u32 *dir_interrupts);
> +
> +/**
> + * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + * @ldb_interrupts: 2-entry array of u32 bitmaps
> + * @dir_interrupts: 4-entry array of u32 bitmaps
> + *
> + * This function ACKs compressed CQ interrupts. Its arguments should be the
> + * same ones passed to dlb2_read_compressed_cq_intr_status().
> + */
> +void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
> +                                u32 *ldb_interrupts,
> +                                u32 *dir_interrupts);
> +
> +/**
> + * dlb2_read_vf_intr_status() - read the VF interrupt status register
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function can be called from a VF's interrupt handler to determine
> + * which interrupts have fired. The first 31 bits correspond to CQ interrupt
> + * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
> + *
> + * Return:
> + * Returns a bit vector indicating which interrupt vectors are active.
> + */
> +u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_ack_vf_intr_status() - ack VF interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + * @interrupts: 32-bit bitmap
> + *
> + * This function ACKs a VF's interrupts. Its interrupts argument should be the
> + * value returned by dlb2_read_vf_intr_status().
> + */
> +void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
> +
> +/**
> + * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + * @interrupts: 32-bit bitmap
> + *
> + * This function clears the VF's MSI interrupt pending register. Its interrupts
> + * argument should be contain the MSI vectors to ACK. For example, if MSI MME
> + * is in mode 0, then one bit 0 should ever be set.
> + */
> +void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
> +
> +/**
> + * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * When done processing the PF mailbox request, this function unsets
> + * the PF's mailbox ISR register.
> + */
> +void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
> + *                                     vdevs
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * When the vdev->PF ISR fires, this function can be called to determine which
> + * vdev(s) are requesting service. This bitvector must be passed to
> + * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
> + * vdevs.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns a bit vector indicating which VFs (0-15) have requested service.
> + */
> +u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
> + *
> + * When done processing all VF mailbox requests, this function unsets the VF's
> + * mailbox ISR register.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
> +
> +/**
> + * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
> + *                                 FLR
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * When the VF FLR ISR fires, this function can be called to determine which
> + * VF(s) are requesting FLRs. This bitvector must passed to
> + * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
> + *
> + * Return:
> + * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
> + */
> +u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
> + * @hw: dlb2_hw handle for a particular device.
> + * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
> + *
> + * When done processing all VF FLR requests, this function unsets the VF's FLR
> + * ISR register.
> + */
> +void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
> +
> +/**
> + * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
> + * @hw: dlb2_hw handle for a particular device.
> + * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
> + * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
> + *
> + * When done processing all VF requests, this function communicates to the
> + * hardware that processing is complete.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
> +                            u32 mbox_bitvec,
> +                            u32 flr_bitvec);
> +
> +/**
> + * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function reads the watchdog timer interrupt cause registers to
> + * determine which port(s) had a watchdog timeout, and notifies the
> + * application(s) that own the port(s).
> + */
> +void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_process_alarm_interrupt() - process an alarm interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function reads and logs the alarm syndrome, then acks the interrupt.
> + * This function should be called from the alarm interrupt handler when
> + * interrupt vector DLB2_INT_ALARM fires.
> + */
> +void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function reads the alarm syndrome, logs it, notifies user-space, and
> + * acks the interrupt. This function should be called from the alarm interrupt
> + * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
> + *
> + * Return:
> + * Returns true if an ingress error interrupt occurred, false otherwise
> + */
> +bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @group_id: sequence number group ID.
> + *
> + * This function returns the configured number of sequence numbers per queue
> + * for the specified group.
> + *
> + * Return:
> + * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
> + */
> +int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
> +                                   unsigned int group_id);
> +
> +/**
> + * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
> + * @hw: dlb2_hw handle for a particular device.
> + * @group_id: sequence number group ID.
> + *
> + * This function returns the group's number of in-use slots (i.e. load-balanced
> + * queues using the specified group).
> + *
> + * Return:
> + * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
> + */
> +int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
> +                                            unsigned int group_id);
> +
> +/**
> + * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @group_id: sequence number group ID.
> + * @val: requested amount of sequence numbers per queue.
> + *
> + * This function configures the group's number of sequence numbers per queue.
> + * val can be a power-of-two between 32 and 1024, inclusive. This setting can
> + * be configured until the first ordered load-balanced queue is configured, at
> + * which point the configuration is locked.
> + *
> + * Return:
> + * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
> + * ordered queue is configured.
> + */
> +int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
> +                                   unsigned int group_id,
> +                                   unsigned long val);
> +
> +/**
> + * dlb2_reset_domain() - reset a scheduling domain
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function resets and frees a DLB 2.0 scheduling domain and its associated
> + * resources.
> + *
> + * Pre-condition: the driver must ensure software has stopped sending QEs
> + * through this domain's producer ports before invoking this function, or
> + * undefined behavior will result.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, -1 otherwise.
> + *
> + * EINVAL - Invalid domain ID, or the domain is not configured.
> + * EFAULT - Internal error. (Possibly caused if software is the pre-condition
> + *         is not met.)
> + * ETIMEDOUT - Hardware component didn't reset in the expected time.
> + */
> +int dlb2_reset_domain(struct dlb2_hw *hw,
> +                     u32 domain_id,
> +                     bool vdev_request,
> +                     unsigned int vdev_id);
> +
> +/**
> + * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @port_id: indicates whether this request came from a VF.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns whether a load-balanced port is owned by a specified
> + * domain.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 if false, 1 if true, <0 otherwise.
> + *
> + * EINVAL - Invalid domain or port ID, or the domain is not configured.
> + */
> +int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
> +                                 u32 domain_id,
> +                                 u32 port_id,
> +                                 bool vdev_request,
> +                                 unsigned int vdev_id);
> +
> +/**
> + * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @port_id: indicates whether this request came from a VF.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns whether a directed port is owned by a specified
> + * domain.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 if false, 1 if true, <0 otherwise.
> + *
> + * EINVAL - Invalid domain or port ID, or the domain is not configured.
> + */
> +int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
> +                                 u32 domain_id,
> +                                 u32 port_id,
> +                                 bool vdev_request,
> +                                 unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_get_num_resources() - query the PCI function's available resources
> + * @hw: dlb2_hw handle for a particular device.
> + * @arg: pointer to resource counts.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns the number of available resources for the PF or for a
> + * VF.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
> + * invalid.
> + */
> +int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
> +                             struct dlb2_get_num_resources_args *arg,
> +                             bool vdev_request,
> +                             unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
> + * @hw: dlb2_hw handle for a particular device.
> + * @arg: pointer to resource counts.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns the number of resources in use by the PF or a VF. It
> + * fills in the fields that args points to, except the following:
> + * - max_contiguous_atomic_inflights
> + * - max_contiguous_hist_list_entries
> + * - max_contiguous_ldb_credits
> + * - max_contiguous_dir_credits
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
> + * invalid.
> + */
> +int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
> +                                  struct dlb2_get_num_resources_args *arg,
> +                                  bool vdev_request,
> +                                  unsigned int vdev_id);
> +
> +/**
> + * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
> + *                                    the PF
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function sends a VF->PF mailbox message. It is asynchronous, so it
> + * returns once the message is sent but potentially before the PF has processed
> + * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
> + * when the PF has finished processing the request.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
> + *                              request
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function returns a boolean indicating whether the PF has finished
> + * processing a VF->PF mailbox request. It should only be called after sending
> + * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_vf_flr_complete() - check the status of a VF FLR
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function returns a boolean indicating whether the PF has finished
> + * executing the VF FLR. It should only be called after setting the VF's FLR
> + * bit.
> + */
> +bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
> + *                                     vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + *
> + * This function sends a PF->vdev mailbox message. It is asynchronous, so it
> + * returns once the message is sent but potentially before the vdev has
> + * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
> + * determine when the vdev has finished processing the request.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
> +
> +/**
> + * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
> + *                            request
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + *
> + * This function returns a boolean indicating whether the vdev has finished
> + * processing a PF->vdev mailbox request. It should only be called after
> + * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
> +
> +/**
> + * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
> + * @hw: dlb2_hw handle for a particular device.
> + * @vf_id: VF ID.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies one of the PF's VF->PF mailboxes into the array pointed
> + * to by data.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
> + */
> +int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
> +                            unsigned int vf_id,
> +                            void *data,
> +                            int len);
> +
> +/**
> + * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
> + * @hw: dlb2_hw handle for a particular device.
> + * @vf_id: VF ID.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies one of the PF's VF->PF mailboxes into the array pointed
> + * to by data.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
> + */
> +int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
> +                             unsigned int vf_id,
> +                             void *data,
> +                             int len);
> +
> +/**
> + * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
> + * @hw: dlb2_hw handle for a particular device.
> + * @vf_id: VF ID.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the user-provided message data into of the PF's VF->PF
> + * mailboxes.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
> + */
> +int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
> +                              unsigned int vf_id,
> +                              void *data,
> +                              int len);
> +
> +/**
> + * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
> + * @hw: dlb2_hw handle for a particular device.
> + * @vf_id: VF ID.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the user-provided message data into of the PF's VF->PF
> + * mailboxes.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
> + */
> +int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
> +                             unsigned int vf_id,
> +                             void *data,
> +                             int len);
> +
> +/**
> + * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
> + * @hw: dlb2_hw handle for a particular device.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the VF's PF->VF mailbox into the array pointed to by
> + * data.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
> + */
> +int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
> +
> +/**
> + * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
> + * @hw: dlb2_hw handle for a particular device.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the VF's PF->VF mailbox into the array pointed to by
> + * data.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
> + */
> +int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
> +
> +/**
> + * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
> + * @hw: dlb2_hw handle for a particular device.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the user-provided message data into of the VF's PF->VF
> + * mailboxes.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
> + */
> +int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
> +
> +/**
> + * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
> + * @hw: dlb2_hw handle for a particular device.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the user-provided message data into of the VF's PF->VF
> + * mailboxes.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
> + */
> +int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
> +
> +/**
> + * dlb2_reset_vdev() - reset the hardware owned by a virtual device
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function resets the hardware owned by a vdev, by resetting the vdev's
> + * domains one by one.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function returns whether or not the vdev's resource assignments are
> + * locked. If locked, no resources can be added to or subtracted from the
> + * group.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_lock_vdev() - lock the vdev's resources
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function sets a flag indicating that the vdev is using its resources.
> + * When the vdev is locked, its resource assignment cannot be changed.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_unlock_vdev() - unlock the vdev's resources
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function unlocks the vdev's resource assignment, allowing it to be
> + * modified.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of scheduling domains to assign to this vdev
> + *
> + * This function assigns num scheduling domains to the specified vdev. If the
> + * vdev already has domains assigned, this existing assignment is adjusted
> + * accordingly.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of LDB queues to assign to this vdev
> + *
> + * This function assigns num LDB queues to the specified vdev. If the vdev
> + * already has LDB queues assigned, this existing assignment is adjusted
> + * accordingly.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of LDB ports to assign to this vdev
> + *
> + * This function assigns num LDB ports to the specified vdev. If the vdev
> + * already has LDB ports assigned, this existing assignment is adjusted
> + * accordingly.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @cos: class-of-service ID
> + * @num: number of LDB ports to assign to this vdev
> + *
> + * This function assigns num LDB ports from class-of-service cos to the
> + * specified vdev. If the vdev already has LDB ports from this class-of-service
> + * assigned, this existing assignment is adjusted accordingly.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
> +                                  u32 id,
> +                                  u32 cos,
> +                                  u32 num);
> +
> +/**
> + * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of DIR ports to assign to this vdev
> + *
> + * This function assigns num DIR ports to the specified vdev. If the vdev
> + * already has DIR ports assigned, this existing assignment is adjusted
> + * accordingly.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of LDB credit credits to assign to this vdev
> + *
> + * This function assigns num LDB credit to the specified vdev. If the vdev
> + * already has LDB credits assigned, this existing assignment is adjusted
> + * accordingly. vdevs are assigned a contiguous chunk of credits, so this
> + * function may fail if a sufficiently large contiguous chunk is not available.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of DIR credits to assign to this vdev
> + *
> + * This function assigns num DIR credit to the specified vdev. If the vdev
> + * already has DIR credits assigned, this existing assignment is adjusted
> + * accordingly. vdevs are assigned a contiguous chunk of credits, so this
> + * function may fail if a sufficiently large contiguous chunk is not available.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of history list entries to assign to this vdev
> + *
> + * This function assigns num history list entries to the specified vdev. If the
> + * vdev already has history list entries assigned, this existing assignment is
> + * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
> + * this function may fail if a sufficiently large contiguous chunk is not
> + * available.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of atomic inflights to assign to this vdev
> + *
> + * This function assigns num atomic inflights to the specified vdev. If the vdev
> + * already has atomic inflights assigned, this existing assignment is adjusted
> + * accordingly. vdevs are assigned a contiguous chunk of entries, so this
> + * function may fail if a sufficiently large contiguous chunk is not available.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function takes any resources currently assigned to the vdev and
> + * reassigns them to the PF.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_notify_vf() - send an alarm to a VF
> + * @hw: dlb2_hw handle for a particular device.
> + * @vf_id: VF ID
> + * @notification: notification
> + *
> + * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
> + *
> + * Return:
> + * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
> + */
> +int dlb2_notify_vf(struct dlb2_hw *hw,
> +                  unsigned int vf_id,
> +                  u32 notification);
> +
> +/**
> + * dlb2_vdev_in_use() - query whether a virtual device is in use
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function sends a mailbox request to the vdev to query whether the vdev
> + * is in use.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
> + * an internal error occurs.
> + */
> +int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * Clearing the PMCSR must be done at initialization to make the device fully
> + * operational.
> + */
> +void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: queue depth args
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns the depth of a load-balanced queue.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the depth.
> + *
> + * Errors:
> + * EINVAL - Invalid domain ID or queue ID.
> + */
> +int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
> +                               u32 domain_id,
> +                               struct dlb2_get_ldb_queue_depth_args *args,
> +                               struct dlb2_cmd_response *resp,
> +                               bool vdev_request,
> +                               unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: queue depth args
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns the depth of a directed queue.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the depth.
> + *
> + * Errors:
> + * EINVAL - Invalid domain ID or queue ID.
> + */
> +int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
> +                               u32 domain_id,
> +                               struct dlb2_get_dir_queue_depth_args *args,
> +                               struct dlb2_cmd_response *resp,
> +                               bool vdev_request,
> +                               unsigned int vdev_id);
> +
> +enum dlb2_virt_mode {
> +       DLB2_VIRT_NONE,
> +       DLB2_VIRT_SRIOV,
> +       DLB2_VIRT_SIOV,
> +
> +       /* NUM_DLB2_VIRT_MODES must be last */
> +       NUM_DLB2_VIRT_MODES,
> +};
> +
> +/**
> + * dlb2_hw_set_virt_mode() - set the device's virtualization mode
> + * @hw: dlb2_hw handle for a particular device.
> + * @mode: either none, SR-IOV, or Scalable IOV.
> + *
> + * This function sets the virtualization mode of the device. This controls
> + * whether the device uses a software or hardware mailbox.
> + *
> + * This should be called by the PF driver when either SR-IOV or Scalable IOV is
> + * selected as the virtualization mechanism, and by the VF/VDEV driver during
> + * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
> + *
> + * Errors:
> + * EINVAL - Invalid mode.
> + */
> +int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
> +
> +/**
> + * dlb2_hw_get_virt_mode() - get the device's virtualization mode
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function gets the virtualization mode of the device.
> + */
> +enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual port ID.
> + * @vdev_id: vdev ID.
> + *
> + * Return:
> + * Returns >= 0 upon success, -1 otherwise.
> + */
> +s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
> +                                u32 id,
> +                                unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual port ID.
> + * @vdev_id: vdev ID.
> + *
> + * Return:
> + * Returns >= 0 upon success, -1 otherwise.
> + */
> +s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
> +                                u32 id,
> +                                unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_register_sw_mbox() - register a software mailbox
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
> + * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
> + * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
> + * @inject_arg: user argument for pf2vdev_inject callback.
> + *
> + * When Scalable IOV is enabled, the VDCM must register a software mailbox for
> + * every virtual device during vdev creation.
> + *
> + * This function notifies the driver to use a software mailbox using the
> + * provided pointers, instead of the device's hardware mailbox. When the driver
> + * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
> + * go to the software mailbox instead of the hardware one. This is used in
> + * Scalable IOV virtualization.
> + */
> +void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
> +                             unsigned int vdev_id,
> +                             u32 *vdev2pf_mbox,
> +                             u32 *pf2vdev_mbox,
> +                             void (*pf2vdev_inject)(void *),
> +                             void *inject_arg);
> +
> +/**
> + * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + *
> + * This function notifies the driver to stop using a previously registered
> + * software mailbox.
> + */
> +void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + * @virt_cq_id: virtual CQ ID.
> + * @is_ldb: CQ is load-balanced.
> + * @addr_lo: least-significant 32 bits of address.
> + * @data: 32 data bits.
> + *
> + * This sets up the CQ's IMS entry with the provided address and data values.
> + * This function should only be called if the device is configured for Scalable
> + * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
> + * not needed.
> + */
> +void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
> +                               unsigned int vdev_id,
> +                               u32 virt_cq_id,
> +                               bool is_ldb,
> +                               u32 addr_lo,
> +                               u32 data);
> +
> +/**
> + * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + * @virt_cq_id: virtual CQ ID.
> + * @is_ldb: CQ is load-balanced.
> + *
> + * This clears the CQ's IMS entry, reverting it to its reset state.
> + */
> +void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
> +                               unsigned int vdev_id,
> +                               u32 virt_cq_id,
> +                               bool is_ldb);
> +
> +/**
> + * dlb2_hw_register_pasid() - register a vdev's PASID
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + * @pasid: the vdev's PASID.
> + *
> + * This function stores the user-supplied PASID, and uses it when configuring
> + * the vdev's CQs.
> + *
> + * Return:
> + * Returns >= 0 upon success, -1 otherwise.
> + */
> +int dlb2_hw_register_pasid(struct dlb2_hw *hw,
> +                          unsigned int vdev_id,
> +                          unsigned int pasid);
> +
> +/**
> + * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
> + *     progress.
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: number of unmaps in progress args
> + * @resp: response structure.
> + * @vf_request: indicates whether this request came from a VF.
> + * @vf_id: If vf_request is true, this contains the VF's ID.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the number of unmaps in progress.
> + *
> + * Errors:
> + * EINVAL - Invalid port ID.
> + */
> +int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
> +                               u32 domain_id,
> +                               struct dlb2_pending_port_unmaps_args *args,
> +                               struct dlb2_cmd_response *resp,
> +                               bool vf_request,
> +                               unsigned int vf_id);
> +
> +/**
> + * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
> + *     to a port class-of-service.
> + * @hw: dlb2_hw handle for a particular device.
> + * @cos_id: class-of-service ID.
> + *
> + * Return:
> + * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
> + */
> +int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
> +
> +/**
> + * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
> + *     port class-of-service.
> + * @hw: dlb2_hw handle for a particular device.
> + * @cos_id: class-of-service ID.
> + * @bandwidth: class-of-service bandwidth.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
> + *         cause the total bandwidth across all classes of service to exceed
> + *         100%.
> + */
> +int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
> +
> +enum dlb2_wd_tmo {
> +       /* 40s watchdog timeout */
> +       DLB2_WD_TMO_40S,
> +       /* 10s watchdog timeout */
> +       DLB2_WD_TMO_10S,
> +       /* 1s watchdog timeout */
> +       DLB2_WD_TMO_1S,
> +
> +       /* Must be last */
> +       NUM_DLB2_WD_TMOS,
> +};
> +
> +/**
> + * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
> + *     caller-specified timeout.
> + * @hw: dlb2_hw handle for a particular device.
> + * @tmo: watchdog timeout.
> + *
> + * This function should be called during device initialization and after reset.
> + * The watchdog timer interrupt must also be enabled per-CQ, using either
> + * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - Invalid timeout.
> + */
> +int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
> +
> +/**
> + * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
> + *     individual CQ.
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: port ID.
> + * @vdev_req: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - Invalid directed port ID.
> + */
> +int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
> +                                u32 id,
> +                                bool vdev_req,
> +                                unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
> + *     individual CQ.
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: port ID.
> + * @vdev_req: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - Invalid load-balanced port ID.
> + */
> +int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
> +                                u32 id,
> +                                bool vdev_req,
> +                                unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
> + *     ports.
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function must be called prior to configuring scheduling domains.
> + */
> +void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function must be called prior to configuring scheduling domains.
> + */
> +void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
> + * @hw: dlb2_hw handle for a particular device.
> + * @weight: 8-entry array of arbiter weights.
> + *
> + * weight[N] programs priority N's weight. In cases where the 8 priorities are
> + * reduced to 4 bins, the mapping is:
> + * - weight[1] programs bin 0
> + * - weight[3] programs bin 1
> + * - weight[5] programs bin 2
> + * - weight[7] programs bin 3
> + */
> +void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
> +
> +/**
> + * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
> + * @hw: dlb2_hw handle for a particular device.
> + * @weight: 8-entry array of arbiter weights.
> + *
> + * weight[N] programs priority N's weight. In cases where the 8 priorities are
> + * reduced to 4 bins, the mapping is:
> + * - weight[1] programs bin 0
> + * - weight[3] programs bin 1
> + * - weight[5] programs bin 2
> + * - weight[7] programs bin 3
> + */
> +void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
> +
> +/**
> + * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: physical load-balanced port ID.
> + *
> + * This function returns whether the load-balanced CQ interrupt is enabled.
> + */
> +int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
> +
> +/**
> + * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: physical load-balanced port ID.
> + * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
> + *
> + * This function can be used to disable (MODE_DIS) and re-enable the
> + * load-balanced CQ's interrupt. It should only be called after the interrupt
> + * has been configured with dlb2_configure_ldb_cq_interrupt().
> + */
> +void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
> +                                      int port_id,
> +                                      int mode);
> +
> +/**
> + * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: physical load-balanced port ID.
> + *
> + * This function returns whether the load-balanced CQ interrupt is enabled.
> + */
> +int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
> +
> +/**
> + * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: physical directed port ID.
> + * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
> + *
> + * This function can be used to disable (MODE_DIS) and re-enable the
> + * directed CQ's interrupt. It should only be called after the interrupt
> + * has been configured with dlb2_configure_dir_cq_interrupt().
> + */
> +void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
> +                                      int port_id,
> +                                      int mode);
> +
> +#endif /* __DLB2_RESOURCE_H */
> diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
> new file mode 100644
> index 0000000..bd8590d
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/dlb2_main.c
> @@ -0,0 +1,615 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <errno.h>
> +#include <assert.h>
> +#include <unistd.h>
> +#include <string.h>
> +
> +#include <rte_malloc.h>
> +#include <rte_errno.h>
> +
> +#include "base/dlb2_resource.h"
> +#include "base/dlb2_osdep.h"
> +#include "base/dlb2_regs.h"
> +#include "dlb2_main.h"
> +#include "../dlb2_user.h"
> +#include "../dlb2_priv.h"
> +#include "../dlb2_iface.h"
> +#include "../dlb2_inline_fns.h"
> +
> +#define PF_ID_ZERO 0   /* PF ONLY! */
> +#define NO_OWNER_VF 0  /* PF ONLY! */
> +#define NOT_VF_REQ false /* PF ONLY! */
> +
> +#define DLB2_PCI_CFG_SPACE_SIZE 256
> +#define DLB2_PCI_CAP_POINTER 0x34
> +#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
> +#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
> +#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
> +#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
> +#define DLB2_PCI_EXT_CAP_ID_ERR 1
> +#define DLB2_PCI_ERR_UNCOR_MASK 8
> +#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
> +
> +#define DLB2_PCI_EXP_DEVCTL 8
> +#define DLB2_PCI_LNKCTL 16
> +#define DLB2_PCI_SLTCTL 24
> +#define DLB2_PCI_RTCTL 28
> +#define DLB2_PCI_EXP_DEVCTL2 40
> +#define DLB2_PCI_LNKCTL2 48
> +#define DLB2_PCI_SLTCTL2 56
> +#define DLB2_PCI_CMD 4
> +#define DLB2_PCI_X_CMD 2
> +#define DLB2_PCI_EXP_DEVSTA 10
> +#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
> +#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
> +
> +#define DLB2_PCI_CAP_ID_EXP       0x10
> +#define DLB2_PCI_CAP_ID_MSIX      0x11
> +#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
> +#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
> +#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
> +
> +#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
> +#define DLB2_PCI_PRI_ALLOC_REQ           0xC
> +#define DLB2_PCI_PRI_CTRL                0x4
> +#define DLB2_PCI_MSIX_FLAGS              0x2
> +#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
> +#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
> +#define DLB2_PCI_ERR_ROOT_STATUS         0x30
> +#define DLB2_PCI_ERR_COR_STATUS          0x10
> +#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
> +#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
> +#define DLB2_PCI_ACS_CAP                 0x4
> +#define DLB2_PCI_ACS_CTRL                0x6
> +#define DLB2_PCI_ACS_SV                  0x1
> +#define DLB2_PCI_ACS_RR                  0x4
> +#define DLB2_PCI_ACS_CR                  0x8
> +#define DLB2_PCI_ACS_UF                  0x10
> +#define DLB2_PCI_ACS_EC                  0x20
> +
> +static int
> +dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
> +{
> +       uint32_t hdr;
> +       size_t sz;
> +       int pos;
> +
> +       pos = DLB2_PCI_CFG_SPACE_SIZE;
> +       sz = sizeof(hdr);
> +
> +       while (pos > 0xFF) {
> +               if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
> +                       return -1;
> +
> +               if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
> +                       return pos;
> +
> +               pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
> +       }
> +
> +       return -1;
> +}
> +
> +static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
> +{
> +       uint8_t pos;
> +       int ret;
> +       uint16_t hdr;
> +
> +       ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
> +       pos &= 0xFC;
> +
> +       if (ret != 1)
> +               return -1;
> +
> +       while (pos > 0x3F) {
> +               ret = rte_pci_read_config(pdev, &hdr, 2, pos);
> +               if (ret != 2)
> +                       return -1;
> +
> +               if (DLB2_PCI_CAP_ID(hdr) == id)
> +                       return pos;
> +
> +               if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
> +                       return -1;
> +
> +               pos = DLB2_PCI_CAP_NEXT(hdr);
> +       }
> +
> +       return -1;
> +}
> +
> +static int
> +dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
> +{
> +       int i;
> +
> +       if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIR64B))
> +               dlb2_dev->enqueue_four = dlb2_movdir64b;
> +       else
> +               dlb2_dev->enqueue_four = dlb2_movntdq;
> +
> +       /* Initialize software state */
> +       for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++)
> +               dlb2_list_init_head(&dlb2_dev->ldb_port_pages[i].list);
> +
> +       for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++)
> +               dlb2_list_init_head(&dlb2_dev->dir_port_pages[i].list);
> +
> +       rte_spinlock_init(&dlb2_dev->resource_mutex);
> +       rte_spinlock_init(&dlb2_dev->measurement_lock);
> +
> +       return 0;
> +}
> +
> +static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
> +{
> +       dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
> +}
> +
> +#define DLB2_READY_RETRY_LIMIT 1000
> +static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
> +{
> +       u32 retries = 0;
> +
> +       /* Allow at least 1s for the device to become active after power-on */
> +       for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
> +               union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
> +               union dlb2_cfg_mstr_cfg_pm_status pm_st;
> +               u32 addr;
> +
> +               addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
> +               pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
> +               addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
> +               idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
> +               if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
> +                       break;
> +
> +               rte_delay_ms(1);
> +       };
> +
> +       if (retries == DLB2_READY_RETRY_LIMIT) {
> +               printf("[%s()] wait for device ready timed out\n",
> +                      __func__);
> +               return -1;
> +       }
> +
> +       return 0;
> +}
> +
> +struct dlb2_dev *
> +dlb2_probe(struct rte_pci_device *pdev)
> +{
> +       struct dlb2_dev *dlb2_dev;
> +       int ret = 0;
> +
> +       DLB2_INFO(dlb2_dev, "probe\n");
> +
> +       dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
> +                             RTE_CACHE_LINE_SIZE);
> +
> +       if (dlb2_dev == NULL) {
> +               ret = -ENOMEM;
> +               goto dlb2_dev_malloc_fail;
> +       }
> +
> +       /* PCI Bus driver has already mapped bar space into process.
> +        * Save off our IO register and FUNC addresses.
> +        */
> +
> +       /* BAR 0 */
> +       if (pdev->mem_resource[0].addr == NULL) {
> +               DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
> +               ret = -EINVAL;
> +               goto pci_mmap_bad_addr;
> +       }
> +       dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
> +       dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
> +
> +       DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
> +                 (void *)dlb2_dev->hw.func_kva,
> +                 (void *)dlb2_dev->hw.func_phys_addr,
> +                 (void *)(pdev->mem_resource[0].len));
> +
> +       /* BAR 2 */
> +       if (pdev->mem_resource[2].addr == NULL) {
> +               DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
> +               ret = -EINVAL;
> +               goto pci_mmap_bad_addr;
> +       }
> +       dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
> +       dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
> +
> +       DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
> +                 (void *)dlb2_dev->hw.csr_kva,
> +                 (void *)dlb2_dev->hw.csr_phys_addr,
> +                 (void *)(pdev->mem_resource[2].len));
> +
> +       dlb2_dev->pdev = pdev;
> +
> +       /* PM enable must be done before any other MMIO accesses, and this
> +        * setting is persistent across device reset.
> +        */
> +       dlb2_pf_enable_pm(dlb2_dev);
> +
> +       ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
> +       if (ret)
> +               goto wait_for_device_ready_fail;
> +
> +       ret = dlb2_pf_reset(dlb2_dev);
> +       if (ret)
> +               goto dlb2_reset_fail;
> +
> +       ret = dlb2_pf_init_driver_state(dlb2_dev);
> +       if (ret)
> +               goto init_driver_state_fail;
> +
> +       ret = dlb2_resource_init(&dlb2_dev->hw);
> +       if (ret)
> +               goto resource_init_fail;
> +
> +       return dlb2_dev;
> +
> +resource_init_fail:
> +       dlb2_resource_free(&dlb2_dev->hw);
> +init_driver_state_fail:
> +dlb2_reset_fail:
> +pci_mmap_bad_addr:
> +wait_for_device_ready_fail:
> +       rte_free(dlb2_dev);
> +dlb2_dev_malloc_fail:
> +       rte_errno = ret;
> +       return NULL;
> +}
> +
> +int
> +dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
> +{
> +       int ret = 0;
> +       int i = 0;
> +       uint32_t dword[16];
> +       uint16_t cmd;
> +       off_t off;
> +
> +       uint16_t dev_ctl_word;
> +       uint16_t dev_ctl2_word;
> +       uint16_t lnk_word;
> +       uint16_t lnk_word2;
> +       uint16_t slt_word;
> +       uint16_t slt_word2;
> +       uint16_t rt_ctl_word;
> +       uint32_t pri_reqs_dword;
> +       uint16_t pri_ctrl_word;
> +
> +       int pcie_cap_offset;
> +       int pri_cap_offset;
> +       int msix_cap_offset;
> +       int err_cap_offset;
> +       int acs_cap_offset;
> +       int wait_count;
> +
> +       uint16_t devsta_busy_word;
> +       uint16_t devctl_word;
> +
> +       struct rte_pci_device *pdev = dlb2_dev->pdev;
> +
> +       /* Save PCI config state */
> +
> +       for (i = 0; i < 16; i++) {
> +               if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
> +                       return ret;
> +       }
> +
> +       pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
> +
> +       if (pcie_cap_offset < 0) {
> +               printf("[%s()] failed to find the pcie capability\n",
> +                      __func__);
> +               return pcie_cap_offset;
> +       }
> +
> +       off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> +       if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
> +               dev_ctl_word = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_LNKCTL;
> +       if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
> +               lnk_word = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_SLTCTL;
> +       if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
> +               slt_word = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_RTCTL;
> +       if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
> +               rt_ctl_word = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
> +       if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
> +               dev_ctl2_word = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
> +       if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
> +               lnk_word2 = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
> +       if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
> +               slt_word2 = 0;
> +
> +       off = DLB2_PCI_EXT_CAP_ID_PRI;
> +       pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> +
> +       if (pri_cap_offset >= 0) {
> +               off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
> +               if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
> +                       pri_reqs_dword = 0;
> +       }
> +
> +       /* clear the PCI command register before issuing the FLR */
> +
> +       off = DLB2_PCI_CMD;
> +       cmd = 0;
> +       if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +               printf("[%s()] failed to write the pci command\n",
> +                      __func__);
> +               return ret;
> +       }
> +
> +       /* issue the FLR */
> +       for (wait_count = 0; wait_count < 4; wait_count++) {
> +               int sleep_time;
> +
> +               off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
> +               ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to read the pci device status\n",
> +                              __func__);
> +                       return ret;
> +               }
> +
> +               if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
> +                       break;
> +
> +               sleep_time = (1 << (wait_count)) * 100;
> +               rte_delay_ms(sleep_time);
> +       }
> +
> +       if (wait_count == 4) {
> +               printf("[%s()] wait for pci pending transactions timed out\n",
> +                      __func__);
> +               return -1;
> +       }
> +
> +       off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> +       ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
> +       if (ret != 2) {
> +               printf("[%s()] failed to read the pcie device control\n",
> +                      __func__);
> +               return ret;
> +       }
> +
> +       devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
> +
> +       ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
> +       if (ret != 2) {
> +               printf("[%s()] failed to write the pcie device control\n",
> +                      __func__);
> +               return ret;
> +       }
> +
> +       rte_delay_ms(100);
> +
> +       /* Restore PCI config state */
> +
> +       if (pcie_cap_offset >= 0) {
> +               off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> +               ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie device control at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_LNKCTL;
> +               ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_SLTCTL;
> +               ret = rte_pci_write_config(pdev, &slt_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_RTCTL;
> +               ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
> +               ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
> +               ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
> +               ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +       }
> +
> +       if (pri_cap_offset >= 0) {
> +               pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
> +
> +               off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
> +               ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
> +               if (ret != 4) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
> +               ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +       }
> +
> +       off = DLB2_PCI_EXT_CAP_ID_ERR;
> +       err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> +
> +       if (err_cap_offset >= 0) {
> +               uint32_t tmp;
> +
> +               off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
> +               if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> +                       tmp = 0;
> +
> +               ret = rte_pci_write_config(pdev, &tmp, 4, off);
> +               if (ret != 4) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
> +               if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> +                       tmp = 0;
> +
> +               ret = rte_pci_write_config(pdev, &tmp, 4, off);
> +               if (ret != 4) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
> +               if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> +                       tmp = 0;
> +
> +               ret = rte_pci_write_config(pdev, &tmp, 4, off);
> +               if (ret != 4) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +       }
> +
> +       for (i = 16; i > 0; i--) {
> +               off = (i - 1) * 4;
> +               ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
> +               if (ret != 4) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +       }
> +
> +       off = DLB2_PCI_CMD;
> +       if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> +               cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
> +               if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +                       printf("[%s()] failed to write the pci command\n",
> +                              __func__);
> +                       return ret;
> +               }
> +       }
> +
> +       msix_cap_offset = dlb2_pci_find_capability(pdev,
> +                                                  DLB2_PCI_CAP_ID_MSIX);
> +       if (msix_cap_offset >= 0) {
> +               off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
> +               if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> +                       cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
> +                       cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
> +                       if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +                               printf("[%s()] failed to write msix flags\n",
> +                                      __func__);
> +                               return ret;
> +                       }
> +               }
> +
> +               off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
> +               if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> +                       cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
> +                       if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +                               printf("[%s()] failed to write msix flags\n",
> +                                      __func__);
> +                               return ret;
> +                       }
> +               }
> +       }
> +
> +       off = DLB2_PCI_EXT_CAP_ID_ACS;
> +       acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> +
> +       if (acs_cap_offset >= 0) {
> +               uint16_t acs_cap, acs_ctrl, acs_mask;
> +               off = acs_cap_offset + DLB2_PCI_ACS_CAP;
> +               if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
> +                       acs_cap = 0;
> +
> +               off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> +               if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
> +                       acs_ctrl = 0;
> +
> +               acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
> +               acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
> +               acs_ctrl |= (acs_cap & acs_mask);
> +
> +               ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> +               if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
> +                       acs_ctrl = 0;
> +
> +               acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
> +               acs_mask |= DLB2_PCI_ACS_EC;
> +               acs_ctrl &= ~acs_mask;
> +
> +               off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> +               ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +       }
> +
> +       return 0;
> +}
> diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
> new file mode 100644
> index 0000000..ec96f11
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/dlb2_main.h
> @@ -0,0 +1,106 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_MAIN_H
> +#define __DLB2_MAIN_H
> +
> +#include <rte_debug.h>
> +#include <rte_log.h>
> +#include <rte_spinlock.h>
> +#include <rte_pci.h>
> +#include <rte_bus_pci.h>
> +
> +#ifndef PAGE_SIZE
> +#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
> +#endif
> +
> +#include "base/dlb2_hw_types.h"
> +#include "../dlb2_user.h"
> +
> +#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
> +
> +struct dlb2_dev;
> +
> +struct dlb2_port_memory {
> +       struct dlb2_list_head list;
> +       void *cq_base;
> +       bool valid;
> +};
> +
> +struct dlb2_dev {
> +       struct rte_pci_device *pdev;
> +       struct dlb2_hw hw;
> +       /* struct list_head list; */
> +       struct device *dlb2_device;
> +       struct dlb2_port_memory ldb_port_pages[DLB2_MAX_NUM_LDB_PORTS];
> +       struct dlb2_port_memory dir_port_pages[DLB2_MAX_NUM_DIR_PORTS];
> +       /* The enqueue_four function enqueues four HCWs (one cache-line worth)
> +        * to the DLB2, using whichever mechanism is supported by the platform
> +        * on which this driver is running.
> +        */
> +       void (*enqueue_four)(void *qe4, void *pp_addr);
> +
> +       bool domain_reset_failed;
> +       /* The resource mutex serializes access to driver data structures and
> +        * hardware registers.
> +        */
> +       rte_spinlock_t resource_mutex;
> +       rte_spinlock_t measurement_lock;
> +       bool worker_launched;
> +       u8 revision;
> +};
> +
> +struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
> +
> +int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
> +int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
> +                               struct dlb2_create_sched_domain_args *args,
> +                               struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_create_ldb_queue_args *args,
> +                            struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_create_dir_queue_args *args,
> +                            struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_create_ldb_port_args *args,
> +                           uintptr_t cq_dma_base,
> +                           struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_create_dir_port_args *args,
> +                           uintptr_t cq_dma_base,
> +                           struct dlb2_cmd_response *resp);
> +int dlb2_pf_start_domain(struct dlb2_hw *hw,
> +                        u32 domain_id,
> +                        struct dlb2_start_domain_args *args,
> +                        struct dlb2_cmd_response *resp);
> +int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_enable_ldb_port_args *args,
> +                           struct dlb2_cmd_response *resp);
> +int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_disable_ldb_port_args *args,
> +                            struct dlb2_cmd_response *resp);
> +int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_enable_dir_port_args *args,
> +                           struct dlb2_cmd_response *resp);
> +int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_disable_dir_port_args *args,
> +                            struct dlb2_cmd_response *resp);
> +int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
> +int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
> +                                    u32 domain_id,
> +                                    u32 port_id);
> +int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
> +                                    u32 domain_id,
> +                                    u32 port_id);
> +
> +#endif /* __DLB2_MAIN_H */
> diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
> new file mode 100644
> index 0000000..7d64309
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/dlb2_pf.c
> @@ -0,0 +1,244 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <sys/mman.h>
> +#include <sys/fcntl.h>
> +#include <sys/time.h>
> +#include <errno.h>
> +#include <assert.h>
> +#include <unistd.h>
> +#include <string.h>
> +#include <rte_debug.h>
> +#include <rte_log.h>
> +#include <rte_dev.h>
> +#include <rte_devargs.h>
> +#include <rte_mbuf.h>
> +#include <rte_ring.h>
> +#include <rte_errno.h>
> +#include <rte_kvargs.h>
> +#include <rte_malloc.h>
> +#include <rte_cycles.h>
> +#include <rte_io.h>
> +#include <rte_pci.h>
> +#include <rte_bus_pci.h>
> +#include <rte_eventdev.h>
> +#include <rte_eventdev_pmd.h>
> +#include <rte_eventdev_pmd_pci.h>
> +#include <rte_memory.h>
> +#include <rte_string_fns.h>
> +
> +#include "../dlb2_priv.h"
> +#include "../dlb2_iface.h"
> +#include "../dlb2_inline_fns.h"
> +#include "dlb2_main.h"
> +#include "base/dlb2_hw_types.h"
> +#include "base/dlb2_osdep.h"
> +#include "base/dlb2_resource.h"
> +
> +static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
> +
> +static void
> +dlb2_pf_low_level_io_init(void)
> +{
> +       int i;
> +       /* Addresses will be initialized at port create */
> +       for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
> +               /* First directed ports */
> +               dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
> +               dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
> +               dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
> +
> +               /* Now load balanced ports */
> +               dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
> +               dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
> +               dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
> +       }
> +}
> +
> +static int
> +dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
> +{
> +       RTE_SET_USED(handle);
> +       RTE_SET_USED(name);
> +
> +       return 0;
> +}
> +
> +static int
> +dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
> +                          uint8_t *revision)
> +{
> +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> +
> +       *revision = dlb2_dev->revision;
> +
> +       return 0;
> +}
> +
> +static void
> +dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
> +{
> +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> +
> +       dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
> +       dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
> +}
> +
> +static int
> +dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
> +                         struct dlb2_get_num_resources_args *rsrcs)
> +{
> +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> +
> +       return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
> +}
> +
> +static int
> +dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
> +                        enum dlb2_cq_poll_modes *mode)
> +{
> +       RTE_SET_USED(handle);
> +
> +       *mode = DLB2_CQ_POLL_MODE_SPARSE;
> +
> +       return 0;
> +}
> +
> +static void
> +dlb2_pf_iface_fn_ptrs_init(void)
> +{
> +       dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
> +       dlb2_iface_open = dlb2_pf_open;
> +       dlb2_iface_get_device_version = dlb2_pf_get_device_version;
> +       dlb2_iface_hardware_init = dlb2_pf_hardware_init;
> +       dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
> +       dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
> +}
> +
> +/* PCI DEV HOOKS */
> +static int
> +dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
> +{
> +       int ret = 0;
> +       struct rte_pci_device *pci_dev;
> +       struct dlb2_devargs dlb2_args = {
> +               .socket_id = rte_socket_id(),
> +               .max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
> +               .num_dir_credits_override = -1,
> +               .qid_depth_thresholds = { {0} },
> +               .cos_id = DLB2_COS_DEFAULT
> +       };
> +       struct dlb2_eventdev *dlb2;
> +
> +       DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
> +                    eventdev->data->dev_id, eventdev->data->socket_id);
> +
> +       dlb2_pf_iface_fn_ptrs_init();
> +
> +       pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
> +
> +       if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> +               dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
> +
> +               /* Probe the DLB2 PF layer */
> +               dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
> +
> +               if (dlb2->qm_instance.pf_dev == NULL) {
> +                       DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
> +                                    rte_errno);
> +                       ret = -rte_errno;
> +                       goto dlb2_probe_failed;
> +               }
> +
> +               /* Were we invoked with runtime parameters? */
> +               if (pci_dev->device.devargs) {
> +                       ret = dlb2_parse_params(pci_dev->device.devargs->args,
> +                                               pci_dev->device.devargs->name,
> +                                               &dlb2_args);
> +                       if (ret) {
> +                               DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
> +                                            ret, rte_errno);
> +                               goto dlb2_probe_failed;
> +                       }
> +               }
> +
> +               ret = dlb2_primary_eventdev_probe(eventdev,
> +                                                 event_dlb2_pf_name,
> +                                                 &dlb2_args);
> +       } else {
> +               ret = dlb2_secondary_eventdev_probe(eventdev,
> +                                                   event_dlb2_pf_name);
> +       }
> +       if (ret)
> +               goto dlb2_probe_failed;
> +
> +       DLB2_LOG_INFO("DLB2 PF Probe success\n");
> +
> +       return 0;
> +
> +dlb2_probe_failed:
> +
> +       DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
> +
> +       return ret;
> +}
> +
> +#define EVENTDEV_INTEL_VENDOR_ID 0x8086
> +
> +static const struct rte_pci_id pci_id_dlb2_map[] = {
> +       {
> +               RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
> +                              PCI_DEVICE_ID_INTEL_DLB2_PF)
> +       },
> +       {
> +               .vendor_id = 0,
> +       },
> +};
> +
> +static int
> +event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
> +                    struct rte_pci_device *pci_dev)
> +{
> +       int ret;
> +
> +       ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
> +                                            sizeof(struct dlb2_eventdev),
> +                                            dlb2_eventdev_pci_init,
> +                                            event_dlb2_pf_name);
> +       if (ret) {
> +               DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
> +                               "ret=%d\n", ret);
> +       }
> +
> +       return ret;
> +}
> +
> +static int
> +event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
> +{
> +       int ret;
> +
> +       ret = rte_event_pmd_pci_remove(pci_dev, NULL);
> +
> +       if (ret) {
> +               DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
> +                               "ret=%d\n", ret);
> +       }
> +
> +       return ret;
> +
> +}
> +
> +static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
> +       .id_table = pci_id_dlb2_map,
> +       .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
> +       .probe = event_dlb2_pci_probe,
> +       .remove = event_dlb2_pci_remove,
> +};
> +
> +RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
> +RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
> --
> 2.6.4
>

Re: [dpdk-dev] [PATCH v2 06/22] event/dlb2: add probe

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Sunday, October 18, 2020 3:40 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; dpdk-dev <dev@dpdk.org>;
> Carrillo, Erik G <Erik.G.Carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> Jerin Jacob <jerinj@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH v2 06/22] event/dlb2: add probe
> 
> On Sat, Oct 17, 2020 at 11:52 PM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > The DLB2 hardware is a PCI device. This commit adds
> > support for probe and other initialization. The
> > dlb2_iface.[ch] files implement a flexible interface
> > that supports both the PF PMD and the bifurcated PMD.
> > The bifurcated PMD will be released in a future
> > patch set. Note that the flexible interface is only
> > used for configuration, and is not used in the data
> > path. The shared code is added in pf/base.
> > This PMD supports command line parameters, and those
> > are parsed at probe-time.
> >
> > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> 
> 
> 
> There is a build issue clang10.
> 
> [for-main]dell[dpdk-next-eventdev] $ clang -v
> clang version 10.0.1
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> InstalledDir: /usr/bin
> Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-
> gnu/10.2.0
> Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/8.4.0
> Found candidate GCC installation:
> /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.2.0
> Found candidate GCC installation:
> /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/8.4.0
> Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0
> Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/8.4.0
> Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0
> Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/8.4.0
> Selected GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.2.0
> Candidate multilib: .;@m64
> Candidate multilib: 32;@m32
> Selected multilib: .;@m64
> 
> meson  -Dexamples=l3fwd --buildtype=debugoptimized --werror
> --default-library=static /export/dpdk-next-eventdev/devtools/..
> ./build-clang-static
> 
> 
> 
> ccache clang -Idrivers/libtmp_rte_pmd_dlb2_event.a.p -Idrivers
> -I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
> -Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
> -I../config -Ilib/librte_eal/include -I../lib/librte_e
> al/include -Ilib/librte_eal/linux/include
> -I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
> -I../lib/librte_eal/x86/include -Ilib/librte_eal/common
> -I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
> -Ilib/librte_kv
> args -I../lib/librte_kvargs -Ilib/librte_metrics
> -I../lib/librte_metrics -Ilib/librte_telemetry
> -I../lib/librte_telemetry -Ilib/librte_ring -I../lib/librte_ring
> -Ilib/librte_ethdev -I../lib/librte_ethdev -Ilib/librte_net
> -I../lib/librte_net
>  -Ilib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
> -I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
> -Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_timer
> -I../lib/librte_timer -Ilib/librte_cryptodev -I../lib/li
> brte_cryptodev -Ilib/librte_pci -I../lib/librte_pci -Idrivers/bus/pci
> -I../drivers/bus/pci -I../drivers/bus/pci/linux -Xclang
> -fcolor-diagnostics -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
> -Werror -O2 -g -include rte_config.h -Wextra
> -Wcast-qual -Wdeprecated -Wformat-nonliteral -Wformat-security
> -Wmissing-declarations -Wmissing-prototypes -Wnested-externs
> -Wold-style-definition -Wpointer-arith -Wsign-compare
> -Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-pa
> cked-member -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC
> -march=native -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -
> MQ
> drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_pf_dlb2_pf.c.o -MF
> drivers/libtmp_rte_pmd_dlb2_event.a.p/ev
> ent_dlb2_pf_dlb2_pf.c.o.d -o
> drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_pf_dlb2_pf.c.o -c
> ../drivers/event/dlb2/pf/dlb2_pf.c
> In file included from ../drivers/event/dlb2/pf/dlb2_pf.c:36:
> ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:45:2: error: use of
> unknown builtin '__builtin_ia32_movntdq'
> [-Wimplicit-function-declaration]
>         __builtin_ia32_movntdq((__v2di *)pp_addr + 0, (__v2di)src_data0);
>         ^
> ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:45:2: note: did you mean
> '__builtin_ia32_movntq'?
> /usr/lib/clang/10.0.1/include/xmmintrin.h:2122:3: note:
> '__builtin_ia32_movntq' declared here
>   __builtin_ia32_movntq(__p, __a);
>   ^
> In file included from ../drivers/event/dlb2/pf/dlb2_pf.c:36:
> ../drivers/event/dlb2/pf/../dlb2_inline_fns.h:61:2: error: use of
> unknown builtin '__builtin_ia32_movntdq'
> [-Wimplicit-function-declaration]
>         __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
>         ^
> 2 errors generated.
> [2124/2479] Compiling C object
> drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_dlb2.c.o
> FAILED: drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_dlb2.c.o
> ccache clang -Idrivers/libtmp_rte_pmd_dlb2_event.a.p -Idrivers
> -I../drivers -Idrivers/event/dlb2 -I../drivers/event/dlb2
> -Ilib/librte_eventdev -I../lib/librte_eventdev -I. -I.. -Iconfig
> -I../config -Ilib/librte_eal/include -I../lib/librte_e
> al/include -Ilib/librte_eal/linux/include
> -I../lib/librte_eal/linux/include -Ilib/librte_eal/x86/include
> -I../lib/librte_eal/x86/include -Ilib/librte_eal/common
> -I../lib/librte_eal/common -Ilib/librte_eal -I../lib/librte_eal
> -Ilib/librte_kv
> args -I../lib/librte_kvargs -Ilib/librte_metrics
> -I../lib/librte_metrics -Ilib/librte_telemetry
> -I../lib/librte_telemetry -Ilib/librte_ring -I../lib/librte_ring
> -Ilib/librte_ethdev -I../lib/librte_ethdev -Ilib/librte_net
> -I../lib/librte_net
>  -Ilib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
> -I../lib/librte_mempool -Ilib/librte_meter -I../lib/librte_meter
> -Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_timer
> -I../lib/librte_timer -Ilib/librte_cryptodev -I../lib/li
> brte_cryptodev -Ilib/librte_pci -I../lib/librte_pci -Idrivers/bus/pci
> -I../drivers/bus/pci -I../drivers/bus/pci/linux -Xclang
> -fcolor-diagnostics -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch
> -Werror -O2 -g -include rte_config.h -Wextra
> -Wcast-qual -Wdeprecated -Wformat-nonliteral -Wformat-security
> -Wmissing-declarations -Wmissing-prototypes -Wnested-externs
> -Wold-style-definition -Wpointer-arith -Wsign-compare
> -Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-pa
> cked-member -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC
> -march=native -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -MD -
> MQ
> drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_dlb2.c.o -MF
> drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dl
> b2_dlb2.c.o.d -o
> drivers/libtmp_rte_pmd_dlb2_event.a.p/event_dlb2_dlb2.c.o -c
> ../drivers/event/dlb2/dlb2.c
> In file included from ../drivers/event/dlb2/dlb2.c:35:
> ../drivers/event/dlb2/dlb2_inline_fns.h:45:2: error: use of unknown
> builtin '__builtin_ia32_movntdq' [-Wimplicit-function-declaration]
>         __builtin_ia32_movntdq((__v2di *)pp_addr + 0, (__v2di)src_data0);
>         ^
> ../drivers/event/dlb2/dlb2_inline_fns.h:45:2: note: did you mean
> '__builtin_ia32_movntq'?
> /usr/lib/clang/10.0.1/include/xmmintrin.h:2122:3: note:
> '__builtin_ia32_movntq' declared here
>   __builtin_ia32_movntq(__p, __a);
>   ^
> In file included from ../drivers/event/dlb2/dlb2.c:35:
> ../drivers/event/dlb2/dlb2_inline_fns.h:61:2: error: use of unknown
> builtin '__builtin_ia32_movntdq' [-Wimplicit-function-declaration]
>         __builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
>         ^
> 2 errors generated.
> [2125/2479] Generating rte_common_sfc_efx.sym_chk with a meson_exe.py
> custom command
> ninja: build stopped: subcommand failed.
> 
> 
> 
> 
> > ---
> >  drivers/event/dlb2/dlb2.c                      |  532 +++++
> >  drivers/event/dlb2/dlb2_iface.c                |   28 +
> >  drivers/event/dlb2/dlb2_iface.h                |   29 +
> >  drivers/event/dlb2/meson.build                 |    6 +-
> >  drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
> >  drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
> >  drivers/event/dlb2/pf/base/dlb2_osdep.h        |  247 +++
> >  drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
> >  drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
> >  drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
> >  drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527
> ++++++++++++++++++++++++
> >  drivers/event/dlb2/pf/base/dlb2_resource.c     |  274 +++
> >  drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
> >  drivers/event/dlb2/pf/dlb2_main.c              |  615 ++++++
> >  drivers/event/dlb2/pf/dlb2_main.h              |  106 +
> >  drivers/event/dlb2/pf/dlb2_pf.c                |  244 +++
> >  16 files changed, 8085 insertions(+), 1 deletion(-)
> >  create mode 100644 drivers/event/dlb2/dlb2.c
> >  create mode 100644 drivers/event/dlb2/dlb2_iface.c
> >  create mode 100644 drivers/event/dlb2/dlb2_iface.h
> >  create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
> >  create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
> >  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
> >  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
> >  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
> >  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
> >  create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
> >  create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
> >  create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
> >  create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
> >  create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
> >  create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
> >
> > diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> > new file mode 100644
> > index 0000000..26985b9
> > --- /dev/null
> > +++ b/drivers/event/dlb2/dlb2.c
> > @@ -0,0 +1,532 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#include <assert.h>
> > +#include <errno.h>
> > +#include <nmmintrin.h>
> > +#include <pthread.h>
> > +#include <stdint.h>
> > +#include <stdbool.h>
> > +#include <stdio.h>
> > +#include <string.h>
> > +#include <sys/mman.h>
> > +#include <sys/fcntl.h>
> > +
> > +#include <rte_common.h>
> > +#include <rte_config.h>
> > +#include <rte_cycles.h>
> > +#include <rte_debug.h>
> > +#include <rte_dev.h>
> > +#include <rte_errno.h>
> > +#include <rte_eventdev.h>
> > +#include <rte_eventdev_pmd.h>
> > +#include <rte_io.h>
> > +#include <rte_kvargs.h>
> > +#include <rte_log.h>
> > +#include <rte_malloc.h>
> > +#include <rte_mbuf.h>
> > +#include <rte_prefetch.h>
> > +#include <rte_ring.h>
> > +#include <rte_string_fns.h>
> > +
> > +#include "dlb2_priv.h"
> > +#include "dlb2_iface.h"
> > +#include "dlb2_inline_fns.h"
> > +
> > +#if !defined RTE_ARCH_X86_64
> > +#error "This implementation only supports RTE_ARCH_X86_64 architecture."
> > +#endif
> > +
> > +/*
> > + * Resources exposed to eventdev. Some values overridden at runtime using
> > + * values returned by the DLB kernel driver.
> > + */
> > +#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
> > +#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member
> max_event_queues"
> > +#endif
> > +static struct rte_event_dev_info evdev_dlb2_default_info = {
> > +       .driver_name = "", /* probe will set */
> > +       .min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
> > +       .max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
> > +#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
> > +       .max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
> > +#else
> > +       .max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
> > +#endif
> > +       .max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
> > +       .max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
> > +       .max_event_priority_levels = DLB2_QID_PRIORITIES,
> > +       .max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
> > +       .max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
> > +       .max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
> > +       .max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
> > +       .max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
> > +       .max_single_link_event_port_queue_pairs =
> DLB2_MAX_NUM_DIR_PORTS,
> > +       .event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
> > +                         RTE_EVENT_DEV_CAP_EVENT_QOS |
> > +                         RTE_EVENT_DEV_CAP_BURST_MODE |
> > +                         RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
> > +                         RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
> > +                         RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
> > +};
> > +
> > +struct process_local_port_data
> > +dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
> > +
> > +/* override defaults with value(s) provided on command line */
> > +static void
> > +dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
> > +                                int *qid_depth_thresholds)
> > +{
> > +       int q;
> > +
> > +       for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
> > +               if (qid_depth_thresholds[q] != 0)
> > +                       dlb2->ev_queues[q].depth_threshold =
> > +                               qid_depth_thresholds[q];
> > +       }
> > +}
> > +
> > +static int
> > +dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
> > +{
> > +       struct dlb2_hw_dev *handle = &dlb2->qm_instance;
> > +       struct dlb2_hw_resource_info *dlb2_info = &handle->info;
> > +       int ret;
> > +
> > +       /* Query driver resources provisioned for this device */
> > +
> > +       ret = dlb2_iface_get_num_resources(handle,
> > +                                          &dlb2->hw_rsrc_query_results);
> > +       if (ret) {
> > +               DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
> > +               return ret;
> > +       }
> > +
> > +       /* Complete filling in device resource info returned to evdev app,
> > +        * overriding any default values.
> > +        * The capabilities (CAPs) were set at compile time.
> > +        */
> > +
> > +       evdev_dlb2_default_info.max_event_queues =
> > +               dlb2->hw_rsrc_query_results.num_ldb_queues;
> > +
> > +       evdev_dlb2_default_info.max_event_ports =
> > +               dlb2->hw_rsrc_query_results.num_ldb_ports;
> > +
> > +       evdev_dlb2_default_info.max_num_events =
> > +               dlb2->hw_rsrc_query_results.num_ldb_credits;
> > +
> > +       /* Save off values used when creating the scheduling domain. */
> > +
> > +       handle->info.num_sched_domains =
> > +               dlb2->hw_rsrc_query_results.num_sched_domains;
> > +
> > +       handle->info.hw_rsrc_max.nb_events_limit =
> > +               dlb2->hw_rsrc_query_results.num_ldb_credits;
> > +
> > +       handle->info.hw_rsrc_max.num_queues =
> > +               dlb2->hw_rsrc_query_results.num_ldb_queues +
> > +               dlb2->hw_rsrc_query_results.num_dir_ports;
> > +
> > +       handle->info.hw_rsrc_max.num_ldb_queues =
> > +               dlb2->hw_rsrc_query_results.num_ldb_queues;
> > +
> > +       handle->info.hw_rsrc_max.num_ldb_ports =
> > +               dlb2->hw_rsrc_query_results.num_ldb_ports;
> > +
> > +       handle->info.hw_rsrc_max.num_dir_ports =
> > +               dlb2->hw_rsrc_query_results.num_dir_ports;
> > +
> > +       handle->info.hw_rsrc_max.reorder_window_size =
> > +               dlb2->hw_rsrc_query_results.num_hist_list_entries;
> > +
> > +       rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
> > +
> > +       return 0;
> > +}
> > +
> > +#define RTE_BASE_10 10
> > +
> > +static int
> > +dlb2_string_to_int(int *result, const char *str)
> > +{
> > +       long ret;
> > +       char *endptr;
> > +
> > +       if (str == NULL || result == NULL)
> > +               return -EINVAL;
> > +
> > +       errno = 0;
> > +       ret = strtol(str, &endptr, RTE_BASE_10);
> > +       if (errno)
> > +               return -errno;
> > +
> > +       /* long int and int may be different width for some architectures */
> > +       if (ret < INT_MIN || ret > INT_MAX || endptr == str)
> > +               return -EINVAL;
> > +
> > +       *result = ret;
> > +       return 0;
> > +}
> > +
> > +static int
> > +set_numa_node(const char *key __rte_unused, const char *value, void
> *opaque)
> > +{
> > +       int *socket_id = opaque;
> > +       int ret;
> > +
> > +       ret = dlb2_string_to_int(socket_id, value);
> > +       if (ret < 0)
> > +               return ret;
> > +
> > +       if (*socket_id > RTE_MAX_NUMA_NODES)
> > +               return -EINVAL;
> > +       return 0;
> > +}
> > +
> > +static int
> > +set_max_num_events(const char *key __rte_unused,
> > +                  const char *value,
> > +                  void *opaque)
> > +{
> > +       int *max_num_events = opaque;
> > +       int ret;
> > +
> > +       if (value == NULL || opaque == NULL) {
> > +               DLB2_LOG_ERR("NULL pointer\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       ret = dlb2_string_to_int(max_num_events, value);
> > +       if (ret < 0)
> > +               return ret;
> > +
> > +       if (*max_num_events < 0 || *max_num_events >
> > +                       DLB2_MAX_NUM_LDB_CREDITS) {
> > +               DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and
> %d\n",
> > +                            DLB2_MAX_NUM_LDB_CREDITS);
> > +               return -EINVAL;
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +static int
> > +set_num_dir_credits(const char *key __rte_unused,
> > +                   const char *value,
> > +                   void *opaque)
> > +{
> > +       int *num_dir_credits = opaque;
> > +       int ret;
> > +
> > +       if (value == NULL || opaque == NULL) {
> > +               DLB2_LOG_ERR("NULL pointer\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       ret = dlb2_string_to_int(num_dir_credits, value);
> > +       if (ret < 0)
> > +               return ret;
> > +
> > +       if (*num_dir_credits < 0 ||
> > +           *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
> > +               DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and
> %d\n",
> > +                            DLB2_MAX_NUM_DIR_CREDITS);
> > +               return -EINVAL;
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +static int
> > +set_dev_id(const char *key __rte_unused,
> > +          const char *value,
> > +          void *opaque)
> > +{
> > +       int *dev_id = opaque;
> > +       int ret;
> > +
> > +       if (value == NULL || opaque == NULL) {
> > +               DLB2_LOG_ERR("NULL pointer\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       ret = dlb2_string_to_int(dev_id, value);
> > +       if (ret < 0)
> > +               return ret;
> > +
> > +       return 0;
> > +}
> > +
> > +static int
> > +set_cos(const char *key __rte_unused,
> > +       const char *value,
> > +       void *opaque)
> > +{
> > +       enum dlb2_cos *cos_id = opaque;
> > +       int x = 0;
> > +       int ret;
> > +
> > +       if (value == NULL || opaque == NULL) {
> > +               DLB2_LOG_ERR("NULL pointer\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       ret = dlb2_string_to_int(&x, value);
> > +       if (ret < 0)
> > +               return ret;
> > +
> > +       if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
> > +               DLB2_LOG_ERR(
> > +                       "COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
> > +                       x);
> > +               return -EINVAL;
> > +       }
> > +
> > +       *cos_id = x;
> > +
> > +       return 0;
> > +}
> > +
> > +
> > +static int
> > +set_qid_depth_thresh(const char *key __rte_unused,
> > +                    const char *value,
> > +                    void *opaque)
> > +{
> > +       struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
> > +       int first, last, thresh, i;
> > +
> > +       if (value == NULL || opaque == NULL) {
> > +               DLB2_LOG_ERR("NULL pointer\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       /* command line override may take one of the following 3 forms:
> > +        * qid_depth_thresh=all:<threshold_value> ... all queues
> > +        * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
> > +        * qid_depth_thresh=qid:<threshold_value> ... just one queue
> > +        */
> > +       if (sscanf(value, "all:%d", &thresh) == 1) {
> > +               first = 0;
> > +               last = DLB2_MAX_NUM_QUEUES - 1;
> > +       } else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
> > +               /* we have everything we need */
> > +       } else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
> > +               last = first;
> > +       } else {
> > +               DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val,
> qid-qid:val, or qid:val\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
> > +               DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
> > +               DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
> > +                            DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
> > +               return -EINVAL;
> > +       }
> > +
> > +       for (i = first; i <= last; i++)
> > +               qid_thresh->val[i] = thresh; /* indexed by qid */
> > +
> > +       return 0;
> > +}
> > +
> > +static void
> > +dlb2_entry_points_init(struct rte_eventdev *dev)
> > +{
> > +       RTE_SET_USED(dev);
> > +
> > +       /* Eventdev PMD entry points */
> > +}
> > +
> > +int
> > +dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
> > +                           const char *name,
> > +                           struct dlb2_devargs *dlb2_args)
> > +{
> > +       struct dlb2_eventdev *dlb2;
> > +       int err;
> > +
> > +       dlb2 = dev->data->dev_private;
> > +
> > +       dlb2->event_dev = dev; /* backlink */
> > +
> > +       evdev_dlb2_default_info.driver_name = name;
> > +
> > +       dlb2->max_num_events_override = dlb2_args->max_num_events;
> > +       dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
> > +       dlb2->qm_instance.cos_id = dlb2_args->cos_id;
> > +
> > +       err = dlb2_iface_open(&dlb2->qm_instance, name);
> > +       if (err < 0) {
> > +               DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
> > +                            err);
> > +               return err;
> > +       }
> > +
> > +       err = dlb2_iface_get_device_version(&dlb2->qm_instance,
> > +                                           &dlb2->revision);
> > +       if (err < 0) {
> > +               DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
> > +                            err);
> > +               return err;
> > +       }
> > +
> > +       err = dlb2_hw_query_resources(dlb2);
> > +       if (err) {
> > +               DLB2_LOG_ERR("get resources err=%d for %s\n",
> > +                            err, name);
> > +               return err;
> > +       }
> > +
> > +       dlb2_iface_hardware_init(&dlb2->qm_instance);
> > +
> > +       err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2-
> >poll_mode);
> > +       if (err < 0) {
> > +               DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
> > +                            err);
> > +               return err;
> > +       }
> > +
> > +       rte_spinlock_init(&dlb2->qm_instance.resource_lock);
> > +
> > +       dlb2_iface_low_level_io_init();
> > +
> > +       dlb2_entry_points_init(dev);
> > +
> > +       dlb2_init_queue_depth_thresholds(dlb2,
> > +                                        dlb2_args->qid_depth_thresholds.val);
> > +
> > +       return 0;
> > +}
> > +
> > +int
> > +dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
> > +                             const char *name)
> > +{
> > +       struct dlb2_eventdev *dlb2;
> > +       int err;
> > +
> > +       dlb2 = dev->data->dev_private;
> > +
> > +       evdev_dlb2_default_info.driver_name = name;
> > +
> > +       err = dlb2_iface_open(&dlb2->qm_instance, name);
> > +       if (err < 0) {
> > +               DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
> > +                            err);
> > +               return err;
> > +       }
> > +
> > +       err = dlb2_hw_query_resources(dlb2);
> > +       if (err) {
> > +               DLB2_LOG_ERR("get resources err=%d for %s\n",
> > +                            err, name);
> > +               return err;
> > +       }
> > +
> > +       dlb2_iface_low_level_io_init();
> > +
> > +       dlb2_entry_points_init(dev);
> > +
> > +       return 0;
> > +}
> > +
> > +int
> > +dlb2_parse_params(const char *params,
> > +                 const char *name,
> > +                 struct dlb2_devargs *dlb2_args)
> > +{
> > +       int ret = 0;
> > +       static const char * const args[] = { NUMA_NODE_ARG,
> > +                                            DLB2_MAX_NUM_EVENTS,
> > +                                            DLB2_NUM_DIR_CREDITS,
> > +                                            DEV_ID_ARG,
> > +                                            DLB2_QID_DEPTH_THRESH_ARG,
> > +                                            DLB2_COS_ARG,
> > +                                            NULL };
> > +
> > +       if (params != NULL && params[0] != '\0') {
> > +               struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
> > +
> > +               if (kvlist == NULL) {
> > +                       RTE_LOG(INFO, PMD,
> > +                               "Ignoring unsupported parameters when creating device
> '%s'\n",
> > +                               name);
> > +               } else {
> > +                       int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
> > +                                                    set_numa_node,
> > +                                                    &dlb2_args->socket_id);
> > +                       if (ret != 0) {
> > +                               DLB2_LOG_ERR("%s: Error parsing numa node parameter",
> > +                                            name);
> > +                               rte_kvargs_free(kvlist);
> > +                               return ret;
> > +                       }
> > +
> > +                       ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
> > +                                                set_max_num_events,
> > +                                                &dlb2_args->max_num_events);
> > +                       if (ret != 0) {
> > +                               DLB2_LOG_ERR("%s: Error parsing max_num_events
> parameter",
> > +                                            name);
> > +                               rte_kvargs_free(kvlist);
> > +                               return ret;
> > +                       }
> > +
> > +                       ret = rte_kvargs_process(kvlist,
> > +                                       DLB2_NUM_DIR_CREDITS,
> > +                                       set_num_dir_credits,
> > +                                       &dlb2_args->num_dir_credits_override);
> > +                       if (ret != 0) {
> > +                               DLB2_LOG_ERR("%s: Error parsing num_dir_credits
> parameter",
> > +                                            name);
> > +                               rte_kvargs_free(kvlist);
> > +                               return ret;
> > +                       }
> > +
> > +                       ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
> > +                                                set_dev_id,
> > +                                                &dlb2_args->dev_id);
> > +                       if (ret != 0) {
> > +                               DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
> > +                                            name);
> > +                               rte_kvargs_free(kvlist);
> > +                               return ret;
> > +                       }
> > +
> > +                       ret = rte_kvargs_process(
> > +                                       kvlist,
> > +                                       DLB2_QID_DEPTH_THRESH_ARG,
> > +                                       set_qid_depth_thresh,
> > +                                       &dlb2_args->qid_depth_thresholds);
> > +                       if (ret != 0) {
> > +                               DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh
> parameter",
> > +                                            name);
> > +                               rte_kvargs_free(kvlist);
> > +                               return ret;
> > +                       }
> > +
> > +                       ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
> > +                                                set_cos,
> > +                                                &dlb2_args->cos_id);
> > +                       if (ret != 0) {
> > +                               DLB2_LOG_ERR("%s: Error parsing cos parameter",
> > +                                            name);
> > +                               rte_kvargs_free(kvlist);
> > +                               return ret;
> > +                       }
> > +
> > +                       rte_kvargs_free(kvlist);
> > +               }
> > +       }
> > +       return ret;
> > +}
> > +RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
> > diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
> > new file mode 100644
> > index 0000000..0d93faf
> > --- /dev/null
> > +++ b/drivers/event/dlb2/dlb2_iface.c
> > @@ -0,0 +1,28 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#include <stdint.h>
> > +
> > +#include "dlb2_priv.h"
> > +
> > +/* DLB2 PMD Internal interface function pointers.
> > + * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
> > + * serviced by DLB kernel module.
> > + * If PCI (PF PMD),  these will be implemented locally in user mode.
> > + */
> > +
> > +void (*dlb2_iface_low_level_io_init)(void);
> > +
> > +int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
> > +
> > +int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
> > +                                    uint8_t *revision);
> > +
> > +void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
> > +
> > +int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
> > +                                  enum dlb2_cq_poll_modes *mode);
> > +
> > +int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
> > +                                   struct dlb2_get_num_resources_args *rsrcs);
> > diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
> > new file mode 100644
> > index 0000000..4fb416e
> > --- /dev/null
> > +++ b/drivers/event/dlb2/dlb2_iface.h
> > @@ -0,0 +1,29 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef _DLB2_IFACE_H_
> > +#define _DLB2_IFACE_H_
> > +
> > +/* DLB2 PMD Internal interface function pointers.
> > + * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
> > + * serviced by DLB kernel module.
> > + * If PCI (PF PMD),  these will be implemented locally in user mode.
> > + */
> > +
> > +extern void (*dlb2_iface_low_level_io_init)(void);
> > +
> > +extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char
> *name);
> > +
> > +extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
> > +                                           uint8_t *revision);
> > +
> > +extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
> > +
> > +extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
> > +                                         enum dlb2_cq_poll_modes *mode);
> > +
> > +extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
> > +                               struct dlb2_get_num_resources_args *rsrcs);
> > +
> > +#endif /* _DLB2_IFACE_H_ */
> > diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
> > index 54ba2c8..99b71f9 100644
> > --- a/drivers/event/dlb2/meson.build
> > +++ b/drivers/event/dlb2/meson.build
> > @@ -1,7 +1,11 @@
> >  # SPDX-License-Identifier: BSD-3-Clause
> >  # Copyright(c) 2019-2020 Intel Corporation
> >
> > -sources = files(
> > +sources = files('dlb2.c',
> > +               'dlb2_iface.c',
> > +               'pf/dlb2_main.c',
> > +               'pf/dlb2_pf.c',
> > +               'pf/base/dlb2_resource.c'
> >  )
> >
> >  deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
> > diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h
> b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
> > new file mode 100644
> > index 0000000..428a5e8
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
> > @@ -0,0 +1,367 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef __DLB2_HW_TYPES_H
> > +#define __DLB2_HW_TYPES_H
> > +
> > +#include "dlb2_user.h"
> > +
> > +#include "dlb2_osdep_list.h"
> > +#include "dlb2_osdep_types.h"
> > +
> > +#define DLB2_MAX_NUM_VDEVS                     16
> > +#define DLB2_MAX_NUM_DOMAINS                   32
> > +#define DLB2_MAX_NUM_LDB_QUEUES                        32 /* LDB == load-
> balanced */
> > +#define DLB2_MAX_NUM_DIR_QUEUES                        64 /* DIR == directed */
> > +#define DLB2_MAX_NUM_LDB_PORTS                 64
> > +#define DLB2_MAX_NUM_DIR_PORTS                 64
> > +#define DLB2_MAX_NUM_LDB_CREDITS               (8 * 1024)
> > +#define DLB2_MAX_NUM_DIR_CREDITS               (2 * 1024)
> > +#define DLB2_MAX_NUM_HIST_LIST_ENTRIES         2048
> > +#define DLB2_MAX_NUM_AQED_ENTRIES              2048
> > +#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ           8
> > +#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS    2
> > +#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES     5
> > +#define DLB2_QID_PRIORITIES                    8
> > +#define DLB2_NUM_ARB_WEIGHTS                   8
> > +#define DLB2_MAX_WEIGHT                                255
> > +#define DLB2_NUM_COS_DOMAINS                   4
> > +#define DLB2_MAX_CQ_COMP_CHECK_LOOPS           409600
> > +#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS         (32 * 64 * 1024 * (800 /
> 30))
> > +#ifdef FPGA
> > +#define DLB2_HZ                                        2000000
> > +#else
> > +#define DLB2_HZ                                        800000000
> > +#endif
> > +
> > +#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
> > +#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
> > +
> > +/* Interrupt related macros */
> > +#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
> > +#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
> > +#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
> > +       (DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
> > +        DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
> > +#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
> > +       (DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
> > +#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
> > +       DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
> > +#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
> > +       DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
> > +
> > +/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
> > +#define DLB2_INT_NON_CQ 0
> > +
> > +#define DLB2_ALARM_HW_SOURCE_SYS 0
> > +#define DLB2_ALARM_HW_SOURCE_DLB 1
> > +
> > +#define DLB2_ALARM_HW_UNIT_CHP 4
> > +
> > +#define DLB2_ALARM_SYS_AID_ILLEGAL_QID         3
> > +#define DLB2_ALARM_SYS_AID_DISABLED_QID                4
> > +#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW         5
> > +#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ      1
> > +#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
> > +
> > +#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
> > +#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
> > +#define DLB2_VF_BASE_CQ_VECTOR_ID           0
> > +#define DLB2_VF_LAST_CQ_VECTOR_ID           30
> > +#define DLB2_VF_MBOX_VECTOR_ID              31
> > +#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
> > +       (DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
> > +        DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
> > +
> > +#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS
> (DLB2_MAX_NUM_LDB_PORTS + \
> > +                                            DLB2_MAX_NUM_DIR_PORTS + 1)
> > +
> > +/*
> > + * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only
> used by
> > + * the PF driver.
> > + */
> > +#define DLB2_DRV_LDB_PP_BASE   0x2300000
> > +#define DLB2_DRV_LDB_PP_STRIDE 0x1000
> > +#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
> > +                               DLB2_DRV_LDB_PP_STRIDE *
> DLB2_MAX_NUM_LDB_PORTS)
> > +#define DLB2_DRV_DIR_PP_BASE   0x2200000
> > +#define DLB2_DRV_DIR_PP_STRIDE 0x1000
> > +#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
> > +                               DLB2_DRV_DIR_PP_STRIDE *
> DLB2_MAX_NUM_DIR_PORTS)
> > +#define DLB2_LDB_PP_BASE       0x2100000
> > +#define DLB2_LDB_PP_STRIDE     0x1000
> > +#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
> > +                               DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
> > +#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
> > +#define DLB2_DIR_PP_BASE       0x2000000
> > +#define DLB2_DIR_PP_STRIDE     0x1000
> > +#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
> > +                               DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
> > +#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
> > +
> > +struct dlb2_resource_id {
> > +       u32 phys_id;
> > +       u32 virt_id;
> > +       u8 vdev_owned;
> > +       u8 vdev_id;
> > +};
> > +
> > +struct dlb2_freelist {
> > +       u32 base;
> > +       u32 bound;
> > +       u32 offset;
> > +};
> > +
> > +static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
> > +{
> > +       return list->bound - list->base - list->offset;
> > +}
> > +
> > +struct dlb2_hcw {
> > +       u64 data;
> > +       /* Word 3 */
> > +       u16 opaque;
> > +       u8 qid;
> > +       u8 sched_type:2;
> > +       u8 priority:3;
> > +       u8 msg_type:3;
> > +       /* Word 4 */
> > +       u16 lock_id;
> > +       u8 ts_flag:1;
> > +       u8 rsvd1:2;
> > +       u8 no_dec:1;
> > +       u8 cmp_id:4;
> > +       u8 cq_token:1;
> > +       u8 qe_comp:1;
> > +       u8 qe_frag:1;
> > +       u8 qe_valid:1;
> > +       u8 int_arm:1;
> > +       u8 error:1;
> > +       u8 rsvd:2;
> > +};
> > +
> > +struct dlb2_ldb_queue {
> > +       struct dlb2_list_entry domain_list;
> > +       struct dlb2_list_entry func_list;
> > +       struct dlb2_resource_id id;
> > +       struct dlb2_resource_id domain_id;
> > +       u32 num_qid_inflights;
> > +       u32 aqed_limit;
> > +       u32 sn_group; /* sn == sequence number */
> > +       u32 sn_slot;
> > +       u32 num_mappings;
> > +       u8 sn_cfg_valid;
> > +       u8 num_pending_additions;
> > +       u8 owned;
> > +       u8 configured;
> > +};
> > +
> > +/*
> > + * Directed ports and queues are paired by nature, so the driver tracks them
> > + * with a single data structure.
> > + */
> > +struct dlb2_dir_pq_pair {
> > +       struct dlb2_list_entry domain_list;
> > +       struct dlb2_list_entry func_list;
> > +       struct dlb2_resource_id id;
> > +       struct dlb2_resource_id domain_id;
> > +       u32 ref_cnt;
> > +       u8 init_tkn_cnt;
> > +       u8 queue_configured;
> > +       u8 port_configured;
> > +       u8 owned;
> > +       u8 enabled;
> > +};
> > +
> > +enum dlb2_qid_map_state {
> > +       /* The slot doesn't contain a valid queue mapping */
> > +       DLB2_QUEUE_UNMAPPED,
> > +       /* The slot contains a valid queue mapping */
> > +       DLB2_QUEUE_MAPPED,
> > +       /* The driver is mapping a queue into this slot */
> > +       DLB2_QUEUE_MAP_IN_PROG,
> > +       /* The driver is unmapping a queue from this slot */
> > +       DLB2_QUEUE_UNMAP_IN_PROG,
> > +       /*
> > +        * The driver is unmapping a queue from this slot, and once complete
> > +        * will replace it with another mapping.
> > +        */
> > +       DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
> > +};
> > +
> > +struct dlb2_ldb_port_qid_map {
> > +       enum dlb2_qid_map_state state;
> > +       u16 qid;
> > +       u16 pending_qid;
> > +       u8 priority;
> > +       u8 pending_priority;
> > +};
> > +
> > +struct dlb2_ldb_port {
> > +       struct dlb2_list_entry domain_list;
> > +       struct dlb2_list_entry func_list;
> > +       struct dlb2_resource_id id;
> > +       struct dlb2_resource_id domain_id;
> > +       /* The qid_map represents the hardware QID mapping state. */
> > +       struct dlb2_ldb_port_qid_map
> qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
> > +       u32 hist_list_entry_base;
> > +       u32 hist_list_entry_limit;
> > +       u32 ref_cnt;
> > +       u8 init_tkn_cnt;
> > +       u8 num_pending_removals;
> > +       u8 num_mappings;
> > +       u8 owned;
> > +       u8 enabled;
> > +       u8 configured;
> > +};
> > +
> > +struct dlb2_sn_group {
> > +       u32 mode;
> > +       u32 sequence_numbers_per_queue;
> > +       u32 slot_use_bitmap;
> > +       u32 id;
> > +};
> > +
> > +static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
> > +{
> > +       u32 mask[] = {
> > +               0x0000ffff,  /* 64 SNs per queue */
> > +               0x000000ff,  /* 128 SNs per queue */
> > +               0x0000000f,  /* 256 SNs per queue */
> > +               0x00000003,  /* 512 SNs per queue */
> > +               0x00000001}; /* 1024 SNs per queue */
> > +
> > +       return group->slot_use_bitmap == mask[group->mode];
> > +}
> > +
> > +static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
> > +{
> > +       u32 bound[6] = {16, 8, 4, 2, 1};
> > +       u32 i;
> > +
> > +       for (i = 0; i < bound[group->mode]; i++) {
> > +               if (!(group->slot_use_bitmap & (1 << i))) {
> > +                       group->slot_use_bitmap |= 1 << i;
> > +                       return i;
> > +               }
> > +       }
> > +
> > +       return -1;
> > +}
> > +
> > +static inline void
> > +dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
> > +{
> > +       group->slot_use_bitmap &= ~(1 << slot);
> > +}
> > +
> > +static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
> > +{
> > +       int i, cnt = 0;
> > +
> > +       for (i = 0; i < 32; i++)
> > +               cnt += !!(group->slot_use_bitmap & (1 << i));
> > +
> > +       return cnt;
> > +}
> > +
> > +struct dlb2_hw_domain {
> > +       struct dlb2_function_resources *parent_func;
> > +       struct dlb2_list_entry func_list;
> > +       struct dlb2_list_head used_ldb_queues;
> > +       struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
> > +       struct dlb2_list_head used_dir_pq_pairs;
> > +       struct dlb2_list_head avail_ldb_queues;
> > +       struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
> > +       struct dlb2_list_head avail_dir_pq_pairs;
> > +       u32 total_hist_list_entries;
> > +       u32 avail_hist_list_entries;
> > +       u32 hist_list_entry_base;
> > +       u32 hist_list_entry_offset;
> > +       u32 num_ldb_credits;
> > +       u32 num_dir_credits;
> > +       u32 num_avail_aqed_entries;
> > +       u32 num_used_aqed_entries;
> > +       struct dlb2_resource_id id;
> > +       int num_pending_removals;
> > +       int num_pending_additions;
> > +       u8 configured;
> > +       u8 started;
> > +};
> > +
> > +struct dlb2_bitmap;
> > +
> > +struct dlb2_function_resources {
> > +       struct dlb2_list_head avail_domains;
> > +       struct dlb2_list_head used_domains;
> > +       struct dlb2_list_head avail_ldb_queues;
> > +       struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
> > +       struct dlb2_list_head avail_dir_pq_pairs;
> > +       struct dlb2_bitmap *avail_hist_list_entries;
> > +       u32 num_avail_domains;
> > +       u32 num_avail_ldb_queues;
> > +       u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
> > +       u32 num_avail_dir_pq_pairs;
> > +       u32 num_avail_qed_entries;
> > +       u32 num_avail_dqed_entries;
> > +       u32 num_avail_aqed_entries;
> > +       u8 locked; /* (VDEV only) */
> > +};
> > +
> > +/*
> > + * After initialization, each resource in dlb2_hw_resources is located in one
> > + * of the following lists:
> > + * -- The PF's available resources list. These are unconfigured resources
> owned
> > + *     by the PF and not allocated to a dlb2 scheduling domain.
> > + * -- A VDEV's available resources list. These are VDEV-owned unconfigured
> > + *     resources not allocated to a dlb2 scheduling domain.
> > + * -- A domain's available resources list. These are domain-owned
> unconfigured
> > + *     resources.
> > + * -- A domain's used resources list. These are are domain-owned configured
> > + *     resources.
> > + *
> > + * A resource moves to a new list when a VDEV or domain is created or
> destroyed,
> > + * or when the resource is configured.
> > + */
> > +struct dlb2_hw_resources {
> > +       struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
> > +       struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
> > +       struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
> > +       struct dlb2_sn_group
> sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
> > +};
> > +
> > +struct dlb2_mbox {
> > +       u32 *mbox;
> > +       u32 *isr_in_progress;
> > +};
> > +
> > +struct dlb2_sw_mbox {
> > +       struct dlb2_mbox vdev_to_pf;
> > +       struct dlb2_mbox pf_to_vdev;
> > +       void (*pf_to_vdev_inject)(void *arg);
> > +       void *pf_to_vdev_inject_arg;
> > +};
> > +
> > +struct dlb2_hw {
> > +       /* BAR 0 address */
> > +       void  *csr_kva;
> > +       unsigned long csr_phys_addr;
> > +       /* BAR 2 address */
> > +       void  *func_kva;
> > +       unsigned long func_phys_addr;
> > +
> > +       /* Resource tracking */
> > +       struct dlb2_hw_resources rsrcs;
> > +       struct dlb2_function_resources pf;
> > +       struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
> > +       struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
> > +       u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
> > +
> > +       /* Virtualization */
> > +       int virt_mode;
> > +       struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
> > +       unsigned int pasid[DLB2_MAX_NUM_VDEVS];
> > +};
> > +
> > +#endif /* __DLB2_HW_TYPES_H */
> > diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h
> b/drivers/event/dlb2/pf/base/dlb2_mbox.h
> > new file mode 100644
> > index 0000000..ce462c0
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
> > @@ -0,0 +1,596 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef __DLB2_BASE_DLB2_MBOX_H
> > +#define __DLB2_BASE_DLB2_MBOX_H
> > +
> > +#include "dlb2_osdep_types.h"
> > +#include "dlb2_regs.h"
> > +
> > +#define DLB2_MBOX_INTERFACE_VERSION 1
> > +
> > +/*
> > + * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
> > + * to send requests of its own (e.g. notifying a VF of an impending FLR).
> > + * To avoid communication race conditions, e.g. the PF sends a response and
> then
> > + * sends a request before the VF reads the response, the PF->VF mailbox is
> > + * divided into two sections:
> > + * - Bytes 0-47: PF responses
> > + * - Bytes 48-63: PF requests
> > + *
> > + * Partitioning the PF->VF mailbox allows responses and requests to occupy
> the
> > + * mailbox simultaneously.
> > + */
> > +#define DLB2_PF2VF_RESP_BYTES    48
> > +#define DLB2_PF2VF_RESP_BASE     0
> > +#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
> > +
> > +#define DLB2_PF2VF_REQ_BYTES     16
> > +#define DLB2_PF2VF_REQ_BASE      (DLB2_PF2VF_RESP_BASE +
> DLB2_PF2VF_RESP_BYTES)
> > +#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
> > +
> > +/*
> > + * Similarly, the VF->PF mailbox is divided into two sections:
> > + * - Bytes 0-239: VF requests
> > + * -- (Bytes 0-3 are unused due to a hardware errata)
> > + * - Bytes 240-255: VF responses
> > + */
> > +#define DLB2_VF2PF_REQ_BYTES    236
> > +#define DLB2_VF2PF_REQ_BASE     4
> > +#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
> > +
> > +#define DLB2_VF2PF_RESP_BYTES    16
> > +#define DLB2_VF2PF_RESP_BASE     (DLB2_VF2PF_REQ_BASE +
> DLB2_VF2PF_REQ_BYTES)
> > +#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
> > +
> > +/* VF-initiated commands */
> > +enum dlb2_mbox_cmd_type {
> > +       DLB2_MBOX_CMD_REGISTER,
> > +       DLB2_MBOX_CMD_UNREGISTER,
> > +       DLB2_MBOX_CMD_GET_NUM_RESOURCES,
> > +       DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
> > +       DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
> > +       DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
> > +       DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
> > +       DLB2_MBOX_CMD_CREATE_LDB_PORT,
> > +       DLB2_MBOX_CMD_CREATE_DIR_PORT,
> > +       DLB2_MBOX_CMD_ENABLE_LDB_PORT,
> > +       DLB2_MBOX_CMD_DISABLE_LDB_PORT,
> > +       DLB2_MBOX_CMD_ENABLE_DIR_PORT,
> > +       DLB2_MBOX_CMD_DISABLE_DIR_PORT,
> > +       DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
> > +       DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
> > +       DLB2_MBOX_CMD_MAP_QID,
> > +       DLB2_MBOX_CMD_UNMAP_QID,
> > +       DLB2_MBOX_CMD_START_DOMAIN,
> > +       DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
> > +       DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
> > +       DLB2_MBOX_CMD_ARM_CQ_INTR,
> > +       DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
> > +       DLB2_MBOX_CMD_GET_SN_ALLOCATION,
> > +       DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
> > +       DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
> > +       DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
> > +       DLB2_MBOX_CMD_GET_COS_BW,
> > +       DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
> > +       DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
> > +
> > +       /* NUM_QE_CMD_TYPES must be last */
> > +       NUM_DLB2_MBOX_CMD_TYPES,
> > +};
> > +
> > +static const char dlb2_mbox_cmd_type_strings[][128] = {
> > +       "DLB2_MBOX_CMD_REGISTER",
> > +       "DLB2_MBOX_CMD_UNREGISTER",
> > +       "DLB2_MBOX_CMD_GET_NUM_RESOURCES",
> > +       "DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
> > +       "DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
> > +       "DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
> > +       "DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
> > +       "DLB2_MBOX_CMD_CREATE_LDB_PORT",
> > +       "DLB2_MBOX_CMD_CREATE_DIR_PORT",
> > +       "DLB2_MBOX_CMD_ENABLE_LDB_PORT",
> > +       "DLB2_MBOX_CMD_DISABLE_LDB_PORT",
> > +       "DLB2_MBOX_CMD_ENABLE_DIR_PORT",
> > +       "DLB2_MBOX_CMD_DISABLE_DIR_PORT",
> > +       "DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
> > +       "DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
> > +       "DLB2_MBOX_CMD_MAP_QID",
> > +       "DLB2_MBOX_CMD_UNMAP_QID",
> > +       "DLB2_MBOX_CMD_START_DOMAIN",
> > +       "DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
> > +       "DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
> > +       "DLB2_MBOX_CMD_ARM_CQ_INTR",
> > +       "DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
> > +       "DLB2_MBOX_CMD_GET_SN_ALLOCATION",
> > +       "DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
> > +       "DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
> > +       "DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
> > +       "DLB2_MBOX_CMD_GET_COS_BW",
> > +       "DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
> > +       "DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
> > +};
> > +
> > +/* PF-initiated commands */
> > +enum dlb2_mbox_vf_cmd_type {
> > +       DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
> > +       DLB2_MBOX_VF_CMD_NOTIFICATION,
> > +       DLB2_MBOX_VF_CMD_IN_USE,
> > +
> > +       /* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
> > +       NUM_DLB2_MBOX_VF_CMD_TYPES,
> > +};
> > +
> > +static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
> > +       "DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
> > +       "DLB2_MBOX_VF_CMD_NOTIFICATION",
> > +       "DLB2_MBOX_VF_CMD_IN_USE",
> > +};
> > +
> > +#define DLB2_MBOX_CMD_TYPE(hdr) \
> > +       (((struct dlb2_mbox_req_hdr *)hdr)->type)
> > +#define DLB2_MBOX_CMD_STRING(hdr) \
> > +       dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
> > +
> > +enum dlb2_mbox_status_type {
> > +       DLB2_MBOX_ST_SUCCESS,
> > +       DLB2_MBOX_ST_INVALID_CMD_TYPE,
> > +       DLB2_MBOX_ST_VERSION_MISMATCH,
> > +       DLB2_MBOX_ST_INVALID_OWNER_VF,
> > +};
> > +
> > +static const char dlb2_mbox_status_type_strings[][128] = {
> > +       "DLB2_MBOX_ST_SUCCESS",
> > +       "DLB2_MBOX_ST_INVALID_CMD_TYPE",
> > +       "DLB2_MBOX_ST_VERSION_MISMATCH",
> > +       "DLB2_MBOX_ST_INVALID_OWNER_VF",
> > +};
> > +
> > +#define DLB2_MBOX_ST_TYPE(hdr) \
> > +       (((struct dlb2_mbox_resp_hdr *)hdr)->status)
> > +#define DLB2_MBOX_ST_STRING(hdr) \
> > +       dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
> > +
> > +/* This structure is always the first field in a request structure */
> > +struct dlb2_mbox_req_hdr {
> > +       u32 type;
> > +};
> > +
> > +/* This structure is always the first field in a response structure */
> > +struct dlb2_mbox_resp_hdr {
> > +       u32 status;
> > +};
> > +
> > +struct dlb2_mbox_register_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u16 min_interface_version;
> > +       u16 max_interface_version;
> > +};
> > +
> > +struct dlb2_mbox_register_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 interface_version;
> > +       u8 pf_id;
> > +       u8 vf_id;
> > +       u8 is_auxiliary_vf;
> > +       u8 primary_vf_id;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_unregister_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_unregister_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_get_num_resources_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_get_num_resources_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u16 num_sched_domains;
> > +       u16 num_ldb_queues;
> > +       u16 num_ldb_ports;
> > +       u16 num_cos_ldb_ports[4];
> > +       u16 num_dir_ports;
> > +       u32 num_atomic_inflights;
> > +       u32 num_hist_list_entries;
> > +       u32 max_contiguous_hist_list_entries;
> > +       u16 num_ldb_credits;
> > +       u16 num_dir_credits;
> > +};
> > +
> > +struct dlb2_mbox_create_sched_domain_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 num_ldb_queues;
> > +       u32 num_ldb_ports;
> > +       u32 num_cos_ldb_ports[4];
> > +       u32 num_dir_ports;
> > +       u32 num_atomic_inflights;
> > +       u32 num_hist_list_entries;
> > +       u32 num_ldb_credits;
> > +       u32 num_dir_credits;
> > +       u8 cos_strict;
> > +       u8 padding0[3];
> > +       u32 padding1;
> > +};
> > +
> > +struct dlb2_mbox_create_sched_domain_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 id;
> > +};
> > +
> > +struct dlb2_mbox_reset_sched_domain_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 id;
> > +};
> > +
> > +struct dlb2_mbox_reset_sched_domain_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +};
> > +
> > +struct dlb2_mbox_create_ldb_queue_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 num_sequence_numbers;
> > +       u32 num_qid_inflights;
> > +       u32 num_atomic_inflights;
> > +       u32 lock_id_comp_level;
> > +       u32 depth_threshold;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_create_ldb_queue_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 id;
> > +};
> > +
> > +struct dlb2_mbox_create_dir_queue_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 port_id;
> > +       u32 depth_threshold;
> > +};
> > +
> > +struct dlb2_mbox_create_dir_queue_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 id;
> > +};
> > +
> > +struct dlb2_mbox_create_ldb_port_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u16 cq_depth;
> > +       u16 cq_history_list_size;
> > +       u8 cos_id;
> > +       u8 cos_strict;
> > +       u16 padding1;
> > +       u64 cq_base_address;
> > +};
> > +
> > +struct dlb2_mbox_create_ldb_port_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 id;
> > +};
> > +
> > +struct dlb2_mbox_create_dir_port_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u64 cq_base_address;
> > +       u16 cq_depth;
> > +       u16 padding0;
> > +       s32 queue_id;
> > +};
> > +
> > +struct dlb2_mbox_create_dir_port_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 id;
> > +};
> > +
> > +struct dlb2_mbox_enable_ldb_port_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 port_id;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_enable_ldb_port_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_disable_ldb_port_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 port_id;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_disable_ldb_port_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_enable_dir_port_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 port_id;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_enable_dir_port_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_disable_dir_port_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 port_id;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_disable_dir_port_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 port_id;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       s32 owned;
> > +};
> > +
> > +struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 port_id;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       s32 owned;
> > +};
> > +
> > +struct dlb2_mbox_map_qid_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 port_id;
> > +       u32 qid;
> > +       u32 priority;
> > +       u32 padding0;
> > +};
> > +
> > +struct dlb2_mbox_map_qid_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 id;
> > +};
> > +
> > +struct dlb2_mbox_unmap_qid_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 port_id;
> > +       u32 qid;
> > +};
> > +
> > +struct dlb2_mbox_unmap_qid_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_start_domain_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +};
> > +
> > +struct dlb2_mbox_start_domain_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u16 port_id;
> > +       u16 thresh;
> > +       u16 vector;
> > +       u16 owner_vf;
> > +       u16 reserved[2];
> > +};
> > +
> > +struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_enable_dir_port_intr_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u16 port_id;
> > +       u16 thresh;
> > +       u16 vector;
> > +       u16 owner_vf;
> > +       u16 reserved[2];
> > +};
> > +
> > +struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_arm_cq_intr_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 port_id;
> > +       u32 is_ldb;
> > +};
> > +
> > +struct dlb2_mbox_arm_cq_intr_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 padding0;
> > +};
> > +
> > +/*
> > + * The alert_id and aux_alert_data follows the format of the alerts defined in
> > + * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value,
> and
> > + * the aux_alert_data value varies depending on the alert.
> > + */
> > +struct dlb2_mbox_vf_alert_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 alert_id;
> > +       u32 aux_alert_data;
> > +};
> > +
> > +enum dlb2_mbox_vf_notification_type {
> > +       DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
> > +       DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
> > +
> > +       /* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
> > +       NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
> > +};
> > +
> > +struct dlb2_mbox_vf_notification_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 notification;
> > +};
> > +
> > +struct dlb2_mbox_vf_in_use_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_vf_in_use_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 in_use;
> > +};
> > +
> > +struct dlb2_mbox_get_sn_allocation_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 group_id;
> > +};
> > +
> > +struct dlb2_mbox_get_sn_allocation_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 num;
> > +};
> > +
> > +struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 queue_id;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 depth;
> > +};
> > +
> > +struct dlb2_mbox_get_dir_queue_depth_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 queue_id;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 depth;
> > +};
> > +
> > +struct dlb2_mbox_pending_port_unmaps_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 domain_id;
> > +       u32 port_id;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_pending_port_unmaps_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 num;
> > +};
> > +
> > +struct dlb2_mbox_get_cos_bw_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 cos_id;
> > +};
> > +
> > +struct dlb2_mbox_get_cos_bw_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 num;
> > +};
> > +
> > +struct dlb2_mbox_get_sn_occupancy_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 group_id;
> > +};
> > +
> > +struct dlb2_mbox_get_sn_occupancy_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 num;
> > +};
> > +
> > +struct dlb2_mbox_query_cq_poll_mode_cmd_req {
> > +       struct dlb2_mbox_req_hdr hdr;
> > +       u32 padding;
> > +};
> > +
> > +struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
> > +       struct dlb2_mbox_resp_hdr hdr;
> > +       u32 error_code;
> > +       u32 status;
> > +       u32 mode;
> > +};
> > +
> > +#endif /* __DLB2_BASE_DLB2_MBOX_H */
> > diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h
> b/drivers/event/dlb2/pf/base/dlb2_osdep.h
> > new file mode 100644
> > index 0000000..43f2125
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
> > @@ -0,0 +1,247 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef __DLB2_OSDEP_H
> > +#define __DLB2_OSDEP_H
> > +
> > +#include <string.h>
> > +#include <time.h>
> > +#include <unistd.h>
> > +#include <pthread.h>
> > +
> > +#include <rte_string_fns.h>
> > +#include <rte_cycles.h>
> > +#include <rte_io.h>
> > +#include <rte_log.h>
> > +#include <rte_spinlock.h>
> > +#include "../dlb2_main.h"
> > +#include "dlb2_resource.h"
> > +#include "../../dlb2_log.h"
> > +#include "../../dlb2_user.h"
> > +
> > +
> > +#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
> > +#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
> > +
> > +/* Read/write register 'reg' in the CSR BAR space */
> > +#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva +
> (reg)))
> > +#define DLB2_CSR_RD(hw, reg) \
> > +       DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
> > +#define DLB2_CSR_WR(hw, reg, value) \
> > +       DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
> > +
> > +/* Read/write register 'reg' in the func BAR space */
> > +#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva +
> (reg)))
> > +#define DLB2_FUNC_RD(hw, reg) \
> > +       DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
> > +#define DLB2_FUNC_WR(hw, reg, value) \
> > +       DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
> > +
> > +/* Map to PMDs logging interface */
> > +#define DLB2_ERR(dev, fmt, args...) \
> > +       DLB2_LOG_ERR(fmt, ## args)
> > +
> > +#define DLB2_INFO(dev, fmt, args...) \
> > +       DLB2_LOG_INFO(fmt, ## args)
> > +
> > +#define DLB2_DEBUG(dev, fmt, args...) \
> > +       DLB2_LOG_DBG(fmt, ## args)
> > +
> > +/**
> > + * os_udelay() - busy-wait for a number of microseconds
> > + * @usecs: delay duration.
> > + */
> > +static inline void os_udelay(int usecs)
> > +{
> > +       rte_delay_us(usecs);
> > +}
> > +
> > +/**
> > + * os_msleep() - sleep for a number of milliseconds
> > + * @usecs: delay duration.
> > + */
> > +static inline void os_msleep(int msecs)
> > +{
> > +       rte_delay_ms(msecs);
> > +}
> > +
> > +#define DLB2_PP_BASE(__is_ldb) \
> > +       ((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
> > +
> > +/**
> > + * os_map_producer_port() - map a producer port into the caller's address
> space
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @port_id: port ID
> > + * @is_ldb: true for load-balanced port, false for a directed port
> > + *
> > + * This function maps the requested producer port memory into the caller's
> > + * address space.
> > + *
> > + * Return:
> > + * Returns the base address at which the PP memory was mapped, else NULL.
> > + */
> > +static inline void *os_map_producer_port(struct dlb2_hw *hw,
> > +                                        u8 port_id,
> > +                                        bool is_ldb)
> > +{
> > +       uint64_t addr;
> > +       uint64_t pp_dma_base;
> > +
> > +       pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
> > +       addr = (pp_dma_base + (PAGE_SIZE * port_id));
> > +
> > +       return (void *)(uintptr_t)addr;
> > +}
> > +
> > +/**
> > + * os_unmap_producer_port() - unmap a producer port
> > + * @addr: mapped producer port address
> > + *
> > + * This function undoes os_map_producer_port() by unmapping the producer
> port
> > + * memory from the caller's address space.
> > + *
> > + * Return:
> > + * Returns the base address at which the PP memory was mapped, else NULL.
> > + */
> > +static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
> > +{
> > +       RTE_SET_USED(hw);
> > +       RTE_SET_USED(addr);
> > +}
> > +
> > +/**
> > + * os_fence_hcw() - fence an HCW to ensure it arrives at the device
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @pp_addr: producer port address
> > + */
> > +static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
> > +{
> > +       RTE_SET_USED(hw);
> > +
> > +       /* To ensure outstanding HCWs reach the device, read the PP address. IA
> > +        * memory ordering prevents reads from passing older writes, and the
> > +        * mfence also ensures this.
> > +        */
> > +       rte_mb();
> > +
> > +       *(volatile u64 *)pp_addr;
> > +}
> > +
> > +/**
> > + * os_enqueue_four_hcws() - enqueue four HCWs to DLB
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @hcw: pointer to the 64B-aligned contiguous HCW memory
> > + * @addr: producer port address
> > + */
> > +static inline void os_enqueue_four_hcws(struct dlb2_hw *hw,
> > +                                       struct dlb2_hcw *hcw,
> > +                                       void *addr)
> > +{
> > +       struct dlb2_dev *dlb2_dev;
> > +
> > +       dlb2_dev = container_of(hw, struct dlb2_dev, hw);
> > +
> > +       dlb2_dev->enqueue_four(hcw, addr);
> > +}
> > +
> > +/**
> > + * DLB2_HW_ERR() - log an error message
> > + * @dlb2: dlb2_hw handle for a particular device.
> > + * @...: variable string args.
> > + */
> > +#define DLB2_HW_ERR(dlb2, ...) do {    \
> > +       RTE_SET_USED(dlb2);             \
> > +       DLB2_ERR(dlb2, __VA_ARGS__);    \
> > +} while (0)
> > +
> > +/**
> > + * DLB2_HW_DBG() - log an info message
> > + * @dlb2: dlb2_hw handle for a particular device.
> > + * @...: variable string args.
> > + */
> > +#define DLB2_HW_DBG(dlb2, ...) do {    \
> > +       RTE_SET_USED(dlb2);             \
> > +       DLB2_DEBUG(dlb2, __VA_ARGS__);  \
> > +} while (0)
> > +
> > +/* The callback runs until it completes all outstanding QID->CQ
> > + * map and unmap requests. To prevent deadlock, this function gives other
> > + * threads a chance to grab the resource mutex and configure hardware.
> > + */
> > +static void *dlb2_complete_queue_map_unmap(void *__args)
> > +{
> > +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
> > +       int ret;
> > +
> > +       while (1) {
> > +               rte_spinlock_lock(&dlb2_dev->resource_mutex);
> > +
> > +               ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
> > +               ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
> > +
> > +               if (ret != 0) {
> > +                       rte_spinlock_unlock(&dlb2_dev->resource_mutex);
> > +                       /* Relinquish the CPU so the application can process
> > +                        * its CQs, so this function doesn't deadlock.
> > +                        */
> > +                       sched_yield();
> > +               } else {
> > +                       break;
> > +               }
> > +       }
> > +
> > +       dlb2_dev->worker_launched = false;
> > +
> > +       rte_spinlock_unlock(&dlb2_dev->resource_mutex);
> > +
> > +       return NULL;
> > +}
> > +
> > +
> > +/**
> > + * os_schedule_work() - launch a thread to process pending map and unmap
> work
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function launches a kernel thread that will run until all pending
> > + * map and unmap procedures are complete.
> > + */
> > +static inline void os_schedule_work(struct dlb2_hw *hw)
> > +{
> > +       struct dlb2_dev *dlb2_dev;
> > +       pthread_t complete_queue_map_unmap_thread;
> > +       int ret;
> > +
> > +       dlb2_dev = container_of(hw, struct dlb2_dev, hw);
> > +
> > +       ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
> > +                                    "dlb_queue_unmap_waiter",
> > +                                    NULL,
> > +                                    dlb2_complete_queue_map_unmap,
> > +                                    dlb2_dev);
> > +       if (ret)
> > +               DLB2_ERR(dlb2_dev,
> > +                        "Could not create queue complete map/unmap thread,
> err=%d\n",
> > +                        ret);
> > +       else
> > +               dlb2_dev->worker_launched = true;
> > +}
> > +
> > +/**
> > + * os_worker_active() - query whether the map/unmap worker thread is
> active
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function returns a boolean indicating whether a thread (launched by
> > + * os_schedule_work()) is active. This function is used to determine
> > + * whether or not to launch a worker thread.
> > + */
> > +static inline bool os_worker_active(struct dlb2_hw *hw)
> > +{
> > +       struct dlb2_dev *dlb2_dev;
> > +
> > +       dlb2_dev = container_of(hw, struct dlb2_dev, hw);
> > +
> > +       return dlb2_dev->worker_launched;
> > +}
> > +
> > +#endif /*  __DLB2_OSDEP_H */
> > diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
> b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
> > new file mode 100644
> > index 0000000..423233b
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
> > @@ -0,0 +1,440 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef __DLB2_OSDEP_BITMAP_H
> > +#define __DLB2_OSDEP_BITMAP_H
> > +
> > +#include <stdint.h>
> > +#include <stdbool.h>
> > +#include <stdio.h>
> > +#include <unistd.h>
> > +#include <rte_bitmap.h>
> > +#include <rte_string_fns.h>
> > +#include <rte_malloc.h>
> > +#include <rte_errno.h>
> > +#include "../dlb2_main.h"
> > +
> > +/*************************/
> > +/*** Bitmap operations ***/
> > +/*************************/
> > +struct dlb2_bitmap {
> > +       struct rte_bitmap *map;
> > +       unsigned int len;
> > +};
> > +
> > +/**
> > + * dlb2_bitmap_alloc() - alloc a bitmap data structure
> > + * @bitmap: pointer to dlb2_bitmap structure pointer.
> > + * @len: number of entries in the bitmap.
> > + *
> > + * This function allocates a bitmap and initializes it with length @len. All
> > + * entries are initially zero.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - bitmap is NULL or len is 0.
> > + * ENOMEM - could not allocate memory for the bitmap data structure.
> > + */
> > +static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
> > +                                   unsigned int len)
> > +{
> > +       struct dlb2_bitmap *bm;
> > +       void *mem;
> > +       uint32_t alloc_size;
> > +       uint32_t nbits = (uint32_t)len;
> > +
> > +       if (bitmap == NULL || nbits == 0)
> > +               return -EINVAL;
> > +
> > +       /* Allocate DLB2 bitmap control struct */
> > +       bm = rte_malloc("DLB2_PF",
> > +                       sizeof(struct dlb2_bitmap),
> > +                       RTE_CACHE_LINE_SIZE);
> > +
> > +       if (bm == NULL)
> > +               return -ENOMEM;
> > +
> > +       /* Allocate bitmap memory */
> > +       alloc_size = rte_bitmap_get_memory_footprint(nbits);
> > +       mem = rte_malloc("DLB2_PF_BITMAP", alloc_size,
> RTE_CACHE_LINE_SIZE);
> > +       if (mem == NULL) {
> > +               rte_free(bm);
> > +               return -ENOMEM;
> > +       }
> > +
> > +       bm->map = rte_bitmap_init(len, mem, alloc_size);
> > +       if (bm->map == NULL) {
> > +               rte_free(mem);
> > +               rte_free(bm);
> > +               return -ENOMEM;
> > +       }
> > +
> > +       bm->len = len;
> > +
> > +       *bitmap = bm;
> > +
> > +       return 0;
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_free() - free a previously allocated bitmap data structure
> > + * @bitmap: pointer to dlb2_bitmap structure.
> > + *
> > + * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
> > + */
> > +static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
> > +{
> > +       if (bitmap == NULL)
> > +               return;
> > +
> > +       rte_free(bitmap->map);
> > +       rte_free(bitmap);
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_fill() - fill a bitmap with all 1s
> > + * @bitmap: pointer to dlb2_bitmap structure.
> > + *
> > + * This function sets all bitmap values to 1.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - bitmap is NULL or is uninitialized.
> > + */
> > +static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
> > +{
> > +       unsigned int i;
> > +
> > +       if (bitmap  == NULL || bitmap->map == NULL)
> > +               return -EINVAL;
> > +
> > +       for (i = 0; i != bitmap->len; i++)
> > +               rte_bitmap_set(bitmap->map, i);
> > +
> > +       return 0;
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_fill() - fill a bitmap with all 0s
> > + * @bitmap: pointer to dlb2_bitmap structure.
> > + *
> > + * This function sets all bitmap values to 0.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - bitmap is NULL or is uninitialized.
> > + */
> > +static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
> > +{
> > +       if (bitmap  == NULL || bitmap->map == NULL)
> > +               return -EINVAL;
> > +
> > +       rte_bitmap_reset(bitmap->map);
> > +
> > +       return 0;
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_set() - set a bitmap entry
> > + * @bitmap: pointer to dlb2_bitmap structure.
> > + * @bit: bit index.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
> > + *         bitmap length.
> > + */
> > +static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
> > +                                 unsigned int bit)
> > +{
> > +       if (bitmap  == NULL || bitmap->map == NULL)
> > +               return -EINVAL;
> > +
> > +       if (bitmap->len <= bit)
> > +               return -EINVAL;
> > +
> > +       rte_bitmap_set(bitmap->map, bit);
> > +
> > +       return 0;
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_set_range() - set a range of bitmap entries
> > + * @bitmap: pointer to dlb2_bitmap structure.
> > + * @bit: starting bit index.
> > + * @len: length of the range.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
> > + *         length.
> > + */
> > +static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
> > +                                       unsigned int bit,
> > +                                       unsigned int len)
> > +{
> > +       unsigned int i;
> > +
> > +       if (bitmap  == NULL || bitmap->map == NULL)
> > +               return -EINVAL;
> > +
> > +       if (bitmap->len <= bit)
> > +               return -EINVAL;
> > +
> > +       for (i = 0; i != len; i++)
> > +               rte_bitmap_set(bitmap->map, bit + i);
> > +
> > +       return 0;
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_clear() - clear a bitmap entry
> > + * @bitmap: pointer to dlb2_bitmap structure.
> > + * @bit: bit index.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
> > + *         bitmap length.
> > + */
> > +static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
> > +                                   unsigned int bit)
> > +{
> > +       if (bitmap  == NULL || bitmap->map == NULL)
> > +               return -EINVAL;
> > +
> > +       if (bitmap->len <= bit)
> > +               return -EINVAL;
> > +
> > +       rte_bitmap_clear(bitmap->map, bit);
> > +
> > +       return 0;
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_clear_range() - clear a range of bitmap entries
> > + * @bitmap: pointer to dlb2_bitmap structure.
> > + * @bit: starting bit index.
> > + * @len: length of the range.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
> > + *         length.
> > + */
> > +static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
> > +                                         unsigned int bit,
> > +                                         unsigned int len)
> > +{
> > +       unsigned int i;
> > +
> > +       if (bitmap  == NULL || bitmap->map == NULL)
> > +               return -EINVAL;
> > +
> > +       if (bitmap->len <= bit)
> > +               return -EINVAL;
> > +
> > +       for (i = 0; i != len; i++)
> > +               rte_bitmap_clear(bitmap->map, bit + i);
> > +
> > +       return 0;
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_find_set_bit_range() - find an range of set bits
> > + * @bitmap: pointer to dlb2_bitmap structure.
> > + * @len: length of the range.
> > + *
> > + * This function looks for a range of set bits of length @len.
> > + *
> > + * Return:
> > + * Returns the base bit index upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * ENOENT - unable to find a length *len* range of set bits.
> > + * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
> > + */
> > +static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
> > +                                                unsigned int len)
> > +{
> > +       unsigned int i, j = 0;
> > +
> > +       if (bitmap  == NULL || bitmap->map == NULL || len == 0)
> > +               return -EINVAL;
> > +
> > +       if (bitmap->len < len)
> > +               return -ENOENT;
> > +
> > +       for (i = 0; i != bitmap->len; i++) {
> > +               if  (rte_bitmap_get(bitmap->map, i)) {
> > +                       if (++j == len)
> > +                               return i - j + 1;
> > +               } else {
> > +                       j = 0;
> > +               }
> > +       }
> > +
> > +       /* No set bit range of length len? */
> > +       return -ENOENT;
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_find_set_bit() - find an range of set bits
> > + * @bitmap: pointer to dlb2_bitmap structure.
> > + *
> > + * This function looks for a single set bit.
> > + *
> > + * Return:
> > + * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
> > + */
> > +static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
> > +{
> > +       unsigned int i;
> > +
> > +       if (bitmap == NULL)
> > +               return -EINVAL;
> > +
> > +       if (bitmap->map == NULL)
> > +               return -EINVAL;
> > +
> > +       for (i = 0; i != bitmap->len; i++) {
> > +               if  (rte_bitmap_get(bitmap->map, i))
> > +                       return i;
> > +       }
> > +
> > +       return -ENOENT;
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_count() - returns the number of set bits
> > + * @bitmap: pointer to dlb2_bitmap structure.
> > + *
> > + * This function looks for a single set bit.
> > + *
> > + * Return:
> > + * Returns the number of set bits upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - bitmap is NULL or is uninitialized.
> > + */
> > +static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
> > +{
> > +       int weight = 0;
> > +       unsigned int i;
> > +
> > +       if (bitmap == NULL)
> > +               return -EINVAL;
> > +
> > +       if (bitmap->map == NULL)
> > +               return -EINVAL;
> > +
> > +       for (i = 0; i != bitmap->len; i++) {
> > +               if  (rte_bitmap_get(bitmap->map, i))
> > +                       weight++;
> > +       }
> > +       return weight;
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
> > + *                                   bits
> > + * @bitmap: pointer to dlb2_bitmap structure.
> > + *
> > + * Return:
> > + * Returns the bitmap's longest contiguous range of of set bits upon success,
> > + * <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - bitmap is NULL or is uninitialized.
> > + */
> > +static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
> > +{
> > +       int max_len = 0, len = 0;
> > +       unsigned int i;
> > +
> > +       if (bitmap == NULL)
> > +               return -EINVAL;
> > +
> > +       if (bitmap->map == NULL)
> > +               return -EINVAL;
> > +
> > +       for (i = 0; i != bitmap->len; i++) {
> > +               if  (rte_bitmap_get(bitmap->map, i)) {
> > +                       len++;
> > +               } else {
> > +                       if (len > max_len)
> > +                               max_len = len;
> > +                       len = 0;
> > +               }
> > +       }
> > +
> > +       if (len > max_len)
> > +               max_len = len;
> > +
> > +       return max_len;
> > +}
> > +
> > +/**
> > + * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
> > + * @dest: pointer to dlb2_bitmap structure, which will contain the results of
> > + *       the 'or' of src1 and src2.
> > + * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
> > + * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
> > + *
> > + * This function 'or's two bitmaps together and stores the result in a third
> > + * bitmap. The source and destination bitmaps can be the same.
> > + *
> > + * Return:
> > + * Returns the number of set bits upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - One of the bitmaps is NULL or is uninitialized.
> > + */
> > +static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
> > +                                struct dlb2_bitmap *src1,
> > +                                struct dlb2_bitmap *src2)
> > +{
> > +       unsigned int i, min;
> > +       int numset = 0;
> > +
> > +       if (dest  == NULL || dest->map == NULL ||
> > +           src1  == NULL || src1->map == NULL ||
> > +           src2  == NULL || src2->map == NULL)
> > +               return -EINVAL;
> > +
> > +       min = dest->len;
> > +       min = (min > src1->len) ? src1->len : min;
> > +       min = (min > src2->len) ? src2->len : min;
> > +
> > +       for (i = 0; i != min; i++) {
> > +               if  (rte_bitmap_get(src1->map, i) ||
> > +                    rte_bitmap_get(src2->map, i)) {
> > +                       rte_bitmap_set(dest->map, i);
> > +                       numset++;
> > +               } else {
> > +                       rte_bitmap_clear(dest->map, i);
> > +               }
> > +       }
> > +
> > +       return numset;
> > +}
> > +
> > +#endif /*  __DLB2_OSDEP_BITMAP_H */
> > diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
> b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
> > new file mode 100644
> > index 0000000..5531739
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
> > @@ -0,0 +1,131 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef __DLB2_OSDEP_LIST_H
> > +#define __DLB2_OSDEP_LIST_H
> > +
> > +#include <rte_tailq.h>
> > +
> > +struct dlb2_list_entry {
> > +       TAILQ_ENTRY(dlb2_list_entry) node;
> > +};
> > +
> > +/* Dummy - just a struct definition */
> > +TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
> > +
> > +/* =================
> > + * TAILQ Supplements
> > + * =================
> > + */
> > +
> > +#ifndef TAILQ_FOREACH_ENTRY
> > +#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)             \
> > +       for ((iter) = TAILQ_FIRST(&head);                       \
> > +           (iter)                                              \
> > +               && (ptr = container_of(iter, typeof(*(ptr)), name)); \
> > +           (iter) = TAILQ_NEXT((iter), node))
> > +#endif
> > +
> > +#ifndef TAILQ_FOREACH_ENTRY_SAFE
> > +#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)  \
> > +       for ((iter) = TAILQ_FIRST(&head);                       \
> > +           (iter) &&                                           \
> > +               (ptr = container_of(iter, typeof(*(ptr)), name)) &&\
> > +               ((tvar) = TAILQ_NEXT((iter), node), 1); \
> > +           (iter) = (tvar))
> > +#endif
> > +
> > +/***********************/
> > +/*** List operations ***/
> > +/***********************/
> > +
> > +/**
> > + * dlb2_list_init_head() - initialize the head of a list
> > + * @head: list head
> > + */
> > +static inline void dlb2_list_init_head(struct dlb2_list_head *head)
> > +{
> > +       TAILQ_INIT(head);
> > +}
> > +
> > +/**
> > + * dlb2_list_add() - add an entry to a list
> > + * @head: list head
> > + * @entry: new list entry
> > + */
> > +static inline void
> > +dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
> > +{
> > +       TAILQ_INSERT_TAIL(head, entry, node);
> > +}
> > +
> > +/**
> > + * dlb2_list_del() - delete an entry from a list
> > + * @entry: list entry
> > + * @head: list head
> > + */
> > +static inline void dlb2_list_del(struct dlb2_list_head *head,
> > +                                struct dlb2_list_entry *entry)
> > +{
> > +       TAILQ_REMOVE(head, entry, node);
> > +}
> > +
> > +/**
> > + * dlb2_list_empty() - check if a list is empty
> > + * @head: list head
> > + *
> > + * Return:
> > + * Returns 1 if empty, 0 if not.
> > + */
> > +static inline int dlb2_list_empty(struct dlb2_list_head *head)
> > +{
> > +       return TAILQ_EMPTY(head);
> > +}
> > +
> > +/**
> > + * dlb2_list_splice() - splice a list
> > + * @src_head: list to be added
> > + * @ head: where src_head will be inserted
> > + */
> > +static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
> > +                                   struct dlb2_list_head *head)
> > +{
> > +       TAILQ_CONCAT(head, src_head, node);
> > +}
> > +
> > +/**
> > + * DLB2_LIST_HEAD() - retrieve the head of the list
> > + * @head: list head
> > + * @type: type of the list variable
> > + * @name: name of the list field within the containing struct
> > + */
> > +#define DLB2_LIST_HEAD(head, type, name)                       \
> > +       (TAILQ_FIRST(&head) ?                                   \
> > +               container_of(TAILQ_FIRST(&head), type, name) :  \
> > +               NULL)
> > +
> > +/**
> > + * DLB2_LIST_FOR_EACH() - iterate over a list
> > + * @head: list head
> > + * @ptr: pointer to struct containing a struct list
> > + * @name: name of the list field within the containing struct
> > + * @iter: iterator variable
> > + */
> > +#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
> > +       TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
> > +
> > +/**
> > + * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
> > + * an element is removed from the list while processing it.
> > + * @ptr: pointer to struct containing a struct list
> > + * @ptr_tmp: pointer to struct containing a struct list (temporary)
> > + * @head: list head
> > + * @name: name of the list field within the containing struct
> > + * @iter: iterator variable
> > + * @iter_tmp: iterator variable (temporary)
> > + */
> > +#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter,
> saf_itr) \
> > +       TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
> > +
> > +#endif /*  __DLB2_OSDEP_LIST_H */
> > diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
> b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
> > new file mode 100644
> > index 0000000..0a48f7e
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
> > @@ -0,0 +1,31 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef __DLB2_OSDEP_TYPES_H
> > +#define __DLB2_OSDEP_TYPES_H
> > +
> > +#include <linux/types.h>
> > +
> > +#include <inttypes.h>
> > +#include <ctype.h>
> > +#include <stdint.h>
> > +#include <stdbool.h>
> > +#include <string.h>
> > +#include <unistd.h>
> > +#include <errno.h>
> > +
> > +/* Types for user mode PF PMD */
> > +typedef uint8_t         u8;
> > +typedef int8_t          s8;
> > +typedef uint16_t        u16;
> > +typedef int16_t         s16;
> > +typedef uint32_t        u32;
> > +typedef int32_t         s32;
> > +typedef uint64_t        u64;
> > +
> > +#define __iomem
> > +
> > +/* END types for user mode PF PMD */
> > +
> > +#endif /* __DLB2_OSDEP_TYPES_H */
> > diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h
> b/drivers/event/dlb2/pf/base/dlb2_regs.h
> > new file mode 100644
> > index 0000000..43ecad4
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
> > @@ -0,0 +1,2527 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef __DLB2_REGS_H
> > +#define __DLB2_REGS_H
> > +
> > +#include "dlb2_osdep_types.h"
> > +
> > +#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
> > +#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
> > +       (0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
> > +#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
> > +union dlb2_func_pf_vf2pf_mailbox {
> > +       struct {
> > +               u32 msg : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
> > +       (0x1f00 + (vf_id) * 0x10000)
> > +#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
> > +union dlb2_func_pf_vf2pf_mailbox_isr {
> > +       struct {
> > +               u32 vf0_isr : 1;
> > +               u32 vf1_isr : 1;
> > +               u32 vf2_isr : 1;
> > +               u32 vf3_isr : 1;
> > +               u32 vf4_isr : 1;
> > +               u32 vf5_isr : 1;
> > +               u32 vf6_isr : 1;
> > +               u32 vf7_isr : 1;
> > +               u32 vf8_isr : 1;
> > +               u32 vf9_isr : 1;
> > +               u32 vf10_isr : 1;
> > +               u32 vf11_isr : 1;
> > +               u32 vf12_isr : 1;
> > +               u32 vf13_isr : 1;
> > +               u32 vf14_isr : 1;
> > +               u32 vf15_isr : 1;
> > +               u32 rsvd0 : 16;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
> > +       (0x1f04 + (vf_id) * 0x10000)
> > +#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
> > +union dlb2_func_pf_vf2pf_flr_isr {
> > +       struct {
> > +               u32 vf0_isr : 1;
> > +               u32 vf1_isr : 1;
> > +               u32 vf2_isr : 1;
> > +               u32 vf3_isr : 1;
> > +               u32 vf4_isr : 1;
> > +               u32 vf5_isr : 1;
> > +               u32 vf6_isr : 1;
> > +               u32 vf7_isr : 1;
> > +               u32 vf8_isr : 1;
> > +               u32 vf9_isr : 1;
> > +               u32 vf10_isr : 1;
> > +               u32 vf11_isr : 1;
> > +               u32 vf12_isr : 1;
> > +               u32 vf13_isr : 1;
> > +               u32 vf14_isr : 1;
> > +               u32 vf15_isr : 1;
> > +               u32 rsvd0 : 16;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
> > +       (0x1f10 + (vf_id) * 0x10000)
> > +#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
> > +union dlb2_func_pf_vf2pf_isr_pend {
> > +       struct {
> > +               u32 isr_pend : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
> > +#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
> > +       (0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
> > +#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
> > +union dlb2_func_pf_pf2vf_mailbox {
> > +       struct {
> > +               u32 msg : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
> > +       (0x2f00 + (vf_id) * 0x10000)
> > +#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
> > +union dlb2_func_pf_pf2vf_mailbox_isr {
> > +       struct {
> > +               u32 vf0_isr : 1;
> > +               u32 vf1_isr : 1;
> > +               u32 vf2_isr : 1;
> > +               u32 vf3_isr : 1;
> > +               u32 vf4_isr : 1;
> > +               u32 vf5_isr : 1;
> > +               u32 vf6_isr : 1;
> > +               u32 vf7_isr : 1;
> > +               u32 vf8_isr : 1;
> > +               u32 vf9_isr : 1;
> > +               u32 vf10_isr : 1;
> > +               u32 vf11_isr : 1;
> > +               u32 vf12_isr : 1;
> > +               u32 vf13_isr : 1;
> > +               u32 vf14_isr : 1;
> > +               u32 vf15_isr : 1;
> > +               u32 rsvd0 : 16;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
> > +       (0x3000 + (vf_id) * 0x10000)
> > +#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
> > +union dlb2_func_pf_vf_reset_in_progress {
> > +       struct {
> > +               u32 vf0_reset_in_progress : 1;
> > +               u32 vf1_reset_in_progress : 1;
> > +               u32 vf2_reset_in_progress : 1;
> > +               u32 vf3_reset_in_progress : 1;
> > +               u32 vf4_reset_in_progress : 1;
> > +               u32 vf5_reset_in_progress : 1;
> > +               u32 vf6_reset_in_progress : 1;
> > +               u32 vf7_reset_in_progress : 1;
> > +               u32 vf8_reset_in_progress : 1;
> > +               u32 vf9_reset_in_progress : 1;
> > +               u32 vf10_reset_in_progress : 1;
> > +               u32 vf11_reset_in_progress : 1;
> > +               u32 vf12_reset_in_progress : 1;
> > +               u32 vf13_reset_in_progress : 1;
> > +               u32 vf14_reset_in_progress : 1;
> > +               u32 vf15_reset_in_progress : 1;
> > +               u32 rsvd0 : 16;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
> > +       (0x100000c + (x) * 0x10)
> > +#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
> > +union dlb2_msix_mem_vector_ctrl {
> > +       struct {
> > +               u32 vec_mask : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
> > +       (0x20 + (x) * 0x4)
> > +#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
> > +union dlb2_iosf_func_vf_bar_dsbl {
> > +       struct {
> > +               u32 func_vf_bar_dis : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_TOTAL_VAS 0x1000011c
> > +#define DLB2_SYS_TOTAL_VAS_RST 0x20
> > +union dlb2_sys_total_vas {
> > +       struct {
> > +               u32 total_vas : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
> > +#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
> > +union dlb2_sys_total_dir_ports {
> > +       struct {
> > +               u32 total_dir_ports : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
> > +#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
> > +union dlb2_sys_total_ldb_ports {
> > +       struct {
> > +               u32 total_ldb_ports : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
> > +#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
> > +union dlb2_sys_total_dir_qid {
> > +       struct {
> > +               u32 total_dir_qid : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
> > +#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
> > +union dlb2_sys_total_ldb_qid {
> > +       struct {
> > +               u32 total_ldb_qid : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
> > +#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
> > +union dlb2_sys_total_dir_crds {
> > +       struct {
> > +               u32 total_dir_credits : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
> > +#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
> > +union dlb2_sys_total_ldb_crds {
> > +       struct {
> > +               u32 total_ldb_credits : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
> > +#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
> > +union dlb2_sys_alarm_pf_synd2 {
> > +       struct {
> > +               u32 lock_id : 16;
> > +               u32 meas : 1;
> > +               u32 debug : 7;
> > +               u32 cq_pop : 1;
> > +               u32 qe_uhl : 1;
> > +               u32 qe_orsp : 1;
> > +               u32 qe_valid : 1;
> > +               u32 cq_int_rearm : 1;
> > +               u32 dsi_error : 1;
> > +               u32 rsvd0 : 2;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
> > +#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
> > +union dlb2_sys_alarm_pf_synd1 {
> > +       struct {
> > +               u32 dsi : 16;
> > +               u32 qid : 8;
> > +               u32 qtype : 2;
> > +               u32 qpri : 3;
> > +               u32 msg_type : 3;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
> > +#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
> > +union dlb2_sys_alarm_pf_synd0 {
> > +       struct {
> > +               u32 syndrome : 8;
> > +               u32 rtype : 2;
> > +               u32 rsvd0 : 3;
> > +               u32 is_ldb : 1;
> > +               u32 cls : 2;
> > +               u32 aid : 6;
> > +               u32 unit : 4;
> > +               u32 source : 4;
> > +               u32 more : 1;
> > +               u32 valid : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_VF_LDB_VPP_V(x) \
> > +       (0x10000f00 + (x) * 0x1000)
> > +#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
> > +union dlb2_sys_vf_ldb_vpp_v {
> > +       struct {
> > +               u32 vpp_v : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_VF_LDB_VPP2PP(x) \
> > +       (0x10000f04 + (x) * 0x1000)
> > +#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
> > +union dlb2_sys_vf_ldb_vpp2pp {
> > +       struct {
> > +               u32 pp : 6;
> > +               u32 rsvd0 : 26;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_VF_DIR_VPP_V(x) \
> > +       (0x10000f08 + (x) * 0x1000)
> > +#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
> > +union dlb2_sys_vf_dir_vpp_v {
> > +       struct {
> > +               u32 vpp_v : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_VF_DIR_VPP2PP(x) \
> > +       (0x10000f0c + (x) * 0x1000)
> > +#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
> > +union dlb2_sys_vf_dir_vpp2pp {
> > +       struct {
> > +               u32 pp : 6;
> > +               u32 rsvd0 : 26;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_VF_LDB_VQID_V(x) \
> > +       (0x10000f10 + (x) * 0x1000)
> > +#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
> > +union dlb2_sys_vf_ldb_vqid_v {
> > +       struct {
> > +               u32 vqid_v : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_VF_LDB_VQID2QID(x) \
> > +       (0x10000f14 + (x) * 0x1000)
> > +#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
> > +union dlb2_sys_vf_ldb_vqid2qid {
> > +       struct {
> > +               u32 qid : 5;
> > +               u32 rsvd0 : 27;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_QID2VQID(x) \
> > +       (0x10000f18 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_QID2VQID_RST 0x0
> > +union dlb2_sys_ldb_qid2vqid {
> > +       struct {
> > +               u32 vqid : 5;
> > +               u32 rsvd0 : 27;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_VF_DIR_VQID_V(x) \
> > +       (0x10000f1c + (x) * 0x1000)
> > +#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
> > +union dlb2_sys_vf_dir_vqid_v {
> > +       struct {
> > +               u32 vqid_v : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_VF_DIR_VQID2QID(x) \
> > +       (0x10000f20 + (x) * 0x1000)
> > +#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
> > +union dlb2_sys_vf_dir_vqid2qid {
> > +       struct {
> > +               u32 qid : 6;
> > +               u32 rsvd0 : 26;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_VASQID_V(x) \
> > +       (0x10000f24 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_VASQID_V_RST 0x0
> > +union dlb2_sys_ldb_vasqid_v {
> > +       struct {
> > +               u32 vasqid_v : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_VASQID_V(x) \
> > +       (0x10000f28 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_VASQID_V_RST 0x0
> > +union dlb2_sys_dir_vasqid_v {
> > +       struct {
> > +               u32 vasqid_v : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_ALARM_VF_SYND2(x) \
> > +       (0x10000f48 + (x) * 0x1000)
> > +#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
> > +union dlb2_sys_alarm_vf_synd2 {
> > +       struct {
> > +               u32 lock_id : 16;
> > +               u32 debug : 8;
> > +               u32 cq_pop : 1;
> > +               u32 qe_uhl : 1;
> > +               u32 qe_orsp : 1;
> > +               u32 qe_valid : 1;
> > +               u32 isz : 1;
> > +               u32 dsi_error : 1;
> > +               u32 dlbrsvd : 2;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_ALARM_VF_SYND1(x) \
> > +       (0x10000f44 + (x) * 0x1000)
> > +#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
> > +union dlb2_sys_alarm_vf_synd1 {
> > +       struct {
> > +               u32 dsi : 16;
> > +               u32 qid : 8;
> > +               u32 qtype : 2;
> > +               u32 qpri : 3;
> > +               u32 msg_type : 3;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_ALARM_VF_SYND0(x) \
> > +       (0x10000f40 + (x) * 0x1000)
> > +#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
> > +union dlb2_sys_alarm_vf_synd0 {
> > +       struct {
> > +               u32 syndrome : 8;
> > +               u32 rtype : 2;
> > +               u32 vf_synd0_parity : 1;
> > +               u32 vf_synd1_parity : 1;
> > +               u32 vf_synd2_parity : 1;
> > +               u32 is_ldb : 1;
> > +               u32 cls : 2;
> > +               u32 aid : 6;
> > +               u32 unit : 4;
> > +               u32 source : 4;
> > +               u32 more : 1;
> > +               u32 valid : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_QID_CFG_V(x) \
> > +       (0x10000f58 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
> > +union dlb2_sys_ldb_qid_cfg_v {
> > +       struct {
> > +               u32 sn_cfg_v : 1;
> > +               u32 fid_cfg_v : 1;
> > +               u32 rsvd0 : 30;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_QID_ITS(x) \
> > +       (0x10000f54 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_QID_ITS_RST 0x0
> > +union dlb2_sys_ldb_qid_its {
> > +       struct {
> > +               u32 qid_its : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_QID_V(x) \
> > +       (0x10000f50 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_QID_V_RST 0x0
> > +union dlb2_sys_ldb_qid_v {
> > +       struct {
> > +               u32 qid_v : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_QID_ITS(x) \
> > +       (0x10000f64 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_QID_ITS_RST 0x0
> > +union dlb2_sys_dir_qid_its {
> > +       struct {
> > +               u32 qid_its : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_QID_V(x) \
> > +       (0x10000f60 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_QID_V_RST 0x0
> > +union dlb2_sys_dir_qid_v {
> > +       struct {
> > +               u32 qid_v : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
> > +       (0x10000fa8 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
> > +union dlb2_sys_ldb_cq_ai_data {
> > +       struct {
> > +               u32 cq_ai_data : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
> > +       (0x10000fa4 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
> > +union dlb2_sys_ldb_cq_ai_addr {
> > +       struct {
> > +               u32 rsvd1 : 2;
> > +               u32 cq_ai_addr : 18;
> > +               u32 rsvd0 : 12;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_CQ_PASID(x) \
> > +       (0x10000fa0 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
> > +union dlb2_sys_ldb_cq_pasid {
> > +       struct {
> > +               u32 pasid : 20;
> > +               u32 exe_req : 1;
> > +               u32 priv_req : 1;
> > +               u32 fmt2 : 1;
> > +               u32 rsvd0 : 9;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_CQ_AT(x) \
> > +       (0x10000f9c + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_CQ_AT_RST 0x0
> > +union dlb2_sys_ldb_cq_at {
> > +       struct {
> > +               u32 cq_at : 2;
> > +               u32 rsvd0 : 30;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_CQ_ISR(x) \
> > +       (0x10000f98 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
> > +/* CQ Interrupt Modes */
> > +#define DLB2_CQ_ISR_MODE_DIS  0
> > +#define DLB2_CQ_ISR_MODE_MSI  1
> > +#define DLB2_CQ_ISR_MODE_MSIX 2
> > +#define DLB2_CQ_ISR_MODE_ADI  3
> > +union dlb2_sys_ldb_cq_isr {
> > +       struct {
> > +               u32 vector : 6;
> > +               u32 vf : 4;
> > +               u32 en_code : 2;
> > +               u32 rsvd0 : 20;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
> > +       (0x10000f94 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
> > +union dlb2_sys_ldb_cq2vf_pf_ro {
> > +       struct {
> > +               u32 vf : 4;
> > +               u32 is_pf : 1;
> > +               u32 ro : 1;
> > +               u32 rsvd0 : 26;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_PP_V(x) \
> > +       (0x10000f90 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_PP_V_RST 0x0
> > +union dlb2_sys_ldb_pp_v {
> > +       struct {
> > +               u32 pp_v : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_PP2VDEV(x) \
> > +       (0x10000f8c + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
> > +union dlb2_sys_ldb_pp2vdev {
> > +       struct {
> > +               u32 vdev : 4;
> > +               u32 rsvd0 : 28;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_PP2VAS(x) \
> > +       (0x10000f88 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_PP2VAS_RST 0x0
> > +union dlb2_sys_ldb_pp2vas {
> > +       struct {
> > +               u32 vas : 5;
> > +               u32 rsvd0 : 27;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
> > +       (0x10000f84 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
> > +union dlb2_sys_ldb_cq_addr_u {
> > +       struct {
> > +               u32 addr_u : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
> > +       (0x10000f80 + (x) * 0x1000)
> > +#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
> > +union dlb2_sys_ldb_cq_addr_l {
> > +       struct {
> > +               u32 rsvd0 : 6;
> > +               u32 addr_l : 26;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ_FMT(x) \
> > +       (0x10000fec + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
> > +union dlb2_sys_dir_cq_fmt {
> > +       struct {
> > +               u32 keep_pf_ppid : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
> > +       (0x10000fe8 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
> > +union dlb2_sys_dir_cq_ai_data {
> > +       struct {
> > +               u32 cq_ai_data : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
> > +       (0x10000fe4 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
> > +union dlb2_sys_dir_cq_ai_addr {
> > +       struct {
> > +               u32 rsvd1 : 2;
> > +               u32 cq_ai_addr : 18;
> > +               u32 rsvd0 : 12;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ_PASID(x) \
> > +       (0x10000fe0 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
> > +union dlb2_sys_dir_cq_pasid {
> > +       struct {
> > +               u32 pasid : 20;
> > +               u32 exe_req : 1;
> > +               u32 priv_req : 1;
> > +               u32 fmt2 : 1;
> > +               u32 rsvd0 : 9;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ_AT(x) \
> > +       (0x10000fdc + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_CQ_AT_RST 0x0
> > +union dlb2_sys_dir_cq_at {
> > +       struct {
> > +               u32 cq_at : 2;
> > +               u32 rsvd0 : 30;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ_ISR(x) \
> > +       (0x10000fd8 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
> > +union dlb2_sys_dir_cq_isr {
> > +       struct {
> > +               u32 vector : 6;
> > +               u32 vf : 4;
> > +               u32 en_code : 2;
> > +               u32 rsvd0 : 20;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
> > +       (0x10000fd4 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
> > +union dlb2_sys_dir_cq2vf_pf_ro {
> > +       struct {
> > +               u32 vf : 4;
> > +               u32 is_pf : 1;
> > +               u32 ro : 1;
> > +               u32 rsvd0 : 26;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_PP_V(x) \
> > +       (0x10000fd0 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_PP_V_RST 0x0
> > +union dlb2_sys_dir_pp_v {
> > +       struct {
> > +               u32 pp_v : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_PP2VDEV(x) \
> > +       (0x10000fcc + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
> > +union dlb2_sys_dir_pp2vdev {
> > +       struct {
> > +               u32 vdev : 4;
> > +               u32 rsvd0 : 28;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_PP2VAS(x) \
> > +       (0x10000fc8 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_PP2VAS_RST 0x0
> > +union dlb2_sys_dir_pp2vas {
> > +       struct {
> > +               u32 vas : 5;
> > +               u32 rsvd0 : 27;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
> > +       (0x10000fc4 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
> > +union dlb2_sys_dir_cq_addr_u {
> > +       struct {
> > +               u32 addr_u : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
> > +       (0x10000fc0 + (x) * 0x1000)
> > +#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
> > +union dlb2_sys_dir_cq_addr_l {
> > +       struct {
> > +               u32 rsvd0 : 6;
> > +               u32 addr_l : 26;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
> > +#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
> > +union dlb2_sys_ingress_alarm_enbl {
> > +       struct {
> > +               u32 illegal_hcw : 1;
> > +               u32 illegal_pp : 1;
> > +               u32 illegal_pasid : 1;
> > +               u32 illegal_qid : 1;
> > +               u32 disabled_qid : 1;
> > +               u32 illegal_ldb_qid_cfg : 1;
> > +               u32 rsvd0 : 26;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_MSIX_ACK 0x10000400
> > +#define DLB2_SYS_MSIX_ACK_RST 0x0
> > +union dlb2_sys_msix_ack {
> > +       struct {
> > +               u32 msix_0_ack : 1;
> > +               u32 msix_1_ack : 1;
> > +               u32 rsvd0 : 30;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
> > +#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
> > +union dlb2_sys_msix_passthru {
> > +       struct {
> > +               u32 msix_0_passthru : 1;
> > +               u32 msix_1_passthru : 1;
> > +               u32 rsvd0 : 30;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_MSIX_MODE 0x10000408
> > +#define DLB2_SYS_MSIX_MODE_RST 0x0
> > +/* MSI-X Modes */
> > +#define DLB2_MSIX_MODE_PACKED     0
> > +#define DLB2_MSIX_MODE_COMPRESSED 1
> > +union dlb2_sys_msix_mode {
> > +       struct {
> > +               u32 mode : 1;
> > +               u32 poll_mode : 1;
> > +               u32 poll_mask : 1;
> > +               u32 poll_lock : 1;
> > +               u32 rsvd0 : 28;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
> > +#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
> > +union dlb2_sys_dir_cq_31_0_occ_int_sts {
> > +       struct {
> > +               u32 cq_0_occ_int : 1;
> > +               u32 cq_1_occ_int : 1;
> > +               u32 cq_2_occ_int : 1;
> > +               u32 cq_3_occ_int : 1;
> > +               u32 cq_4_occ_int : 1;
> > +               u32 cq_5_occ_int : 1;
> > +               u32 cq_6_occ_int : 1;
> > +               u32 cq_7_occ_int : 1;
> > +               u32 cq_8_occ_int : 1;
> > +               u32 cq_9_occ_int : 1;
> > +               u32 cq_10_occ_int : 1;
> > +               u32 cq_11_occ_int : 1;
> > +               u32 cq_12_occ_int : 1;
> > +               u32 cq_13_occ_int : 1;
> > +               u32 cq_14_occ_int : 1;
> > +               u32 cq_15_occ_int : 1;
> > +               u32 cq_16_occ_int : 1;
> > +               u32 cq_17_occ_int : 1;
> > +               u32 cq_18_occ_int : 1;
> > +               u32 cq_19_occ_int : 1;
> > +               u32 cq_20_occ_int : 1;
> > +               u32 cq_21_occ_int : 1;
> > +               u32 cq_22_occ_int : 1;
> > +               u32 cq_23_occ_int : 1;
> > +               u32 cq_24_occ_int : 1;
> > +               u32 cq_25_occ_int : 1;
> > +               u32 cq_26_occ_int : 1;
> > +               u32 cq_27_occ_int : 1;
> > +               u32 cq_28_occ_int : 1;
> > +               u32 cq_29_occ_int : 1;
> > +               u32 cq_30_occ_int : 1;
> > +               u32 cq_31_occ_int : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
> > +#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
> > +union dlb2_sys_dir_cq_63_32_occ_int_sts {
> > +       struct {
> > +               u32 cq_32_occ_int : 1;
> > +               u32 cq_33_occ_int : 1;
> > +               u32 cq_34_occ_int : 1;
> > +               u32 cq_35_occ_int : 1;
> > +               u32 cq_36_occ_int : 1;
> > +               u32 cq_37_occ_int : 1;
> > +               u32 cq_38_occ_int : 1;
> > +               u32 cq_39_occ_int : 1;
> > +               u32 cq_40_occ_int : 1;
> > +               u32 cq_41_occ_int : 1;
> > +               u32 cq_42_occ_int : 1;
> > +               u32 cq_43_occ_int : 1;
> > +               u32 cq_44_occ_int : 1;
> > +               u32 cq_45_occ_int : 1;
> > +               u32 cq_46_occ_int : 1;
> > +               u32 cq_47_occ_int : 1;
> > +               u32 cq_48_occ_int : 1;
> > +               u32 cq_49_occ_int : 1;
> > +               u32 cq_50_occ_int : 1;
> > +               u32 cq_51_occ_int : 1;
> > +               u32 cq_52_occ_int : 1;
> > +               u32 cq_53_occ_int : 1;
> > +               u32 cq_54_occ_int : 1;
> > +               u32 cq_55_occ_int : 1;
> > +               u32 cq_56_occ_int : 1;
> > +               u32 cq_57_occ_int : 1;
> > +               u32 cq_58_occ_int : 1;
> > +               u32 cq_59_occ_int : 1;
> > +               u32 cq_60_occ_int : 1;
> > +               u32 cq_61_occ_int : 1;
> > +               u32 cq_62_occ_int : 1;
> > +               u32 cq_63_occ_int : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
> > +#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
> > +union dlb2_sys_ldb_cq_31_0_occ_int_sts {
> > +       struct {
> > +               u32 cq_0_occ_int : 1;
> > +               u32 cq_1_occ_int : 1;
> > +               u32 cq_2_occ_int : 1;
> > +               u32 cq_3_occ_int : 1;
> > +               u32 cq_4_occ_int : 1;
> > +               u32 cq_5_occ_int : 1;
> > +               u32 cq_6_occ_int : 1;
> > +               u32 cq_7_occ_int : 1;
> > +               u32 cq_8_occ_int : 1;
> > +               u32 cq_9_occ_int : 1;
> > +               u32 cq_10_occ_int : 1;
> > +               u32 cq_11_occ_int : 1;
> > +               u32 cq_12_occ_int : 1;
> > +               u32 cq_13_occ_int : 1;
> > +               u32 cq_14_occ_int : 1;
> > +               u32 cq_15_occ_int : 1;
> > +               u32 cq_16_occ_int : 1;
> > +               u32 cq_17_occ_int : 1;
> > +               u32 cq_18_occ_int : 1;
> > +               u32 cq_19_occ_int : 1;
> > +               u32 cq_20_occ_int : 1;
> > +               u32 cq_21_occ_int : 1;
> > +               u32 cq_22_occ_int : 1;
> > +               u32 cq_23_occ_int : 1;
> > +               u32 cq_24_occ_int : 1;
> > +               u32 cq_25_occ_int : 1;
> > +               u32 cq_26_occ_int : 1;
> > +               u32 cq_27_occ_int : 1;
> > +               u32 cq_28_occ_int : 1;
> > +               u32 cq_29_occ_int : 1;
> > +               u32 cq_30_occ_int : 1;
> > +               u32 cq_31_occ_int : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
> > +#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
> > +union dlb2_sys_ldb_cq_63_32_occ_int_sts {
> > +       struct {
> > +               u32 cq_32_occ_int : 1;
> > +               u32 cq_33_occ_int : 1;
> > +               u32 cq_34_occ_int : 1;
> > +               u32 cq_35_occ_int : 1;
> > +               u32 cq_36_occ_int : 1;
> > +               u32 cq_37_occ_int : 1;
> > +               u32 cq_38_occ_int : 1;
> > +               u32 cq_39_occ_int : 1;
> > +               u32 cq_40_occ_int : 1;
> > +               u32 cq_41_occ_int : 1;
> > +               u32 cq_42_occ_int : 1;
> > +               u32 cq_43_occ_int : 1;
> > +               u32 cq_44_occ_int : 1;
> > +               u32 cq_45_occ_int : 1;
> > +               u32 cq_46_occ_int : 1;
> > +               u32 cq_47_occ_int : 1;
> > +               u32 cq_48_occ_int : 1;
> > +               u32 cq_49_occ_int : 1;
> > +               u32 cq_50_occ_int : 1;
> > +               u32 cq_51_occ_int : 1;
> > +               u32 cq_52_occ_int : 1;
> > +               u32 cq_53_occ_int : 1;
> > +               u32 cq_54_occ_int : 1;
> > +               u32 cq_55_occ_int : 1;
> > +               u32 cq_56_occ_int : 1;
> > +               u32 cq_57_occ_int : 1;
> > +               u32 cq_58_occ_int : 1;
> > +               u32 cq_59_occ_int : 1;
> > +               u32 cq_60_occ_int : 1;
> > +               u32 cq_61_occ_int : 1;
> > +               u32 cq_62_occ_int : 1;
> > +               u32 cq_63_occ_int : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
> > +#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
> > +union dlb2_sys_dir_cq_opt_clr {
> > +       struct {
> > +               u32 cq : 6;
> > +               u32 rsvd0 : 26;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
> > +#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
> > +union dlb2_sys_alarm_hw_synd {
> > +       struct {
> > +               u32 syndrome : 8;
> > +               u32 rtype : 2;
> > +               u32 alarm : 1;
> > +               u32 cwd : 1;
> > +               u32 vf_pf_mb : 1;
> > +               u32 rsvd0 : 1;
> > +               u32 cls : 2;
> > +               u32 aid : 6;
> > +               u32 unit : 4;
> > +               u32 source : 4;
> > +               u32 more : 1;
> > +               u32 valid : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
> > +       (0x20000000 + (x) * 0x1000)
> > +#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
> > +union dlb2_aqed_pipe_qid_fid_lim {
> > +       struct {
> > +               u32 qid_fid_limit : 13;
> > +               u32 rsvd0 : 19;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
> > +       (0x20080000 + (x) * 0x1000)
> > +#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
> > +union dlb2_aqed_pipe_qid_hid_width {
> > +       struct {
> > +               u32 compress_code : 3;
> > +               u32 rsvd0 : 29;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
> > +#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST
> 0xfefcfaf8
> > +union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
> > +       struct {
> > +               u32 pri0 : 8;
> > +               u32 pri1 : 8;
> > +               u32 pri2 : 8;
> > +               u32 pri3 : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_ATM_QID2CQIDIX_00(x) \
> > +       (0x30080000 + (x) * 0x1000)
> > +#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
> > +#define DLB2_ATM_QID2CQIDIX(x, y) \
> > +       (DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
> > +#define DLB2_ATM_QID2CQIDIX_NUM 16
> > +union dlb2_atm_qid2cqidix_00 {
> > +       struct {
> > +               u32 cq_p0 : 8;
> > +               u32 cq_p1 : 8;
> > +               u32 cq_p2 : 8;
> > +               u32 cq_p3 : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
> > +#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
> > +union dlb2_atm_cfg_arb_weights_rdy_bin {
> > +       struct {
> > +               u32 bin0 : 8;
> > +               u32 bin1 : 8;
> > +               u32 bin2 : 8;
> > +               u32 bin3 : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
> > +#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
> > +union dlb2_atm_cfg_arb_weights_sched_bin {
> > +       struct {
> > +               u32 bin0 : 8;
> > +               u32 bin1 : 8;
> > +               u32 bin2 : 8;
> > +               u32 bin3 : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
> > +       (0x40000000 + (x) * 0x1000)
> > +#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
> > +union dlb2_chp_cfg_dir_vas_crd {
> > +       struct {
> > +               u32 count : 14;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
> > +       (0x40080000 + (x) * 0x1000)
> > +#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
> > +union dlb2_chp_cfg_ldb_vas_crd {
> > +       struct {
> > +               u32 count : 15;
> > +               u32 rsvd0 : 17;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_ORD_QID_SN(x) \
> > +       (0x40100000 + (x) * 0x1000)
> > +#define DLB2_CHP_ORD_QID_SN_RST 0x0
> > +union dlb2_chp_ord_qid_sn {
> > +       struct {
> > +               u32 sn : 10;
> > +               u32 rsvd0 : 22;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_ORD_QID_SN_MAP(x) \
> > +       (0x40180000 + (x) * 0x1000)
> > +#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
> > +union dlb2_chp_ord_qid_sn_map {
> > +       struct {
> > +               u32 mode : 3;
> > +               u32 slot : 4;
> > +               u32 rsvz0 : 1;
> > +               u32 grp : 1;
> > +               u32 rsvz1 : 1;
> > +               u32 rsvd0 : 22;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_SN_CHK_ENBL(x) \
> > +       (0x40200000 + (x) * 0x1000)
> > +#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
> > +union dlb2_chp_sn_chk_enbl {
> > +       struct {
> > +               u32 en : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_DIR_CQ_DEPTH(x) \
> > +       (0x40280000 + (x) * 0x1000)
> > +#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
> > +union dlb2_chp_dir_cq_depth {
> > +       struct {
> > +               u32 depth : 13;
> > +               u32 rsvd0 : 19;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
> > +       (0x40300000 + (x) * 0x1000)
> > +#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
> > +union dlb2_chp_dir_cq_int_depth_thrsh {
> > +       struct {
> > +               u32 depth_threshold : 13;
> > +               u32 rsvd0 : 19;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
> > +       (0x40380000 + (x) * 0x1000)
> > +#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
> > +union dlb2_chp_dir_cq_int_enb {
> > +       struct {
> > +               u32 en_tim : 1;
> > +               u32 en_depth : 1;
> > +               u32 rsvd0 : 30;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
> > +       (0x40480000 + (x) * 0x1000)
> > +#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
> > +union dlb2_chp_dir_cq_tmr_thrsh {
> > +       struct {
> > +               u32 thrsh_0 : 1;
> > +               u32 thrsh_13_1 : 13;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
> > +       (0x40500000 + (x) * 0x1000)
> > +#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
> > +union dlb2_chp_dir_cq_tkn_depth_sel {
> > +       struct {
> > +               u32 token_depth_select : 4;
> > +               u32 rsvd0 : 28;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
> > +       (0x40580000 + (x) * 0x1000)
> > +#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
> > +union dlb2_chp_dir_cq_wd_enb {
> > +       struct {
> > +               u32 wd_enable : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_DIR_CQ_WPTR(x) \
> > +       (0x40600000 + (x) * 0x1000)
> > +#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
> > +union dlb2_chp_dir_cq_wptr {
> > +       struct {
> > +               u32 write_pointer : 13;
> > +               u32 rsvd0 : 19;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_DIR_CQ2VAS(x) \
> > +       (0x40680000 + (x) * 0x1000)
> > +#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
> > +union dlb2_chp_dir_cq2vas {
> > +       struct {
> > +               u32 cq2vas : 5;
> > +               u32 rsvd0 : 27;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_HIST_LIST_BASE(x) \
> > +       (0x40700000 + (x) * 0x1000)
> > +#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
> > +union dlb2_chp_hist_list_base {
> > +       struct {
> > +               u32 base : 13;
> > +               u32 rsvd0 : 19;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_HIST_LIST_LIM(x) \
> > +       (0x40780000 + (x) * 0x1000)
> > +#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
> > +union dlb2_chp_hist_list_lim {
> > +       struct {
> > +               u32 limit : 13;
> > +               u32 rsvd0 : 19;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
> > +       (0x40800000 + (x) * 0x1000)
> > +#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
> > +union dlb2_chp_hist_list_pop_ptr {
> > +       struct {
> > +               u32 pop_ptr : 13;
> > +               u32 generation : 1;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
> > +       (0x40880000 + (x) * 0x1000)
> > +#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
> > +union dlb2_chp_hist_list_push_ptr {
> > +       struct {
> > +               u32 push_ptr : 13;
> > +               u32 generation : 1;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_LDB_CQ_DEPTH(x) \
> > +       (0x40900000 + (x) * 0x1000)
> > +#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
> > +union dlb2_chp_ldb_cq_depth {
> > +       struct {
> > +               u32 depth : 11;
> > +               u32 rsvd0 : 21;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
> > +       (0x40980000 + (x) * 0x1000)
> > +#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
> > +union dlb2_chp_ldb_cq_int_depth_thrsh {
> > +       struct {
> > +               u32 depth_threshold : 11;
> > +               u32 rsvd0 : 21;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
> > +       (0x40a00000 + (x) * 0x1000)
> > +#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
> > +union dlb2_chp_ldb_cq_int_enb {
> > +       struct {
> > +               u32 en_tim : 1;
> > +               u32 en_depth : 1;
> > +               u32 rsvd0 : 30;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
> > +       (0x40b00000 + (x) * 0x1000)
> > +#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
> > +union dlb2_chp_ldb_cq_tmr_thrsh {
> > +       struct {
> > +               u32 thrsh_0 : 1;
> > +               u32 thrsh_13_1 : 13;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
> > +       (0x40b80000 + (x) * 0x1000)
> > +#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
> > +union dlb2_chp_ldb_cq_tkn_depth_sel {
> > +       struct {
> > +               u32 token_depth_select : 4;
> > +               u32 rsvd0 : 28;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
> > +       (0x40c00000 + (x) * 0x1000)
> > +#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
> > +union dlb2_chp_ldb_cq_wd_enb {
> > +       struct {
> > +               u32 wd_enable : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_LDB_CQ_WPTR(x) \
> > +       (0x40c80000 + (x) * 0x1000)
> > +#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
> > +union dlb2_chp_ldb_cq_wptr {
> > +       struct {
> > +               u32 write_pointer : 11;
> > +               u32 rsvd0 : 21;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_LDB_CQ2VAS(x) \
> > +       (0x40d00000 + (x) * 0x1000)
> > +#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
> > +union dlb2_chp_ldb_cq2vas {
> > +       struct {
> > +               u32 cq2vas : 5;
> > +               u32 rsvd0 : 27;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
> > +#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
> > +union dlb2_chp_cfg_chp_csr_ctrl {
> > +       struct {
> > +               u32 int_cor_alarm_dis : 1;
> > +               u32 int_cor_synd_dis : 1;
> > +               u32 int_uncr_alarm_dis : 1;
> > +               u32 int_unc_synd_dis : 1;
> > +               u32 int_inf0_alarm_dis : 1;
> > +               u32 int_inf0_synd_dis : 1;
> > +               u32 int_inf1_alarm_dis : 1;
> > +               u32 int_inf1_synd_dis : 1;
> > +               u32 int_inf2_alarm_dis : 1;
> > +               u32 int_inf2_synd_dis : 1;
> > +               u32 int_inf3_alarm_dis : 1;
> > +               u32 int_inf3_synd_dis : 1;
> > +               u32 int_inf4_alarm_dis : 1;
> > +               u32 int_inf4_synd_dis : 1;
> > +               u32 int_inf5_alarm_dis : 1;
> > +               u32 int_inf5_synd_dis : 1;
> > +               u32 dlb_cor_alarm_enable : 1;
> > +               u32 cfg_64bytes_qe_ldb_cq_mode : 1;
> > +               u32 cfg_64bytes_qe_dir_cq_mode : 1;
> > +               u32 pad_write_ldb : 1;
> > +               u32 pad_write_dir : 1;
> > +               u32 pad_first_write_ldb : 1;
> > +               u32 pad_first_write_dir : 1;
> > +               u32 rsvz0 : 9;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
> > +#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
> > +union dlb2_chp_dir_cq_intr_armed0 {
> > +       struct {
> > +               u32 armed : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
> > +#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
> > +union dlb2_chp_dir_cq_intr_armed1 {
> > +       struct {
> > +               u32 armed : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
> > +#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
> > +union dlb2_chp_cfg_dir_cq_timer_ctl {
> > +       struct {
> > +               u32 sample_interval : 8;
> > +               u32 enb : 1;
> > +               u32 rsvz0 : 23;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
> > +#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
> > +union dlb2_chp_cfg_dir_wdto_0 {
> > +       struct {
> > +               u32 wdto : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
> > +#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
> > +union dlb2_chp_cfg_dir_wdto_1 {
> > +       struct {
> > +               u32 wdto : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
> > +#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
> > +union dlb2_chp_cfg_dir_wd_disable0 {
> > +       struct {
> > +               u32 wd_disable : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
> > +#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
> > +union dlb2_chp_cfg_dir_wd_disable1 {
> > +       struct {
> > +               u32 wd_disable : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
> > +#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
> > +union dlb2_chp_cfg_dir_wd_enb_interval {
> > +       struct {
> > +               u32 sample_interval : 28;
> > +               u32 enb : 1;
> > +               u32 rsvz0 : 3;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
> > +#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
> > +union dlb2_chp_cfg_dir_wd_threshold {
> > +       struct {
> > +               u32 wd_threshold : 8;
> > +               u32 rsvz0 : 24;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
> > +#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
> > +union dlb2_chp_ldb_cq_intr_armed0 {
> > +       struct {
> > +               u32 armed : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
> > +#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
> > +union dlb2_chp_ldb_cq_intr_armed1 {
> > +       struct {
> > +               u32 armed : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
> > +#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
> > +union dlb2_chp_cfg_ldb_cq_timer_ctl {
> > +       struct {
> > +               u32 sample_interval : 8;
> > +               u32 enb : 1;
> > +               u32 rsvz0 : 23;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
> > +#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
> > +union dlb2_chp_cfg_ldb_wdto_0 {
> > +       struct {
> > +               u32 wdto : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
> > +#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
> > +union dlb2_chp_cfg_ldb_wdto_1 {
> > +       struct {
> > +               u32 wdto : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
> > +#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
> > +union dlb2_chp_cfg_ldb_wd_disable0 {
> > +       struct {
> > +               u32 wd_disable : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
> > +#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
> > +union dlb2_chp_cfg_ldb_wd_disable1 {
> > +       struct {
> > +               u32 wd_disable : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
> > +#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
> > +union dlb2_chp_cfg_ldb_wd_enb_interval {
> > +       struct {
> > +               u32 sample_interval : 28;
> > +               u32 enb : 1;
> > +               u32 rsvz0 : 3;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
> > +#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
> > +union dlb2_chp_cfg_ldb_wd_threshold {
> > +       struct {
> > +               u32 wd_threshold : 8;
> > +               u32 rsvz0 : 24;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
> > +#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
> > +union dlb2_chp_ctrl_diag_02 {
> > +       struct {
> > +               u32 egress_credit_status_empty : 1;
> > +               u32 egress_credit_status_afull : 1;
> > +               u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
> > +               u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
> > +               u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
> > +               u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
> > +               u32 chp_lsp_tok_pipe_credit_status_empty : 1;
> > +               u32 chp_lsp_tok_pipe_credit_status_afull : 1;
> > +               u32 chp_rop_pipe_credit_status_empty : 1;
> > +               u32 chp_rop_pipe_credit_status_afull : 1;
> > +               u32 qed_to_cq_pipe_credit_status_empty : 1;
> > +               u32 qed_to_cq_pipe_credit_status_afull : 1;
> > +               u32 egress_lsp_token_credit_status_empty : 1;
> > +               u32 egress_lsp_token_credit_status_afull : 1;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
> > +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
> > +union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
> > +       struct {
> > +               u32 pri0 : 8;
> > +               u32 pri1 : 8;
> > +               u32 pri2 : 8;
> > +               u32 pri3 : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
> > +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
> > +union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
> > +       struct {
> > +               u32 rsvz0 : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
> > +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
> > +union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
> > +       struct {
> > +               u32 pri0 : 8;
> > +               u32 pri1 : 8;
> > +               u32 pri2 : 8;
> > +               u32 pri3 : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
> > +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
> > +union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
> > +       struct {
> > +               u32 rsvz0 : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_DP_DIR_CSR_CTRL 0x54000010
> > +#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
> > +union dlb2_dp_dir_csr_ctrl {
> > +       struct {
> > +               u32 int_cor_alarm_dis : 1;
> > +               u32 int_cor_synd_dis : 1;
> > +               u32 int_uncr_alarm_dis : 1;
> > +               u32 int_unc_synd_dis : 1;
> > +               u32 int_inf0_alarm_dis : 1;
> > +               u32 int_inf0_synd_dis : 1;
> > +               u32 int_inf1_alarm_dis : 1;
> > +               u32 int_inf1_synd_dis : 1;
> > +               u32 int_inf2_alarm_dis : 1;
> > +               u32 int_inf2_synd_dis : 1;
> > +               u32 int_inf3_alarm_dis : 1;
> > +               u32 int_inf3_synd_dis : 1;
> > +               u32 int_inf4_alarm_dis : 1;
> > +               u32 int_inf4_synd_dis : 1;
> > +               u32 int_inf5_alarm_dis : 1;
> > +               u32 int_inf5_synd_dis : 1;
> > +               u32 rsvz0 : 16;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST
> 0xfefcfaf8
> > +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
> > +       struct {
> > +               u32 pri0 : 8;
> > +               u32 pri1 : 8;
> > +               u32 pri2 : 8;
> > +               u32 pri3 : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
> > +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
> > +       struct {
> > +               u32 rsvz0 : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST
> 0xfefcfaf8
> > +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
> > +       struct {
> > +               u32 pri0 : 8;
> > +               u32 pri1 : 8;
> > +               u32 pri2 : 8;
> > +               u32 pri3 : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
> > +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
> > +       struct {
> > +               u32 rsvz0 : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0
> 0x84000010
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST
> 0xfefcfaf8
> > +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
> > +       struct {
> > +               u32 pri0 : 8;
> > +               u32 pri1 : 8;
> > +               u32 pri2 : 8;
> > +               u32 pri3 : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1
> 0x84000014
> > +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
> > +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
> > +       struct {
> > +               u32 rsvz0 : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
> > +       (0x96000000 + (x) * 0x4)
> > +#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
> > +union dlb2_ro_pipe_grp_0_slt_shft {
> > +       struct {
> > +               u32 change : 10;
> > +               u32 rsvd0 : 22;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
> > +       (0x96010000 + (x) * 0x4)
> > +#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
> > +union dlb2_ro_pipe_grp_1_slt_shft {
> > +       struct {
> > +               u32 change : 10;
> > +               u32 rsvd0 : 22;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
> > +#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
> > +union dlb2_ro_pipe_grp_sn_mode {
> > +       struct {
> > +               u32 sn_mode_0 : 3;
> > +               u32 rszv0 : 5;
> > +               u32 sn_mode_1 : 3;
> > +               u32 rszv1 : 21;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
> > +#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
> > +union dlb2_ro_pipe_cfg_ctrl_general_0 {
> > +       struct {
> > +               u32 unit_single_step_mode : 1;
> > +               u32 rr_en : 1;
> > +               u32 rszv0 : 30;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ2PRIOV(x) \
> > +       (0xa0000000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ2PRIOV_RST 0x0
> > +union dlb2_lsp_cq2priov {
> > +       struct {
> > +               u32 prio : 24;
> > +               u32 v : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ2QID0(x) \
> > +       (0xa0080000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ2QID0_RST 0x0
> > +union dlb2_lsp_cq2qid0 {
> > +       struct {
> > +               u32 qid_p0 : 7;
> > +               u32 rsvd3 : 1;
> > +               u32 qid_p1 : 7;
> > +               u32 rsvd2 : 1;
> > +               u32 qid_p2 : 7;
> > +               u32 rsvd1 : 1;
> > +               u32 qid_p3 : 7;
> > +               u32 rsvd0 : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ2QID1(x) \
> > +       (0xa0100000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ2QID1_RST 0x0
> > +union dlb2_lsp_cq2qid1 {
> > +       struct {
> > +               u32 qid_p4 : 7;
> > +               u32 rsvd3 : 1;
> > +               u32 qid_p5 : 7;
> > +               u32 rsvd2 : 1;
> > +               u32 qid_p6 : 7;
> > +               u32 rsvd1 : 1;
> > +               u32 qid_p7 : 7;
> > +               u32 rsvd0 : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_DIR_DSBL(x) \
> > +       (0xa0180000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
> > +union dlb2_lsp_cq_dir_dsbl {
> > +       struct {
> > +               u32 disabled : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
> > +       (0xa0200000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
> > +union dlb2_lsp_cq_dir_tkn_cnt {
> > +       struct {
> > +               u32 count : 13;
> > +               u32 rsvd0 : 19;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
> > +       (0xa0280000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
> > +union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
> > +       struct {
> > +               u32 token_depth_select : 4;
> > +               u32 disable_wb_opt : 1;
> > +               u32 ignore_depth : 1;
> > +               u32 rsvd0 : 26;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
> > +       (0xa0300000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
> > +union dlb2_lsp_cq_dir_tot_sch_cntl {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
> > +       (0xa0380000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
> > +union dlb2_lsp_cq_dir_tot_sch_cnth {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_LDB_DSBL(x) \
> > +       (0xa0400000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
> > +union dlb2_lsp_cq_ldb_dsbl {
> > +       struct {
> > +               u32 disabled : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
> > +       (0xa0480000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
> > +union dlb2_lsp_cq_ldb_infl_cnt {
> > +       struct {
> > +               u32 count : 12;
> > +               u32 rsvd0 : 20;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
> > +       (0xa0500000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
> > +union dlb2_lsp_cq_ldb_infl_lim {
> > +       struct {
> > +               u32 limit : 12;
> > +               u32 rsvd0 : 20;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
> > +       (0xa0580000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
> > +union dlb2_lsp_cq_ldb_tkn_cnt {
> > +       struct {
> > +               u32 token_count : 11;
> > +               u32 rsvd0 : 21;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
> > +       (0xa0600000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
> > +union dlb2_lsp_cq_ldb_tkn_depth_sel {
> > +       struct {
> > +               u32 token_depth_select : 4;
> > +               u32 ignore_depth : 1;
> > +               u32 rsvd0 : 27;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
> > +       (0xa0680000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
> > +union dlb2_lsp_cq_ldb_tot_sch_cntl {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
> > +       (0xa0700000 + (x) * 0x1000)
> > +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
> > +union dlb2_lsp_cq_ldb_tot_sch_cnth {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
> > +       (0xa0780000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
> > +union dlb2_lsp_qid_dir_max_depth {
> > +       struct {
> > +               u32 depth : 13;
> > +               u32 rsvd0 : 19;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
> > +       (0xa0800000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
> > +union dlb2_lsp_qid_dir_tot_enq_cntl {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
> > +       (0xa0880000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
> > +union dlb2_lsp_qid_dir_tot_enq_cnth {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
> > +       (0xa0900000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
> > +union dlb2_lsp_qid_dir_enqueue_cnt {
> > +       struct {
> > +               u32 count : 13;
> > +               u32 rsvd0 : 19;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
> > +       (0xa0980000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
> > +union dlb2_lsp_qid_dir_depth_thrsh {
> > +       struct {
> > +               u32 thresh : 13;
> > +               u32 rsvd0 : 19;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
> > +       (0xa0a00000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
> > +union dlb2_lsp_qid_aqed_active_cnt {
> > +       struct {
> > +               u32 count : 12;
> > +               u32 rsvd0 : 20;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
> > +       (0xa0a80000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
> > +union dlb2_lsp_qid_aqed_active_lim {
> > +       struct {
> > +               u32 limit : 12;
> > +               u32 rsvd0 : 20;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
> > +       (0xa0b00000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
> > +union dlb2_lsp_qid_atm_tot_enq_cntl {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
> > +       (0xa0b80000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
> > +union dlb2_lsp_qid_atm_tot_enq_cnth {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
> > +       (0xa0c00000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
> > +union dlb2_lsp_qid_atq_enqueue_cnt {
> > +       struct {
> > +               u32 count : 14;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
> > +       (0xa0c80000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
> > +union dlb2_lsp_qid_ldb_enqueue_cnt {
> > +       struct {
> > +               u32 count : 14;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
> > +       (0xa0d00000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
> > +union dlb2_lsp_qid_ldb_infl_cnt {
> > +       struct {
> > +               u32 count : 12;
> > +               u32 rsvd0 : 20;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
> > +       (0xa0d80000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
> > +union dlb2_lsp_qid_ldb_infl_lim {
> > +       struct {
> > +               u32 limit : 12;
> > +               u32 rsvd0 : 20;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID2CQIDIX_00(x) \
> > +       (0xa0e00000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
> > +#define DLB2_LSP_QID2CQIDIX(x, y) \
> > +       (DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
> > +#define DLB2_LSP_QID2CQIDIX_NUM 16
> > +union dlb2_lsp_qid2cqidix_00 {
> > +       struct {
> > +               u32 cq_p0 : 8;
> > +               u32 cq_p1 : 8;
> > +               u32 cq_p2 : 8;
> > +               u32 cq_p3 : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID2CQIDIX2_00(x) \
> > +       (0xa1600000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
> > +#define DLB2_LSP_QID2CQIDIX2(x, y) \
> > +       (DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
> > +#define DLB2_LSP_QID2CQIDIX2_NUM 16
> > +union dlb2_lsp_qid2cqidix2_00 {
> > +       struct {
> > +               u32 cq_p0 : 8;
> > +               u32 cq_p1 : 8;
> > +               u32 cq_p2 : 8;
> > +               u32 cq_p3 : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
> > +       (0xa1e00000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
> > +union dlb2_lsp_qid_ldb_replay_cnt {
> > +       struct {
> > +               u32 count : 14;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
> > +       (0xa1f00000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
> > +union dlb2_lsp_qid_naldb_max_depth {
> > +       struct {
> > +               u32 depth : 14;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
> > +       (0xa1f80000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
> > +union dlb2_lsp_qid_naldb_tot_enq_cntl {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
> > +       (0xa2000000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
> > +union dlb2_lsp_qid_naldb_tot_enq_cnth {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
> > +       (0xa2080000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
> > +union dlb2_lsp_qid_atm_depth_thrsh {
> > +       struct {
> > +               u32 thresh : 14;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
> > +       (0xa2100000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
> > +union dlb2_lsp_qid_naldb_depth_thrsh {
> > +       struct {
> > +               u32 thresh : 14;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_QID_ATM_ACTIVE(x) \
> > +       (0xa2180000 + (x) * 0x1000)
> > +#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
> > +union dlb2_lsp_qid_atm_active {
> > +       struct {
> > +               u32 count : 14;
> > +               u32 rsvd0 : 18;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
> > +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
> > +union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
> > +       struct {
> > +               u32 pri0_weight : 8;
> > +               u32 pri1_weight : 8;
> > +               u32 pri2_weight : 8;
> > +               u32 pri3_weight : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
> > +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
> > +union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
> > +       struct {
> > +               u32 rsvz0 : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
> > +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
> > +union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
> > +       struct {
> > +               u32 pri0_weight : 8;
> > +               u32 pri1_weight : 8;
> > +               u32 pri2_weight : 8;
> > +               u32 pri3_weight : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
> > +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
> > +union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
> > +       struct {
> > +               u32 rsvz0 : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
> > +#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
> > +union dlb2_lsp_ldb_sched_ctrl {
> > +       struct {
> > +               u32 cq : 8;
> > +               u32 qidix : 3;
> > +               u32 value : 1;
> > +               u32 nalb_haswork_v : 1;
> > +               u32 rlist_haswork_v : 1;
> > +               u32 slist_haswork_v : 1;
> > +               u32 inflight_ok_v : 1;
> > +               u32 aqed_nfull_v : 1;
> > +               u32 rsvz0 : 15;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
> > +#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
> > +union dlb2_lsp_dir_sch_cnt_l {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
> > +#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
> > +union dlb2_lsp_dir_sch_cnt_h {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
> > +#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
> > +union dlb2_lsp_ldb_sch_cnt_l {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
> > +#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
> > +union dlb2_lsp_ldb_sch_cnt_h {
> > +       struct {
> > +               u32 count : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
> > +#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
> > +union dlb2_lsp_cfg_shdw_ctrl {
> > +       struct {
> > +               u32 transfer : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
> > +       (0xa4000074 + (x) * 4)
> > +#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
> > +union dlb2_lsp_cfg_shdw_range_cos {
> > +       struct {
> > +               u32 bw_range : 9;
> > +               u32 rsvz0 : 22;
> > +               u32 no_extra_credit : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
> > +#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
> > +union dlb2_lsp_cfg_ctrl_general_0 {
> > +       struct {
> > +               u32 disab_atq_empty_arb : 1;
> > +               u32 inc_tok_unit_idle : 1;
> > +               u32 disab_rlist_pri : 1;
> > +               u32 inc_cmp_unit_idle : 1;
> > +               u32 rsvz0 : 2;
> > +               u32 dir_single_op : 1;
> > +               u32 dir_half_bw : 1;
> > +               u32 dir_single_out : 1;
> > +               u32 dir_disab_multi : 1;
> > +               u32 atq_single_op : 1;
> > +               u32 atq_half_bw : 1;
> > +               u32 atq_single_out : 1;
> > +               u32 atq_disab_multi : 1;
> > +               u32 dirrpl_single_op : 1;
> > +               u32 dirrpl_half_bw : 1;
> > +               u32 dirrpl_single_out : 1;
> > +               u32 lbrpl_single_op : 1;
> > +               u32 lbrpl_half_bw : 1;
> > +               u32 lbrpl_single_out : 1;
> > +               u32 ldb_single_op : 1;
> > +               u32 ldb_half_bw : 1;
> > +               u32 ldb_disab_multi : 1;
> > +               u32 atm_single_sch : 1;
> > +               u32 atm_single_cmp : 1;
> > +               u32 ldb_ce_tog_arb : 1;
> > +               u32 rsvz1 : 1;
> > +               u32 smon0_valid_sel : 2;
> > +               u32 smon0_value_sel : 1;
> > +               u32 smon0_compare_sel : 2;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
> > +#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
> > +union dlb2_cfg_mstr_diag_reset_sts {
> > +       struct {
> > +               u32 chp_pf_reset_done : 1;
> > +               u32 rop_pf_reset_done : 1;
> > +               u32 lsp_pf_reset_done : 1;
> > +               u32 nalb_pf_reset_done : 1;
> > +               u32 ap_pf_reset_done : 1;
> > +               u32 dp_pf_reset_done : 1;
> > +               u32 qed_pf_reset_done : 1;
> > +               u32 dqed_pf_reset_done : 1;
> > +               u32 aqed_pf_reset_done : 1;
> > +               u32 sys_pf_reset_done : 1;
> > +               u32 pf_reset_active : 1;
> > +               u32 flrsm_state : 7;
> > +               u32 rsvd0 : 13;
> > +               u32 dlb_proc_reset_done : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
> > +#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
> > +union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
> > +       struct {
> > +               u32 chp_pipeidle : 1;
> > +               u32 rop_pipeidle : 1;
> > +               u32 lsp_pipeidle : 1;
> > +               u32 nalb_pipeidle : 1;
> > +               u32 ap_pipeidle : 1;
> > +               u32 dp_pipeidle : 1;
> > +               u32 qed_pipeidle : 1;
> > +               u32 dqed_pipeidle : 1;
> > +               u32 aqed_pipeidle : 1;
> > +               u32 sys_pipeidle : 1;
> > +               u32 chp_unit_idle : 1;
> > +               u32 rop_unit_idle : 1;
> > +               u32 lsp_unit_idle : 1;
> > +               u32 nalb_unit_idle : 1;
> > +               u32 ap_unit_idle : 1;
> > +               u32 dp_unit_idle : 1;
> > +               u32 qed_unit_idle : 1;
> > +               u32 dqed_unit_idle : 1;
> > +               u32 aqed_unit_idle : 1;
> > +               u32 sys_unit_idle : 1;
> > +               u32 rsvd1 : 4;
> > +               u32 mstr_cfg_ring_idle : 1;
> > +               u32 mstr_cfg_mstr_idle : 1;
> > +               u32 mstr_flr_clkreq_b : 1;
> > +               u32 mstr_proc_idle : 1;
> > +               u32 mstr_proc_idle_masked : 1;
> > +               u32 rsvd0 : 2;
> > +               u32 dlb_func_idle : 1;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
> > +#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
> > +union dlb2_cfg_mstr_cfg_pm_status {
> > +       struct {
> > +               u32 prochot : 1;
> > +               u32 pgcb_dlb_idle : 1;
> > +               u32 pgcb_dlb_pg_rdy_ack_b : 1;
> > +               u32 pmsm_pgcb_req_b : 1;
> > +               u32 pgbc_pmc_pg_req_b : 1;
> > +               u32 pmc_pgcb_pg_ack_b : 1;
> > +               u32 pmc_pgcb_fet_en_b : 1;
> > +               u32 pgcb_fet_en_b : 1;
> > +               u32 rsvz0 : 1;
> > +               u32 rsvz1 : 1;
> > +               u32 fuse_force_on : 1;
> > +               u32 fuse_proc_disable : 1;
> > +               u32 rsvz2 : 1;
> > +               u32 rsvz3 : 1;
> > +               u32 pm_fsm_d0tod3_ok : 1;
> > +               u32 pm_fsm_d3tod0_ok : 1;
> > +               u32 dlb_in_d3 : 1;
> > +               u32 rsvz4 : 7;
> > +               u32 pmsm : 8;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
> > +#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
> > +union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
> > +       struct {
> > +               u32 disable : 1;
> > +               u32 rsvz0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
> > +#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
> > +       (0x1000 + (x) * 0x4)
> > +#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
> > +union dlb2_func_vf_vf2pf_mailbox {
> > +       struct {
> > +               u32 msg : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
> > +#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
> > +#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
> > +union dlb2_func_vf_vf2pf_mailbox_isr {
> > +       struct {
> > +               u32 isr : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
> > +#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
> > +       (0x2000 + (x) * 0x4)
> > +#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
> > +union dlb2_func_vf_pf2vf_mailbox {
> > +       struct {
> > +               u32 msg : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
> > +#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
> > +union dlb2_func_vf_pf2vf_mailbox_isr {
> > +       struct {
> > +               u32 pf_isr : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
> > +#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
> > +union dlb2_func_vf_vf_msi_isr_pend {
> > +       struct {
> > +               u32 isr_pend : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
> > +#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
> > +union dlb2_func_vf_vf_reset_in_progress {
> > +       struct {
> > +               u32 reset_in_progress : 1;
> > +               u32 rsvd0 : 31;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
> > +#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
> > +union dlb2_func_vf_vf_msi_isr {
> > +       struct {
> > +               u32 vf_msi_isr : 32;
> > +       } field;
> > +       u32 val;
> > +};
> > +
> > +#endif /* __DLB2_REGS_H */
> > diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c
> b/drivers/event/dlb2/pf/base/dlb2_resource.c
> > new file mode 100644
> > index 0000000..6de8b95
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
> > @@ -0,0 +1,274 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#include "dlb2_user.h"
> > +
> > +#include "dlb2_hw_types.h"
> > +#include "dlb2_mbox.h"
> > +#include "dlb2_osdep.h"
> > +#include "dlb2_osdep_bitmap.h"
> > +#include "dlb2_osdep_types.h"
> > +#include "dlb2_regs.h"
> > +#include "dlb2_resource.h"
> > +
> > +static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
> > +{
> > +       int i;
> > +
> > +       dlb2_list_init_head(&domain->used_ldb_queues);
> > +       dlb2_list_init_head(&domain->used_dir_pq_pairs);
> > +       dlb2_list_init_head(&domain->avail_ldb_queues);
> > +       dlb2_list_init_head(&domain->avail_dir_pq_pairs);
> > +
> > +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> > +               dlb2_list_init_head(&domain->used_ldb_ports[i]);
> > +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> > +               dlb2_list_init_head(&domain->avail_ldb_ports[i]);
> > +}
> > +
> > +static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
> > +{
> > +       int i;
> > +
> > +       dlb2_list_init_head(&rsrc->avail_domains);
> > +       dlb2_list_init_head(&rsrc->used_domains);
> > +       dlb2_list_init_head(&rsrc->avail_ldb_queues);
> > +       dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
> > +
> > +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> > +               dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
> > +}
> > +
> > +void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
> > +{
> > +       union dlb2_chp_cfg_chp_csr_ctrl r0;
> > +
> > +       r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
> > +
> > +       r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
> > +
> > +       DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
> > +}
> > +
> > +int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
> > +                             struct dlb2_get_num_resources_args *arg,
> > +                             bool vdev_req,
> > +                             unsigned int vdev_id)
> > +{
> > +       struct dlb2_function_resources *rsrcs;
> > +       struct dlb2_bitmap *map;
> > +       int i;
> > +
> > +       if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
> > +               return -EINVAL;
> > +
> > +       if (vdev_req)
> > +               rsrcs = &hw->vdev[vdev_id];
> > +       else
> > +               rsrcs = &hw->pf;
> > +
> > +       arg->num_sched_domains = rsrcs->num_avail_domains;
> > +
> > +       arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
> > +
> > +       arg->num_ldb_ports = 0;
> > +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> > +               arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
> > +
> > +       arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
> > +       arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
> > +       arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
> > +       arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
> > +
> > +       arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
> > +
> > +       arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
> > +
> > +       map = rsrcs->avail_hist_list_entries;
> > +
> > +       arg->num_hist_list_entries = dlb2_bitmap_count(map);
> > +
> > +       arg->max_contiguous_hist_list_entries =
> > +               dlb2_bitmap_longest_set_range(map);
> > +
> > +       arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
> > +
> > +       arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
> > +
> > +       return 0;
> > +}
> > +
> > +void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
> > +{
> > +       union dlb2_chp_cfg_chp_csr_ctrl r0;
> > +
> > +       r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
> > +
> > +       r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
> > +
> > +       DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
> > +}
> > +
> > +void dlb2_resource_free(struct dlb2_hw *hw)
> > +{
> > +       int i;
> > +
> > +       if (hw->pf.avail_hist_list_entries)
> > +               dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
> > +
> > +       for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
> > +               if (hw->vdev[i].avail_hist_list_entries)
> > +                       dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
> > +       }
> > +}
> > +
> > +int dlb2_resource_init(struct dlb2_hw *hw)
> > +{
> > +       struct dlb2_list_entry *list;
> > +       unsigned int i;
> > +       int ret;
> > +
> > +       /*
> > +        * For optimal load-balancing, ports that map to one or more QIDs in
> > +        * common should not be in numerical sequence. This is application
> > +        * dependent, but the driver interleaves port IDs as much as possible
> > +        * to reduce the likelihood of this. This initial allocation maximizes
> > +        * the average distance between an ID and its immediate neighbors (i.e.
> > +        * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
> > +        * 3, etc.).
> > +        */
> > +       u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
> > +               0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
> > +               16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
> > +               32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
> > +               48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
> > +       };
> > +
> > +       /* Zero-out resource tracking data structures */
> > +       memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
> > +       memset(&hw->pf, 0, sizeof(hw->pf));
> > +
> > +       dlb2_init_fn_rsrc_lists(&hw->pf);
> > +
> > +       for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
> > +               memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
> > +               dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
> > +       }
> > +
> > +       for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
> > +               memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
> > +               dlb2_init_domain_rsrc_lists(&hw->domains[i]);
> > +               hw->domains[i].parent_func = &hw->pf;
> > +       }
> > +
> > +       /* Give all resources to the PF driver */
> > +       hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
> > +       for (i = 0; i < hw->pf.num_avail_domains; i++) {
> > +               list = &hw->domains[i].func_list;
> > +
> > +               dlb2_list_add(&hw->pf.avail_domains, list);
> > +       }
> > +
> > +       hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
> > +       for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
> > +               list = &hw->rsrcs.ldb_queues[i].func_list;
> > +
> > +               dlb2_list_add(&hw->pf.avail_ldb_queues, list);
> > +       }
> > +
> > +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> > +               hw->pf.num_avail_ldb_ports[i] =
> > +                       DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
> > +
> > +       for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
> > +               int cos_id = i >> DLB2_NUM_COS_DOMAINS;
> > +               struct dlb2_ldb_port *port;
> > +
> > +               port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
> > +
> > +               dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
> > +                             &port->func_list);
> > +       }
> > +
> > +       hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
> > +       for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
> > +               list = &hw->rsrcs.dir_pq_pairs[i].func_list;
> > +
> > +               dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
> > +       }
> > +
> > +       hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
> > +       hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
> > +       hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
> > +
> > +       ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
> > +                               DLB2_MAX_NUM_HIST_LIST_ENTRIES);
> > +       if (ret)
> > +               goto unwind;
> > +
> > +       ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
> > +       if (ret)
> > +               goto unwind;
> > +
> > +       for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
> > +               ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
> > +                                       DLB2_MAX_NUM_HIST_LIST_ENTRIES);
> > +               if (ret)
> > +                       goto unwind;
> > +
> > +               ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
> > +               if (ret)
> > +                       goto unwind;
> > +       }
> > +
> > +       /* Initialize the hardware resource IDs */
> > +       for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
> > +               hw->domains[i].id.phys_id = i;
> > +               hw->domains[i].id.vdev_owned = false;
> > +       }
> > +
> > +       for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
> > +               hw->rsrcs.ldb_queues[i].id.phys_id = i;
> > +               hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
> > +       }
> > +
> > +       for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
> > +               hw->rsrcs.ldb_ports[i].id.phys_id = i;
> > +               hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
> > +       }
> > +
> > +       for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
> > +               hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
> > +               hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
> > +       }
> > +
> > +       for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
> > +               hw->rsrcs.sn_groups[i].id = i;
> > +               /* Default mode (0) is 64 sequence numbers per queue */
> > +               hw->rsrcs.sn_groups[i].mode = 0;
> > +               hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
> > +               hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
> > +       }
> > +
> > +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> > +               hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
> > +
> > +       return 0;
> > +
> > +unwind:
> > +       dlb2_resource_free(hw);
> > +
> > +       return ret;
> > +}
> > +
> > +void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
> > +{
> > +       union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
> > +
> > +       r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
> > +
> > +       r0.field.disable = 0;
> > +
> > +       DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
> > +}
> > diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h
> b/drivers/event/dlb2/pf/base/dlb2_resource.h
> > new file mode 100644
> > index 0000000..503fdf3
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
> > @@ -0,0 +1,1913 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef __DLB2_RESOURCE_H
> > +#define __DLB2_RESOURCE_H
> > +
> > +#include "dlb2_user.h"
> > +
> > +#include "dlb2_hw_types.h"
> > +#include "dlb2_osdep_types.h"
> > +
> > +/**
> > + * dlb2_resource_init() - initialize the device
> > + * @hw: pointer to struct dlb2_hw.
> > + *
> > + * This function initializes the device's software state (pointed to by the hw
> > + * argument) and programs global scheduling QoS registers. This function
> should
> > + * be called during driver initialization.
> > + *
> > + * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
> > + * device is reset.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + */
> > +int dlb2_resource_init(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_resource_free() - free device state memory
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function frees software state pointed to by dlb2_hw. This function
> > + * should be called when resetting the device or unloading the driver.
> > + */
> > +void dlb2_resource_free(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_resource_reset() - reset in-use resources to their initial state
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function resets in-use resources, and makes them available for use.
> > + * All resources go back to their owning function, whether a PF or a VF.
> > + */
> > +void dlb2_resource_reset(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_hw_create_sched_domain() - create a scheduling domain
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @args: scheduling domain creation arguments.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function creates a scheduling domain containing the resources
> specified
> > + * in args. The individual resources (queues, ports, credits) can be configured
> > + * after creating a scheduling domain.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error. If successful, resp-
> >id
> > + * contains the domain ID.
> > + *
> > + * resp->id contains a virtual ID if vdev_request is true.
> > + *
> > + * Errors:
> > + * EINVAL - A requested resource is unavailable, or the requested domain
> name
> > + *         is already in use.
> > + * EFAULT - Internal error (resp->status not set).
> > + */
> > +int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
> > +                               struct dlb2_create_sched_domain_args *args,
> > +                               struct dlb2_cmd_response *resp,
> > +                               bool vdev_request,
> > +                               unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_create_ldb_queue() - create a load-balanced queue
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: queue creation arguments.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function creates a load-balanced queue.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error. If successful, resp-
> >id
> > + * contains the queue ID.
> > + *
> > + * resp->id contains a virtual ID if vdev_request is true.
> > + *
> > + * Errors:
> > + * EINVAL - A requested resource is unavailable, the domain is not configured,
> > + *         the domain has already been started, or the requested queue name is
> > + *         already in use.
> > + * EFAULT - Internal error (resp->status not set).
> > + */
> > +int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
> > +                            u32 domain_id,
> > +                            struct dlb2_create_ldb_queue_args *args,
> > +                            struct dlb2_cmd_response *resp,
> > +                            bool vdev_request,
> > +                            unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_create_dir_queue() - create a directed queue
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: queue creation arguments.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function creates a directed queue.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error. If successful, resp-
> >id
> > + * contains the queue ID.
> > + *
> > + * resp->id contains a virtual ID if vdev_request is true.
> > + *
> > + * Errors:
> > + * EINVAL - A requested resource is unavailable, the domain is not configured,
> > + *         or the domain has already been started.
> > + * EFAULT - Internal error (resp->status not set).
> > + */
> > +int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
> > +                            u32 domain_id,
> > +                            struct dlb2_create_dir_queue_args *args,
> > +                            struct dlb2_cmd_response *resp,
> > +                            bool vdev_request,
> > +                            unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_create_dir_port() - create a directed port
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: port creation arguments.
> > + * @cq_dma_base: base address of the CQ memory. This can be a PA or an
> IOVA.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function creates a directed port.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error. If successful, resp-
> >id
> > + * contains the port ID.
> > + *
> > + * resp->id contains a virtual ID if vdev_request is true.
> > + *
> > + * Errors:
> > + * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
> > + *         pointer address is not properly aligned, the domain is not
> > + *         configured, or the domain has already been started.
> > + * EFAULT - Internal error (resp->status not set).
> > + */
> > +int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
> > +                           u32 domain_id,
> > +                           struct dlb2_create_dir_port_args *args,
> > +                           uintptr_t cq_dma_base,
> > +                           struct dlb2_cmd_response *resp,
> > +                           bool vdev_request,
> > +                           unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_create_ldb_port() - create a load-balanced port
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: port creation arguments.
> > + * @cq_dma_base: base address of the CQ memory. This can be a PA or an
> IOVA.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function creates a load-balanced port.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error. If successful, resp-
> >id
> > + * contains the port ID.
> > + *
> > + * resp->id contains a virtual ID if vdev_request is true.
> > + *
> > + * Errors:
> > + * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
> > + *         pointer address is not properly aligned, the domain is not
> > + *         configured, or the domain has already been started.
> > + * EFAULT - Internal error (resp->status not set).
> > + */
> > +int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
> > +                           u32 domain_id,
> > +                           struct dlb2_create_ldb_port_args *args,
> > +                           uintptr_t cq_dma_base,
> > +                           struct dlb2_cmd_response *resp,
> > +                           bool vdev_request,
> > +                           unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_start_domain() - start a scheduling domain
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: start domain arguments.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function starts a scheduling domain, which allows applications to send
> > + * traffic through it. Once a domain is started, its resources can no longer be
> > + * configured (besides QID remapping and port enable/disable).
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error.
> > + *
> > + * Errors:
> > + * EINVAL - the domain is not configured, or the domain is already started.
> > + */
> > +int dlb2_hw_start_domain(struct dlb2_hw *hw,
> > +                        u32 domain_id,
> > +                        struct dlb2_start_domain_args *args,
> > +                        struct dlb2_cmd_response *resp,
> > +                        bool vdev_request,
> > +                        unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: map QID arguments.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function configures the DLB to schedule QEs from the specified queue
> > + * to the specified port. Each load-balanced port can be mapped to up to 8
> > + * queues; each load-balanced queue can potentially map to all the
> > + * load-balanced ports.
> > + *
> > + * A successful return does not necessarily mean the mapping was
> configured. If
> > + * this function is unable to immediately map the queue to the port, it will
> > + * add the requested operation to a per-port list of pending map/unmap
> > + * operations, and (if it's not already running) launch a kernel thread that
> > + * periodically attempts to process all pending operations. In a sense, this is
> > + * an asynchronous function.
> > + *
> > + * This asynchronicity creates two views of the state of hardware: the actual
> > + * hardware state and the requested state (as if every request completed
> > + * immediately). If there are any pending map/unmap operations, the
> requested
> > + * state will differ from the actual state. All validation is performed with
> > + * respect to the pending state; for instance, if there are 8 pending map
> > + * operations for port X, a request for a 9th will fail because a load-balanced
> > + * port can only map up to 8 queues.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error.
> > + *
> > + * Errors:
> > + * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
> > + *         the domain is not configured.
> > + * EFAULT - Internal error (resp->status not set).
> > + */
> > +int dlb2_hw_map_qid(struct dlb2_hw *hw,
> > +                   u32 domain_id,
> > +                   struct dlb2_map_qid_args *args,
> > +                   struct dlb2_cmd_response *resp,
> > +                   bool vdev_request,
> > +                   unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-
> balanced port
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: unmap QID arguments.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function configures the DLB to stop scheduling QEs from the specified
> > + * queue to the specified port.
> > + *
> > + * A successful return does not necessarily mean the mapping was removed.
> If
> > + * this function is unable to immediately unmap the queue from the port, it
> > + * will add the requested operation to a per-port list of pending map/unmap
> > + * operations, and (if it's not already running) launch a kernel thread that
> > + * periodically attempts to process all pending operations. See
> > + * dlb2_hw_map_qid() for more details.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error.
> > + *
> > + * Errors:
> > + * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
> > + *         the domain is not configured.
> > + * EFAULT - Internal error (resp->status not set).
> > + */
> > +int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
> > +                     u32 domain_id,
> > +                     struct dlb2_unmap_qid_args *args,
> > +                     struct dlb2_cmd_response *resp,
> > +                     bool vdev_request,
> > +                     unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_finish_unmap_qid_procedures() - finish any pending unmap
> procedures
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function attempts to finish any outstanding unmap procedures.
> > + * This function should be called by the kernel thread responsible for
> > + * finishing map/unmap procedures.
> > + *
> > + * Return:
> > + * Returns the number of procedures that weren't completed.
> > + */
> > +unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_finish_map_qid_procedures() - finish any pending map procedures
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function attempts to finish any outstanding map procedures.
> > + * This function should be called by the kernel thread responsible for
> > + * finishing map/unmap procedures.
> > + *
> > + * Return:
> > + * Returns the number of procedures that weren't completed.
> > + */
> > +unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: port enable arguments.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function configures the DLB to schedule QEs to a load-balanced port.
> > + * Ports are enabled by default.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error.
> > + *
> > + * Errors:
> > + * EINVAL - The port ID is invalid or the domain is not configured.
> > + * EFAULT - Internal error (resp->status not set).
> > + */
> > +int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
> > +                           u32 domain_id,
> > +                           struct dlb2_enable_ldb_port_args *args,
> > +                           struct dlb2_cmd_response *resp,
> > +                           bool vdev_request,
> > +                           unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: port disable arguments.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function configures the DLB to stop scheduling QEs to a load-balanced
> > + * port. Ports are enabled by default.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error.
> > + *
> > + * Errors:
> > + * EINVAL - The port ID is invalid or the domain is not configured.
> > + * EFAULT - Internal error (resp->status not set).
> > + */
> > +int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
> > +                            u32 domain_id,
> > +                            struct dlb2_disable_ldb_port_args *args,
> > +                            struct dlb2_cmd_response *resp,
> > +                            bool vdev_request,
> > +                            unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: port enable arguments.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function configures the DLB to schedule QEs to a directed port.
> > + * Ports are enabled by default.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error.
> > + *
> > + * Errors:
> > + * EINVAL - The port ID is invalid or the domain is not configured.
> > + * EFAULT - Internal error (resp->status not set).
> > + */
> > +int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
> > +                           u32 domain_id,
> > +                           struct dlb2_enable_dir_port_args *args,
> > +                           struct dlb2_cmd_response *resp,
> > +                           bool vdev_request,
> > +                           unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: port disable arguments.
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function configures the DLB to stop scheduling QEs to a directed port.
> > + * Ports are enabled by default.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error.
> > + *
> > + * Errors:
> > + * EINVAL - The port ID is invalid or the domain is not configured.
> > + * EFAULT - Internal error (resp->status not set).
> > + */
> > +int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
> > +                            u32 domain_id,
> > +                            struct dlb2_disable_dir_port_args *args,
> > +                            struct dlb2_cmd_response *resp,
> > +                            bool vdev_request,
> > +                            unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
> > + *                                     interrupts
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @port_id: load-balanced port ID.
> > + * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X
> mode,
> > + *         else a value up to 64.
> > + * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or
> DLB2_CQ_ISR_MODE_MSIX)
> > + * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
> > + *     virtual port ID to a physical port ID. Ignored if mode is not MSI.
> > + * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
> > + * @threshold: the minimum CQ depth at which the interrupt can fire. Must
> be
> > + *     greater than 0.
> > + *
> > + * This function configures the DLB registers for load-balanced CQ's
> > + * interrupts. This doesn't enable the CQ's interrupt; that can be done with
> > + * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - The port ID is invalid.
> > + */
> > +int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
> > +                                   int port_id,
> > +                                   int vector,
> > +                                   int mode,
> > +                                   unsigned int vf,
> > +                                   unsigned int owner_vf,
> > +                                   u16 threshold);
> > +
> > +/**
> > + * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @port_id: load-balanced port ID.
> > + * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X
> mode,
> > + *         else a value up to 64.
> > + * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or
> DLB2_CQ_ISR_MODE_MSIX)
> > + * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
> > + *     virtual port ID to a physical port ID. Ignored if mode is not MSI.
> > + * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
> > + * @threshold: the minimum CQ depth at which the interrupt can fire. Must
> be
> > + *     greater than 0.
> > + *
> > + * This function configures the DLB registers for directed CQ's interrupts.
> > + * This doesn't enable the CQ's interrupt; that can be done with
> > + * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - The port ID is invalid.
> > + */
> > +int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
> > +                                   int port_id,
> > +                                   int vector,
> > +                                   int mode,
> > +                                   unsigned int vf,
> > +                                   unsigned int owner_vf,
> > +                                   u16 threshold);
> > +
> > +/**
> > + * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
> > + * @hw: dlb2_hw handle for a particular device.
> > + */
> > +void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
> > + * @hw: dlb2_hw handle for a particular device.
> > + */
> > +void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or
> DLB2_MSIX_MODE_COMPRESSED)
> > + *
> > + * This function configures the hardware to use either packed or compressed
> > + * mode. This function should not be called if using MSI interrupts.
> > + */
> > +void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
> > +
> > +/**
> > + * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vector: interrupt vector.
> > + *
> > + * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
> > + * acked in dlb2_ack_compressed_cq_intr().
> > + */
> > +void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
> > +
> > +/**
> > + * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @port_id: port ID
> > + * @is_ldb: true for load-balanced port, false for a directed port
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function arms the CQ's interrupt. The CQ must be configured prior to
> > + * calling this function.
> > + *
> > + * The function does no parameter validation; that is the caller's
> > + * responsibility.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return: returns 0 upon success, <0 otherwise.
> > + *
> > + * EINVAL - Invalid port ID.
> > + */
> > +int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
> > +                         int port_id,
> > +                         bool is_ldb,
> > +                         bool vdev_request,
> > +                         unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt
> status
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @ldb_interrupts: 2-entry array of u32 bitmaps
> > + * @dir_interrupts: 4-entry array of u32 bitmaps
> > + *
> > + * This function can be called from a compressed CQ interrupt handler to
> > + * determine which CQ interrupts have fired. The caller should take
> appropriate
> > + * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
> > + * with dlb2_ack_compressed_cq_intr().
> > + */
> > +void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
> > +                                        u32 *ldb_interrupts,
> > +                                        u32 *dir_interrupts);
> > +
> > +/**
> > + * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @ldb_interrupts: 2-entry array of u32 bitmaps
> > + * @dir_interrupts: 4-entry array of u32 bitmaps
> > + *
> > + * This function ACKs compressed CQ interrupts. Its arguments should be the
> > + * same ones passed to dlb2_read_compressed_cq_intr_status().
> > + */
> > +void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
> > +                                u32 *ldb_interrupts,
> > +                                u32 *dir_interrupts);
> > +
> > +/**
> > + * dlb2_read_vf_intr_status() - read the VF interrupt status register
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function can be called from a VF's interrupt handler to determine
> > + * which interrupts have fired. The first 31 bits correspond to CQ interrupt
> > + * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
> > + *
> > + * Return:
> > + * Returns a bit vector indicating which interrupt vectors are active.
> > + */
> > +u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_ack_vf_intr_status() - ack VF interrupts
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @interrupts: 32-bit bitmap
> > + *
> > + * This function ACKs a VF's interrupts. Its interrupts argument should be the
> > + * value returned by dlb2_read_vf_intr_status().
> > + */
> > +void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
> > +
> > +/**
> > + * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @interrupts: 32-bit bitmap
> > + *
> > + * This function clears the VF's MSI interrupt pending register. Its interrupts
> > + * argument should be contain the MSI vectors to ACK. For example, if MSI
> MME
> > + * is in mode 0, then one bit 0 should ever be set.
> > + */
> > +void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
> > +
> > +/**
> > + * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * When done processing the PF mailbox request, this function unsets
> > + * the PF's mailbox ISR register.
> > + */
> > +void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
> > + *                                     vdevs
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * When the vdev->PF ISR fires, this function can be called to determine which
> > + * vdev(s) are requesting service. This bitvector must be passed to
> > + * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
> > + * vdevs.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns a bit vector indicating which VFs (0-15) have requested service.
> > + */
> > +u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
> > + *
> > + * When done processing all VF mailbox requests, this function unsets the VF's
> > + * mailbox ISR register.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + */
> > +void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
> > +
> > +/**
> > + * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
> > + *                                 FLR
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * When the VF FLR ISR fires, this function can be called to determine which
> > + * VF(s) are requesting FLRs. This bitvector must passed to
> > + * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
> > + *
> > + * Return:
> > + * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
> > + */
> > +u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
> > + *
> > + * When done processing all VF FLR requests, this function unsets the VF's FLR
> > + * ISR register.
> > + */
> > +void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
> > +
> > +/**
> > + * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
> > + * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
> > + *
> > + * When done processing all VF requests, this function communicates to the
> > + * hardware that processing is complete.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + */
> > +void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
> > +                            u32 mbox_bitvec,
> > +                            u32 flr_bitvec);
> > +
> > +/**
> > + * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function reads the watchdog timer interrupt cause registers to
> > + * determine which port(s) had a watchdog timeout, and notifies the
> > + * application(s) that own the port(s).
> > + */
> > +void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_process_alarm_interrupt() - process an alarm interrupt
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function reads and logs the alarm syndrome, then acks the interrupt.
> > + * This function should be called from the alarm interrupt handler when
> > + * interrupt vector DLB2_INT_ALARM fires.
> > + */
> > +void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function reads the alarm syndrome, logs it, notifies user-space, and
> > + * acks the interrupt. This function should be called from the alarm interrupt
> > + * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
> > + *
> > + * Return:
> > + * Returns true if an ingress error interrupt occurred, false otherwise
> > + */
> > +bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_get_group_sequence_numbers() - return a group's number of SNs per
> queue
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @group_id: sequence number group ID.
> > + *
> > + * This function returns the configured number of sequence numbers per
> queue
> > + * for the specified group.
> > + *
> > + * Return:
> > + * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
> > + */
> > +int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
> > +                                   unsigned int group_id);
> > +
> > +/**
> > + * dlb2_get_group_sequence_number_occupancy() - return a group's in-use
> slots
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @group_id: sequence number group ID.
> > + *
> > + * This function returns the group's number of in-use slots (i.e. load-balanced
> > + * queues using the specified group).
> > + *
> > + * Return:
> > + * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
> > + */
> > +int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
> > +                                            unsigned int group_id);
> > +
> > +/**
> > + * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per
> queue
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @group_id: sequence number group ID.
> > + * @val: requested amount of sequence numbers per queue.
> > + *
> > + * This function configures the group's number of sequence numbers per
> queue.
> > + * val can be a power-of-two between 32 and 1024, inclusive. This setting can
> > + * be configured until the first ordered load-balanced queue is configured, at
> > + * which point the configuration is locked.
> > + *
> > + * Return:
> > + * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
> > + * ordered queue is configured.
> > + */
> > +int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
> > +                                   unsigned int group_id,
> > +                                   unsigned long val);
> > +
> > +/**
> > + * dlb2_reset_domain() - reset a scheduling domain
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function resets and frees a DLB 2.0 scheduling domain and its
> associated
> > + * resources.
> > + *
> > + * Pre-condition: the driver must ensure software has stopped sending QEs
> > + * through this domain's producer ports before invoking this function, or
> > + * undefined behavior will result.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, -1 otherwise.
> > + *
> > + * EINVAL - Invalid domain ID, or the domain is not configured.
> > + * EFAULT - Internal error. (Possibly caused if software is the pre-condition
> > + *         is not met.)
> > + * ETIMEDOUT - Hardware component didn't reset in the expected time.
> > + */
> > +int dlb2_reset_domain(struct dlb2_hw *hw,
> > +                     u32 domain_id,
> > +                     bool vdev_request,
> > +                     unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a
> domain
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @port_id: indicates whether this request came from a VF.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function returns whether a load-balanced port is owned by a specified
> > + * domain.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 if false, 1 if true, <0 otherwise.
> > + *
> > + * EINVAL - Invalid domain or port ID, or the domain is not configured.
> > + */
> > +int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
> > +                                 u32 domain_id,
> > +                                 u32 port_id,
> > +                                 bool vdev_request,
> > +                                 unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a
> domain
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @port_id: indicates whether this request came from a VF.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function returns whether a directed port is owned by a specified
> > + * domain.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 if false, 1 if true, <0 otherwise.
> > + *
> > + * EINVAL - Invalid domain or port ID, or the domain is not configured.
> > + */
> > +int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
> > +                                 u32 domain_id,
> > +                                 u32 port_id,
> > +                                 bool vdev_request,
> > +                                 unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_get_num_resources() - query the PCI function's available
> resources
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @arg: pointer to resource counts.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function returns the number of available resources for the PF or for a
> > + * VF.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
> > + * invalid.
> > + */
> > +int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
> > +                             struct dlb2_get_num_resources_args *arg,
> > +                             bool vdev_request,
> > +                             unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_get_num_used_resources() - query the PCI function's used
> resources
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @arg: pointer to resource counts.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function returns the number of resources in use by the PF or a VF. It
> > + * fills in the fields that args points to, except the following:
> > + * - max_contiguous_atomic_inflights
> > + * - max_contiguous_hist_list_entries
> > + * - max_contiguous_ldb_credits
> > + * - max_contiguous_dir_credits
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
> > + * invalid.
> > + */
> > +int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
> > +                                  struct dlb2_get_num_resources_args *arg,
> > +                                  bool vdev_request,
> > +                                  unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message
> to
> > + *                                    the PF
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function sends a VF->PF mailbox message. It is asynchronous, so it
> > + * returns once the message is sent but potentially before the PF has
> processed
> > + * the message. The caller must call dlb2_vdev_to_pf_complete() to
> determine
> > + * when the PF has finished processing the request.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + */
> > +void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_vdev_to_pf_complete() - check the status of an asynchronous
> mailbox
> > + *                              request
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function returns a boolean indicating whether the PF has finished
> > + * processing a VF->PF mailbox request. It should only be called after sending
> > + * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + */
> > +bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_vf_flr_complete() - check the status of a VF FLR
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function returns a boolean indicating whether the PF has finished
> > + * executing the VF FLR. It should only be called after setting the VF's FLR
> > + * bit.
> > + */
> > +bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to
> a
> > + *                                     vdev
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vdev_id: vdev ID.
> > + *
> > + * This function sends a PF->vdev mailbox message. It is asynchronous, so it
> > + * returns once the message is sent but potentially before the vdev has
> > + * processed the message. The caller must call dlb2_pf_to_vdev_complete()
> to
> > + * determine when the vdev has finished processing the request.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + */
> > +void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int
> vdev_id);
> > +
> > +/**
> > + * dlb2_pf_to_vdev_complete() - check the status of an asynchronous
> mailbox
> > + *                            request
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vdev_id: vdev ID.
> > + *
> > + * This function returns a boolean indicating whether the vdev has finished
> > + * processing a PF->vdev mailbox request. It should only be called after
> > + * sending an asynchronous request with
> dlb2_send_async_pf_to_vdev_msg().
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + */
> > +bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vf_id: VF ID.
> > + * @data: pointer to message data.
> > + * @len: size, in bytes, of the data array.
> > + *
> > + * This function copies one of the PF's VF->PF mailboxes into the array
> pointed
> > + * to by data.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
> > + */
> > +int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
> > +                            unsigned int vf_id,
> > +                            void *data,
> > +                            int len);
> > +
> > +/**
> > + * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vf_id: VF ID.
> > + * @data: pointer to message data.
> > + * @len: size, in bytes, of the data array.
> > + *
> > + * This function copies one of the PF's VF->PF mailboxes into the array
> pointed
> > + * to by data.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
> > + */
> > +int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
> > +                             unsigned int vf_id,
> > +                             void *data,
> > +                             int len);
> > +
> > +/**
> > + * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vf_id: VF ID.
> > + * @data: pointer to message data.
> > + * @len: size, in bytes, of the data array.
> > + *
> > + * This function copies the user-provided message data into of the PF's VF-
> >PF
> > + * mailboxes.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
> > + */
> > +int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
> > +                              unsigned int vf_id,
> > +                              void *data,
> > +                              int len);
> > +
> > +/**
> > + * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vf_id: VF ID.
> > + * @data: pointer to message data.
> > + * @len: size, in bytes, of the data array.
> > + *
> > + * This function copies the user-provided message data into of the PF's VF-
> >PF
> > + * mailboxes.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
> > + */
> > +int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
> > +                             unsigned int vf_id,
> > +                             void *data,
> > +                             int len);
> > +
> > +/**
> > + * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @data: pointer to message data.
> > + * @len: size, in bytes, of the data array.
> > + *
> > + * This function copies the VF's PF->VF mailbox into the array pointed to by
> > + * data.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
> > + */
> > +int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
> > +
> > +/**
> > + * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @data: pointer to message data.
> > + * @len: size, in bytes, of the data array.
> > + *
> > + * This function copies the VF's PF->VF mailbox into the array pointed to by
> > + * data.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
> > + */
> > +int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
> > +
> > +/**
> > + * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @data: pointer to message data.
> > + * @len: size, in bytes, of the data array.
> > + *
> > + * This function copies the user-provided message data into of the VF's PF-
> >VF
> > + * mailboxes.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
> > + */
> > +int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
> > +
> > +/**
> > + * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @data: pointer to message data.
> > + * @len: size, in bytes, of the data array.
> > + *
> > + * This function copies the user-provided message data into of the VF's PF-
> >VF
> > + * mailboxes.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
> > + */
> > +int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
> > +
> > +/**
> > + * dlb2_reset_vdev() - reset the hardware owned by a virtual device
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + *
> > + * This function resets the hardware owned by a vdev, by resetting the vdev's
> > + * domains one by one.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + */
> > +int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
> > +
> > +/**
> > + * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + *
> > + * This function returns whether or not the vdev's resource assignments are
> > + * locked. If locked, no resources can be added to or subtracted from the
> > + * group.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + */
> > +bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
> > +
> > +/**
> > + * dlb2_lock_vdev() - lock the vdev's resources
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + *
> > + * This function sets a flag indicating that the vdev is using its resources.
> > + * When the vdev is locked, its resource assignment cannot be changed.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + */
> > +void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
> > +
> > +/**
> > + * dlb2_unlock_vdev() - unlock the vdev's resources
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + *
> > + * This function unlocks the vdev's resource assignment, allowing it to be
> > + * modified.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + */
> > +void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
> > +
> > +/**
> > + * dlb2_update_vdev_sched_domains() - update the domains assigned to a
> vdev
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + * @num: number of scheduling domains to assign to this vdev
> > + *
> > + * This function assigns num scheduling domains to the specified vdev. If the
> > + * vdev already has domains assigned, this existing assignment is adjusted
> > + * accordingly.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - id is invalid, or the requested number of resources are
> > + *         unavailable.
> > + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> > + */
> > +int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
> > +
> > +/**
> > + * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a
> vdev
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + * @num: number of LDB queues to assign to this vdev
> > + *
> > + * This function assigns num LDB queues to the specified vdev. If the vdev
> > + * already has LDB queues assigned, this existing assignment is adjusted
> > + * accordingly.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - id is invalid, or the requested number of resources are
> > + *         unavailable.
> > + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> > + */
> > +int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
> > +
> > +/**
> > + * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + * @num: number of LDB ports to assign to this vdev
> > + *
> > + * This function assigns num LDB ports to the specified vdev. If the vdev
> > + * already has LDB ports assigned, this existing assignment is adjusted
> > + * accordingly.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - id is invalid, or the requested number of resources are
> > + *         unavailable.
> > + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> > + */
> > +int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
> > +
> > +/**
> > + * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a
> vdev
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + * @cos: class-of-service ID
> > + * @num: number of LDB ports to assign to this vdev
> > + *
> > + * This function assigns num LDB ports from class-of-service cos to the
> > + * specified vdev. If the vdev already has LDB ports from this class-of-service
> > + * assigned, this existing assignment is adjusted accordingly.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - id is invalid, or the requested number of resources are
> > + *         unavailable.
> > + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> > + */
> > +int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
> > +                                  u32 id,
> > +                                  u32 cos,
> > +                                  u32 num);
> > +
> > +/**
> > + * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + * @num: number of DIR ports to assign to this vdev
> > + *
> > + * This function assigns num DIR ports to the specified vdev. If the vdev
> > + * already has DIR ports assigned, this existing assignment is adjusted
> > + * accordingly.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - id is invalid, or the requested number of resources are
> > + *         unavailable.
> > + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> > + */
> > +int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
> > +
> > +/**
> > + * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + * @num: number of LDB credit credits to assign to this vdev
> > + *
> > + * This function assigns num LDB credit to the specified vdev. If the vdev
> > + * already has LDB credits assigned, this existing assignment is adjusted
> > + * accordingly. vdevs are assigned a contiguous chunk of credits, so this
> > + * function may fail if a sufficiently large contiguous chunk is not available.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - id is invalid, or the requested number of resources are
> > + *         unavailable.
> > + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> > + */
> > +int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
> > +
> > +/**
> > + * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + * @num: number of DIR credits to assign to this vdev
> > + *
> > + * This function assigns num DIR credit to the specified vdev. If the vdev
> > + * already has DIR credits assigned, this existing assignment is adjusted
> > + * accordingly. vdevs are assigned a contiguous chunk of credits, so this
> > + * function may fail if a sufficiently large contiguous chunk is not available.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - id is invalid, or the requested number of resources are
> > + *         unavailable.
> > + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> > + */
> > +int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
> > +
> > +/**
> > + * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL
> entries
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + * @num: number of history list entries to assign to this vdev
> > + *
> > + * This function assigns num history list entries to the specified vdev. If the
> > + * vdev already has history list entries assigned, this existing assignment is
> > + * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
> > + * this function may fail if a sufficiently large contiguous chunk is not
> > + * available.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - id is invalid, or the requested number of resources are
> > + *         unavailable.
> > + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> > + */
> > +int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
> > +
> > +/**
> > + * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + * @num: number of atomic inflights to assign to this vdev
> > + *
> > + * This function assigns num atomic inflights to the specified vdev. If the vdev
> > + * already has atomic inflights assigned, this existing assignment is adjusted
> > + * accordingly. vdevs are assigned a contiguous chunk of entries, so this
> > + * function may fail if a sufficiently large contiguous chunk is not available.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - id is invalid, or the requested number of resources are
> > + *         unavailable.
> > + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> > + */
> > +int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32
> num);
> > +
> > +/**
> > + * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + *
> > + * This function takes any resources currently assigned to the vdev and
> > + * reassigns them to the PF.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - id is invalid
> > + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> > + */
> > +int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
> > +
> > +/**
> > + * dlb2_notify_vf() - send an alarm to a VF
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vf_id: VF ID
> > + * @notification: notification
> > + *
> > + * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
> > + *
> > + * Return:
> > + * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
> > + */
> > +int dlb2_notify_vf(struct dlb2_hw *hw,
> > +                  unsigned int vf_id,
> > +                  u32 notification);
> > +
> > +/**
> > + * dlb2_vdev_in_use() - query whether a virtual device is in use
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual device ID
> > + *
> > + * This function sends a mailbox request to the vdev to query whether the
> vdev
> > + * is in use.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
> > + * an internal error occurs.
> > + */
> > +int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
> > +
> > +/**
> > + * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * Clearing the PMCSR must be done at initialization to make the device fully
> > + * operational.
> > + */
> > +void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced
> queue
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: queue depth args
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function returns the depth of a load-balanced queue.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error. If successful, resp-
> >id
> > + * contains the depth.
> > + *
> > + * Errors:
> > + * EINVAL - Invalid domain ID or queue ID.
> > + */
> > +int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
> > +                               u32 domain_id,
> > +                               struct dlb2_get_ldb_queue_depth_args *args,
> > +                               struct dlb2_cmd_response *resp,
> > +                               bool vdev_request,
> > +                               unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: queue depth args
> > + * @resp: response structure.
> > + * @vdev_request: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * This function returns the depth of a directed queue.
> > + *
> > + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> > + * device.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error. If successful, resp-
> >id
> > + * contains the depth.
> > + *
> > + * Errors:
> > + * EINVAL - Invalid domain ID or queue ID.
> > + */
> > +int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
> > +                               u32 domain_id,
> > +                               struct dlb2_get_dir_queue_depth_args *args,
> > +                               struct dlb2_cmd_response *resp,
> > +                               bool vdev_request,
> > +                               unsigned int vdev_id);
> > +
> > +enum dlb2_virt_mode {
> > +       DLB2_VIRT_NONE,
> > +       DLB2_VIRT_SRIOV,
> > +       DLB2_VIRT_SIOV,
> > +
> > +       /* NUM_DLB2_VIRT_MODES must be last */
> > +       NUM_DLB2_VIRT_MODES,
> > +};
> > +
> > +/**
> > + * dlb2_hw_set_virt_mode() - set the device's virtualization mode
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @mode: either none, SR-IOV, or Scalable IOV.
> > + *
> > + * This function sets the virtualization mode of the device. This controls
> > + * whether the device uses a software or hardware mailbox.
> > + *
> > + * This should be called by the PF driver when either SR-IOV or Scalable IOV is
> > + * selected as the virtualization mechanism, and by the VF/VDEV driver during
> > + * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
> > + *
> > + * Errors:
> > + * EINVAL - Invalid mode.
> > + */
> > +int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode
> mode);
> > +
> > +/**
> > + * dlb2_hw_get_virt_mode() - get the device's virtualization mode
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function gets the virtualization mode of the device.
> > + */
> > +enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual port ID.
> > + * @vdev_id: vdev ID.
> > + *
> > + * Return:
> > + * Returns >= 0 upon success, -1 otherwise.
> > + */
> > +s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
> > +                                u32 id,
> > +                                unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: virtual port ID.
> > + * @vdev_id: vdev ID.
> > + *
> > + * Return:
> > + * Returns >= 0 upon success, -1 otherwise.
> > + */
> > +s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
> > +                                u32 id,
> > +                                unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_register_sw_mbox() - register a software mailbox
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vdev_id: vdev ID.
> > + * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF
> communication.
> > + * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev
> communication.
> > + * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
> > + * @inject_arg: user argument for pf2vdev_inject callback.
> > + *
> > + * When Scalable IOV is enabled, the VDCM must register a software mailbox
> for
> > + * every virtual device during vdev creation.
> > + *
> > + * This function notifies the driver to use a software mailbox using the
> > + * provided pointers, instead of the device's hardware mailbox. When the
> driver
> > + * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
> > + * go to the software mailbox instead of the hardware one. This is used in
> > + * Scalable IOV virtualization.
> > + */
> > +void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
> > +                             unsigned int vdev_id,
> > +                             u32 *vdev2pf_mbox,
> > +                             u32 *pf2vdev_mbox,
> > +                             void (*pf2vdev_inject)(void *),
> > +                             void *inject_arg);
> > +
> > +/**
> > + * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vdev_id: vdev ID.
> > + *
> > + * This function notifies the driver to stop using a previously registered
> > + * software mailbox.
> > + */
> > +void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int
> vdev_id);
> > +
> > +/**
> > + * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vdev_id: vdev ID.
> > + * @virt_cq_id: virtual CQ ID.
> > + * @is_ldb: CQ is load-balanced.
> > + * @addr_lo: least-significant 32 bits of address.
> > + * @data: 32 data bits.
> > + *
> > + * This sets up the CQ's IMS entry with the provided address and data values.
> > + * This function should only be called if the device is configured for Scalable
> > + * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
> > + * not needed.
> > + */
> > +void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
> > +                               unsigned int vdev_id,
> > +                               u32 virt_cq_id,
> > +                               bool is_ldb,
> > +                               u32 addr_lo,
> > +                               u32 data);
> > +
> > +/**
> > + * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vdev_id: vdev ID.
> > + * @virt_cq_id: virtual CQ ID.
> > + * @is_ldb: CQ is load-balanced.
> > + *
> > + * This clears the CQ's IMS entry, reverting it to its reset state.
> > + */
> > +void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
> > +                               unsigned int vdev_id,
> > +                               u32 virt_cq_id,
> > +                               bool is_ldb);
> > +
> > +/**
> > + * dlb2_hw_register_pasid() - register a vdev's PASID
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @vdev_id: vdev ID.
> > + * @pasid: the vdev's PASID.
> > + *
> > + * This function stores the user-supplied PASID, and uses it when configuring
> > + * the vdev's CQs.
> > + *
> > + * Return:
> > + * Returns >= 0 upon success, -1 otherwise.
> > + */
> > +int dlb2_hw_register_pasid(struct dlb2_hw *hw,
> > +                          unsigned int vdev_id,
> > +                          unsigned int pasid);
> > +
> > +/**
> > + * dlb2_hw_pending_port_unmaps() - returns the number of unmap
> operations in
> > + *     progress.
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @domain_id: domain ID.
> > + * @args: number of unmaps in progress args
> > + * @resp: response structure.
> > + * @vf_request: indicates whether this request came from a VF.
> > + * @vf_id: If vf_request is true, this contains the VF's ID.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> > + * assigned a detailed error code from enum dlb2_error. If successful, resp-
> >id
> > + * contains the number of unmaps in progress.
> > + *
> > + * Errors:
> > + * EINVAL - Invalid port ID.
> > + */
> > +int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
> > +                               u32 domain_id,
> > +                               struct dlb2_pending_port_unmaps_args *args,
> > +                               struct dlb2_cmd_response *resp,
> > +                               bool vf_request,
> > +                               unsigned int vf_id);
> > +
> > +/**
> > + * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth
> allocated
> > + *     to a port class-of-service.
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @cos_id: class-of-service ID.
> > + *
> > + * Return:
> > + * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
> > + */
> > +int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
> > +
> > +/**
> > + * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for
> a
> > + *     port class-of-service.
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @cos_id: class-of-service ID.
> > + * @bandwidth: class-of-service bandwidth.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
> > + *         cause the total bandwidth across all classes of service to exceed
> > + *         100%.
> > + */
> > +int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8
> bandwidth);
> > +
> > +enum dlb2_wd_tmo {
> > +       /* 40s watchdog timeout */
> > +       DLB2_WD_TMO_40S,
> > +       /* 10s watchdog timeout */
> > +       DLB2_WD_TMO_10S,
> > +       /* 1s watchdog timeout */
> > +       DLB2_WD_TMO_1S,
> > +
> > +       /* Must be last */
> > +       NUM_DLB2_WD_TMOS,
> > +};
> > +
> > +/**
> > + * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
> > + *     caller-specified timeout.
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @tmo: watchdog timeout.
> > + *
> > + * This function should be called during device initialization and after reset.
> > + * The watchdog timer interrupt must also be enabled per-CQ, using either
> > + * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - Invalid timeout.
> > + */
> > +int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo
> tmo);
> > +
> > +/**
> > + * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on
> an
> > + *     individual CQ.
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: port ID.
> > + * @vdev_req: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - Invalid directed port ID.
> > + */
> > +int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
> > +                                u32 id,
> > +                                bool vdev_req,
> > +                                unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on
> an
> > + *     individual CQ.
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @id: port ID.
> > + * @vdev_req: indicates whether this request came from a vdev.
> > + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> > + *
> > + * Return:
> > + * Returns 0 upon success, < 0 otherwise.
> > + *
> > + * Errors:
> > + * EINVAL - Invalid load-balanced port ID.
> > + */
> > +int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
> > +                                u32 id,
> > +                                bool vdev_req,
> > +                                unsigned int vdev_id);
> > +
> > +/**
> > + * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-
> balanced
> > + *     ports.
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function must be called prior to configuring scheduling domains.
> > + */
> > +void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed
> ports.
> > + * @hw: dlb2_hw handle for a particular device.
> > + *
> > + * This function must be called prior to configuring scheduling domains.
> > + */
> > +void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
> > +
> > +/**
> > + * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @weight: 8-entry array of arbiter weights.
> > + *
> > + * weight[N] programs priority N's weight. In cases where the 8 priorities are
> > + * reduced to 4 bins, the mapping is:
> > + * - weight[1] programs bin 0
> > + * - weight[3] programs bin 1
> > + * - weight[5] programs bin 2
> > + * - weight[7] programs bin 3
> > + */
> > +void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
> > +
> > +/**
> > + * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @weight: 8-entry array of arbiter weights.
> > + *
> > + * weight[N] programs priority N's weight. In cases where the 8 priorities are
> > + * reduced to 4 bins, the mapping is:
> > + * - weight[1] programs bin 0
> > + * - weight[3] programs bin 1
> > + * - weight[5] programs bin 2
> > + * - weight[7] programs bin 3
> > + */
> > +void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
> > +
> > +/**
> > + * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @port_id: physical load-balanced port ID.
> > + *
> > + * This function returns whether the load-balanced CQ interrupt is enabled.
> > + */
> > +int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
> > +
> > +/**
> > + * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @port_id: physical load-balanced port ID.
> > + * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
> > + *
> > + * This function can be used to disable (MODE_DIS) and re-enable the
> > + * load-balanced CQ's interrupt. It should only be called after the interrupt
> > + * has been configured with dlb2_configure_ldb_cq_interrupt().
> > + */
> > +void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
> > +                                      int port_id,
> > +                                      int mode);
> > +
> > +/**
> > + * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @port_id: physical load-balanced port ID.
> > + *
> > + * This function returns whether the load-balanced CQ interrupt is enabled.
> > + */
> > +int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
> > +
> > +/**
> > + * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
> > + * @hw: dlb2_hw handle for a particular device.
> > + * @port_id: physical directed port ID.
> > + * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
> > + *
> > + * This function can be used to disable (MODE_DIS) and re-enable the
> > + * directed CQ's interrupt. It should only be called after the interrupt
> > + * has been configured with dlb2_configure_dir_cq_interrupt().
> > + */
> > +void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
> > +                                      int port_id,
> > +                                      int mode);
> > +
> > +#endif /* __DLB2_RESOURCE_H */
> > diff --git a/drivers/event/dlb2/pf/dlb2_main.c
> b/drivers/event/dlb2/pf/dlb2_main.c
> > new file mode 100644
> > index 0000000..bd8590d
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/dlb2_main.c
> > @@ -0,0 +1,615 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#include <stdint.h>
> > +#include <stdbool.h>
> > +#include <stdio.h>
> > +#include <errno.h>
> > +#include <assert.h>
> > +#include <unistd.h>
> > +#include <string.h>
> > +
> > +#include <rte_malloc.h>
> > +#include <rte_errno.h>
> > +
> > +#include "base/dlb2_resource.h"
> > +#include "base/dlb2_osdep.h"
> > +#include "base/dlb2_regs.h"
> > +#include "dlb2_main.h"
> > +#include "../dlb2_user.h"
> > +#include "../dlb2_priv.h"
> > +#include "../dlb2_iface.h"
> > +#include "../dlb2_inline_fns.h"
> > +
> > +#define PF_ID_ZERO 0   /* PF ONLY! */
> > +#define NO_OWNER_VF 0  /* PF ONLY! */
> > +#define NOT_VF_REQ false /* PF ONLY! */
> > +
> > +#define DLB2_PCI_CFG_SPACE_SIZE 256
> > +#define DLB2_PCI_CAP_POINTER 0x34
> > +#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
> > +#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
> > +#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
> > +#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
> > +#define DLB2_PCI_EXT_CAP_ID_ERR 1
> > +#define DLB2_PCI_ERR_UNCOR_MASK 8
> > +#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
> > +
> > +#define DLB2_PCI_EXP_DEVCTL 8
> > +#define DLB2_PCI_LNKCTL 16
> > +#define DLB2_PCI_SLTCTL 24
> > +#define DLB2_PCI_RTCTL 28
> > +#define DLB2_PCI_EXP_DEVCTL2 40
> > +#define DLB2_PCI_LNKCTL2 48
> > +#define DLB2_PCI_SLTCTL2 56
> > +#define DLB2_PCI_CMD 4
> > +#define DLB2_PCI_X_CMD 2
> > +#define DLB2_PCI_EXP_DEVSTA 10
> > +#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
> > +#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
> > +
> > +#define DLB2_PCI_CAP_ID_EXP       0x10
> > +#define DLB2_PCI_CAP_ID_MSIX      0x11
> > +#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
> > +#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
> > +#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
> > +
> > +#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
> > +#define DLB2_PCI_PRI_ALLOC_REQ           0xC
> > +#define DLB2_PCI_PRI_CTRL                0x4
> > +#define DLB2_PCI_MSIX_FLAGS              0x2
> > +#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
> > +#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
> > +#define DLB2_PCI_ERR_ROOT_STATUS         0x30
> > +#define DLB2_PCI_ERR_COR_STATUS          0x10
> > +#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
> > +#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
> > +#define DLB2_PCI_ACS_CAP                 0x4
> > +#define DLB2_PCI_ACS_CTRL                0x6
> > +#define DLB2_PCI_ACS_SV                  0x1
> > +#define DLB2_PCI_ACS_RR                  0x4
> > +#define DLB2_PCI_ACS_CR                  0x8
> > +#define DLB2_PCI_ACS_UF                  0x10
> > +#define DLB2_PCI_ACS_EC                  0x20
> > +
> > +static int
> > +dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
> > +{
> > +       uint32_t hdr;
> > +       size_t sz;
> > +       int pos;
> > +
> > +       pos = DLB2_PCI_CFG_SPACE_SIZE;
> > +       sz = sizeof(hdr);
> > +
> > +       while (pos > 0xFF) {
> > +               if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
> > +                       return -1;
> > +
> > +               if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
> > +                       return pos;
> > +
> > +               pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
> > +       }
> > +
> > +       return -1;
> > +}
> > +
> > +static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
> > +{
> > +       uint8_t pos;
> > +       int ret;
> > +       uint16_t hdr;
> > +
> > +       ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
> > +       pos &= 0xFC;
> > +
> > +       if (ret != 1)
> > +               return -1;
> > +
> > +       while (pos > 0x3F) {
> > +               ret = rte_pci_read_config(pdev, &hdr, 2, pos);
> > +               if (ret != 2)
> > +                       return -1;
> > +
> > +               if (DLB2_PCI_CAP_ID(hdr) == id)
> > +                       return pos;
> > +
> > +               if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
> > +                       return -1;
> > +
> > +               pos = DLB2_PCI_CAP_NEXT(hdr);
> > +       }
> > +
> > +       return -1;
> > +}
> > +
> > +static int
> > +dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
> > +{
> > +       int i;
> > +
> > +       if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIR64B))
> > +               dlb2_dev->enqueue_four = dlb2_movdir64b;
> > +       else
> > +               dlb2_dev->enqueue_four = dlb2_movntdq;
> > +
> > +       /* Initialize software state */
> > +       for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++)
> > +               dlb2_list_init_head(&dlb2_dev->ldb_port_pages[i].list);
> > +
> > +       for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++)
> > +               dlb2_list_init_head(&dlb2_dev->dir_port_pages[i].list);
> > +
> > +       rte_spinlock_init(&dlb2_dev->resource_mutex);
> > +       rte_spinlock_init(&dlb2_dev->measurement_lock);
> > +
> > +       return 0;
> > +}
> > +
> > +static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
> > +{
> > +       dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
> > +}
> > +
> > +#define DLB2_READY_RETRY_LIMIT 1000
> > +static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
> > +{
> > +       u32 retries = 0;
> > +
> > +       /* Allow at least 1s for the device to become active after power-on */
> > +       for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
> > +               union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
> > +               union dlb2_cfg_mstr_cfg_pm_status pm_st;
> > +               u32 addr;
> > +
> > +               addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
> > +               pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
> > +               addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
> > +               idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
> > +               if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
> > +                       break;
> > +
> > +               rte_delay_ms(1);
> > +       };
> > +
> > +       if (retries == DLB2_READY_RETRY_LIMIT) {
> > +               printf("[%s()] wait for device ready timed out\n",
> > +                      __func__);
> > +               return -1;
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +struct dlb2_dev *
> > +dlb2_probe(struct rte_pci_device *pdev)
> > +{
> > +       struct dlb2_dev *dlb2_dev;
> > +       int ret = 0;
> > +
> > +       DLB2_INFO(dlb2_dev, "probe\n");
> > +
> > +       dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
> > +                             RTE_CACHE_LINE_SIZE);
> > +
> > +       if (dlb2_dev == NULL) {
> > +               ret = -ENOMEM;
> > +               goto dlb2_dev_malloc_fail;
> > +       }
> > +
> > +       /* PCI Bus driver has already mapped bar space into process.
> > +        * Save off our IO register and FUNC addresses.
> > +        */
> > +
> > +       /* BAR 0 */
> > +       if (pdev->mem_resource[0].addr == NULL) {
> > +               DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
> > +               ret = -EINVAL;
> > +               goto pci_mmap_bad_addr;
> > +       }
> > +       dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev-
> >mem_resource[0].addr;
> > +       dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
> > +
> > +       DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
> > +                 (void *)dlb2_dev->hw.func_kva,
> > +                 (void *)dlb2_dev->hw.func_phys_addr,
> > +                 (void *)(pdev->mem_resource[0].len));
> > +
> > +       /* BAR 2 */
> > +       if (pdev->mem_resource[2].addr == NULL) {
> > +               DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
> > +               ret = -EINVAL;
> > +               goto pci_mmap_bad_addr;
> > +       }
> > +       dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
> > +       dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
> > +
> > +       DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
> > +                 (void *)dlb2_dev->hw.csr_kva,
> > +                 (void *)dlb2_dev->hw.csr_phys_addr,
> > +                 (void *)(pdev->mem_resource[2].len));
> > +
> > +       dlb2_dev->pdev = pdev;
> > +
> > +       /* PM enable must be done before any other MMIO accesses, and this
> > +        * setting is persistent across device reset.
> > +        */
> > +       dlb2_pf_enable_pm(dlb2_dev);
> > +
> > +       ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
> > +       if (ret)
> > +               goto wait_for_device_ready_fail;
> > +
> > +       ret = dlb2_pf_reset(dlb2_dev);
> > +       if (ret)
> > +               goto dlb2_reset_fail;
> > +
> > +       ret = dlb2_pf_init_driver_state(dlb2_dev);
> > +       if (ret)
> > +               goto init_driver_state_fail;
> > +
> > +       ret = dlb2_resource_init(&dlb2_dev->hw);
> > +       if (ret)
> > +               goto resource_init_fail;
> > +
> > +       return dlb2_dev;
> > +
> > +resource_init_fail:
> > +       dlb2_resource_free(&dlb2_dev->hw);
> > +init_driver_state_fail:
> > +dlb2_reset_fail:
> > +pci_mmap_bad_addr:
> > +wait_for_device_ready_fail:
> > +       rte_free(dlb2_dev);
> > +dlb2_dev_malloc_fail:
> > +       rte_errno = ret;
> > +       return NULL;
> > +}
> > +
> > +int
> > +dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
> > +{
> > +       int ret = 0;
> > +       int i = 0;
> > +       uint32_t dword[16];
> > +       uint16_t cmd;
> > +       off_t off;
> > +
> > +       uint16_t dev_ctl_word;
> > +       uint16_t dev_ctl2_word;
> > +       uint16_t lnk_word;
> > +       uint16_t lnk_word2;
> > +       uint16_t slt_word;
> > +       uint16_t slt_word2;
> > +       uint16_t rt_ctl_word;
> > +       uint32_t pri_reqs_dword;
> > +       uint16_t pri_ctrl_word;
> > +
> > +       int pcie_cap_offset;
> > +       int pri_cap_offset;
> > +       int msix_cap_offset;
> > +       int err_cap_offset;
> > +       int acs_cap_offset;
> > +       int wait_count;
> > +
> > +       uint16_t devsta_busy_word;
> > +       uint16_t devctl_word;
> > +
> > +       struct rte_pci_device *pdev = dlb2_dev->pdev;
> > +
> > +       /* Save PCI config state */
> > +
> > +       for (i = 0; i < 16; i++) {
> > +               if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
> > +                       return ret;
> > +       }
> > +
> > +       pcie_cap_offset = dlb2_pci_find_capability(pdev,
> DLB2_PCI_CAP_ID_EXP);
> > +
> > +       if (pcie_cap_offset < 0) {
> > +               printf("[%s()] failed to find the pcie capability\n",
> > +                      __func__);
> > +               return pcie_cap_offset;
> > +       }
> > +
> > +       off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> > +       if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
> > +               dev_ctl_word = 0;
> > +
> > +       off = pcie_cap_offset + DLB2_PCI_LNKCTL;
> > +       if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
> > +               lnk_word = 0;
> > +
> > +       off = pcie_cap_offset + DLB2_PCI_SLTCTL;
> > +       if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
> > +               slt_word = 0;
> > +
> > +       off = pcie_cap_offset + DLB2_PCI_RTCTL;
> > +       if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
> > +               rt_ctl_word = 0;
> > +
> > +       off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
> > +       if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
> > +               dev_ctl2_word = 0;
> > +
> > +       off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
> > +       if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
> > +               lnk_word2 = 0;
> > +
> > +       off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
> > +       if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
> > +               slt_word2 = 0;
> > +
> > +       off = DLB2_PCI_EXT_CAP_ID_PRI;
> > +       pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> > +
> > +       if (pri_cap_offset >= 0) {
> > +               off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
> > +               if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
> > +                       pri_reqs_dword = 0;
> > +       }
> > +
> > +       /* clear the PCI command register before issuing the FLR */
> > +
> > +       off = DLB2_PCI_CMD;
> > +       cmd = 0;
> > +       if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> > +               printf("[%s()] failed to write the pci command\n",
> > +                      __func__);
> > +               return ret;
> > +       }
> > +
> > +       /* issue the FLR */
> > +       for (wait_count = 0; wait_count < 4; wait_count++) {
> > +               int sleep_time;
> > +
> > +               off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
> > +               ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
> > +               if (ret != 2) {
> > +                       printf("[%s()] failed to read the pci device status\n",
> > +                              __func__);
> > +                       return ret;
> > +               }
> > +
> > +               if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
> > +                       break;
> > +
> > +               sleep_time = (1 << (wait_count)) * 100;
> > +               rte_delay_ms(sleep_time);
> > +       }
> > +
> > +       if (wait_count == 4) {
> > +               printf("[%s()] wait for pci pending transactions timed out\n",
> > +                      __func__);
> > +               return -1;
> > +       }
> > +
> > +       off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> > +       ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
> > +       if (ret != 2) {
> > +               printf("[%s()] failed to read the pcie device control\n",
> > +                      __func__);
> > +               return ret;
> > +       }
> > +
> > +       devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
> > +
> > +       ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
> > +       if (ret != 2) {
> > +               printf("[%s()] failed to write the pcie device control\n",
> > +                      __func__);
> > +               return ret;
> > +       }
> > +
> > +       rte_delay_ms(100);
> > +
> > +       /* Restore PCI config state */
> > +
> > +       if (pcie_cap_offset >= 0) {
> > +               off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> > +               ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
> > +               if (ret != 2) {
> > +                       printf("[%s()] failed to write the pcie device control at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +
> > +               off = pcie_cap_offset + DLB2_PCI_LNKCTL;
> > +               ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
> > +               if (ret != 2) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +
> > +               off = pcie_cap_offset + DLB2_PCI_SLTCTL;
> > +               ret = rte_pci_write_config(pdev, &slt_word, 2, off);
> > +               if (ret != 2) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +
> > +               off = pcie_cap_offset + DLB2_PCI_RTCTL;
> > +               ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
> > +               if (ret != 2) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +
> > +               off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
> > +               ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
> > +               if (ret != 2) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +
> > +               off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
> > +               ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
> > +               if (ret != 2) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +
> > +               off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
> > +               ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
> > +               if (ret != 2) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +       }
> > +
> > +       if (pri_cap_offset >= 0) {
> > +               pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
> > +
> > +               off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
> > +               ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
> > +               if (ret != 4) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +
> > +               off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
> > +               ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
> > +               if (ret != 2) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +       }
> > +
> > +       off = DLB2_PCI_EXT_CAP_ID_ERR;
> > +       err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> > +
> > +       if (err_cap_offset >= 0) {
> > +               uint32_t tmp;
> > +
> > +               off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
> > +               if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> > +                       tmp = 0;
> > +
> > +               ret = rte_pci_write_config(pdev, &tmp, 4, off);
> > +               if (ret != 4) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +
> > +               off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
> > +               if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> > +                       tmp = 0;
> > +
> > +               ret = rte_pci_write_config(pdev, &tmp, 4, off);
> > +               if (ret != 4) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +
> > +               off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
> > +               if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> > +                       tmp = 0;
> > +
> > +               ret = rte_pci_write_config(pdev, &tmp, 4, off);
> > +               if (ret != 4) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +       }
> > +
> > +       for (i = 16; i > 0; i--) {
> > +               off = (i - 1) * 4;
> > +               ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
> > +               if (ret != 4) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +       }
> > +
> > +       off = DLB2_PCI_CMD;
> > +       if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> > +               cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
> > +               if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> > +                       printf("[%s()] failed to write the pci command\n",
> > +                              __func__);
> > +                       return ret;
> > +               }
> > +       }
> > +
> > +       msix_cap_offset = dlb2_pci_find_capability(pdev,
> > +                                                  DLB2_PCI_CAP_ID_MSIX);
> > +       if (msix_cap_offset >= 0) {
> > +               off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
> > +               if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> > +                       cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
> > +                       cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
> > +                       if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> > +                               printf("[%s()] failed to write msix flags\n",
> > +                                      __func__);
> > +                               return ret;
> > +                       }
> > +               }
> > +
> > +               off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
> > +               if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> > +                       cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
> > +                       if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> > +                               printf("[%s()] failed to write msix flags\n",
> > +                                      __func__);
> > +                               return ret;
> > +                       }
> > +               }
> > +       }
> > +
> > +       off = DLB2_PCI_EXT_CAP_ID_ACS;
> > +       acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> > +
> > +       if (acs_cap_offset >= 0) {
> > +               uint16_t acs_cap, acs_ctrl, acs_mask;
> > +               off = acs_cap_offset + DLB2_PCI_ACS_CAP;
> > +               if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
> > +                       acs_cap = 0;
> > +
> > +               off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> > +               if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
> > +                       acs_ctrl = 0;
> > +
> > +               acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
> > +               acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
> > +               acs_ctrl |= (acs_cap & acs_mask);
> > +
> > +               ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
> > +               if (ret != 2) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +
> > +               off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> > +               if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
> > +                       acs_ctrl = 0;
> > +
> > +               acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
> > +               acs_mask |= DLB2_PCI_ACS_EC;
> > +               acs_ctrl &= ~acs_mask;
> > +
> > +               off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> > +               ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
> > +               if (ret != 2) {
> > +                       printf("[%s()] failed to write the pcie config space at offset
> %d\n",
> > +                               __func__, (int)off);
> > +                       return ret;
> > +               }
> > +       }
> > +
> > +       return 0;
> > +}
> > diff --git a/drivers/event/dlb2/pf/dlb2_main.h
> b/drivers/event/dlb2/pf/dlb2_main.h
> > new file mode 100644
> > index 0000000..ec96f11
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/dlb2_main.h
> > @@ -0,0 +1,106 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#ifndef __DLB2_MAIN_H
> > +#define __DLB2_MAIN_H
> > +
> > +#include <rte_debug.h>
> > +#include <rte_log.h>
> > +#include <rte_spinlock.h>
> > +#include <rte_pci.h>
> > +#include <rte_bus_pci.h>
> > +
> > +#ifndef PAGE_SIZE
> > +#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
> > +#endif
> > +
> > +#include "base/dlb2_hw_types.h"
> > +#include "../dlb2_user.h"
> > +
> > +#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
> > +
> > +struct dlb2_dev;
> > +
> > +struct dlb2_port_memory {
> > +       struct dlb2_list_head list;
> > +       void *cq_base;
> > +       bool valid;
> > +};
> > +
> > +struct dlb2_dev {
> > +       struct rte_pci_device *pdev;
> > +       struct dlb2_hw hw;
> > +       /* struct list_head list; */
> > +       struct device *dlb2_device;
> > +       struct dlb2_port_memory
> ldb_port_pages[DLB2_MAX_NUM_LDB_PORTS];
> > +       struct dlb2_port_memory
> dir_port_pages[DLB2_MAX_NUM_DIR_PORTS];
> > +       /* The enqueue_four function enqueues four HCWs (one cache-line
> worth)
> > +        * to the DLB2, using whichever mechanism is supported by the platform
> > +        * on which this driver is running.
> > +        */
> > +       void (*enqueue_four)(void *qe4, void *pp_addr);
> > +
> > +       bool domain_reset_failed;
> > +       /* The resource mutex serializes access to driver data structures and
> > +        * hardware registers.
> > +        */
> > +       rte_spinlock_t resource_mutex;
> > +       rte_spinlock_t measurement_lock;
> > +       bool worker_launched;
> > +       u8 revision;
> > +};
> > +
> > +struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
> > +
> > +int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
> > +int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
> > +                               struct dlb2_create_sched_domain_args *args,
> > +                               struct dlb2_cmd_response *resp);
> > +int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
> > +                            u32 domain_id,
> > +                            struct dlb2_create_ldb_queue_args *args,
> > +                            struct dlb2_cmd_response *resp);
> > +int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
> > +                            u32 domain_id,
> > +                            struct dlb2_create_dir_queue_args *args,
> > +                            struct dlb2_cmd_response *resp);
> > +int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
> > +                           u32 domain_id,
> > +                           struct dlb2_create_ldb_port_args *args,
> > +                           uintptr_t cq_dma_base,
> > +                           struct dlb2_cmd_response *resp);
> > +int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
> > +                           u32 domain_id,
> > +                           struct dlb2_create_dir_port_args *args,
> > +                           uintptr_t cq_dma_base,
> > +                           struct dlb2_cmd_response *resp);
> > +int dlb2_pf_start_domain(struct dlb2_hw *hw,
> > +                        u32 domain_id,
> > +                        struct dlb2_start_domain_args *args,
> > +                        struct dlb2_cmd_response *resp);
> > +int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
> > +                           u32 domain_id,
> > +                           struct dlb2_enable_ldb_port_args *args,
> > +                           struct dlb2_cmd_response *resp);
> > +int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
> > +                            u32 domain_id,
> > +                            struct dlb2_disable_ldb_port_args *args,
> > +                            struct dlb2_cmd_response *resp);
> > +int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
> > +                           u32 domain_id,
> > +                           struct dlb2_enable_dir_port_args *args,
> > +                           struct dlb2_cmd_response *resp);
> > +int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
> > +                            u32 domain_id,
> > +                            struct dlb2_disable_dir_port_args *args,
> > +                            struct dlb2_cmd_response *resp);
> > +int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
> > +int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
> > +                                    u32 domain_id,
> > +                                    u32 port_id);
> > +int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
> > +                                    u32 domain_id,
> > +                                    u32 port_id);
> > +
> > +#endif /* __DLB2_MAIN_H */
> > diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
> > new file mode 100644
> > index 0000000..7d64309
> > --- /dev/null
> > +++ b/drivers/event/dlb2/pf/dlb2_pf.c
> > @@ -0,0 +1,244 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2016-2020 Intel Corporation
> > + */
> > +
> > +#include <stdint.h>
> > +#include <stdbool.h>
> > +#include <stdio.h>
> > +#include <sys/mman.h>
> > +#include <sys/fcntl.h>
> > +#include <sys/time.h>
> > +#include <errno.h>
> > +#include <assert.h>
> > +#include <unistd.h>
> > +#include <string.h>
> > +#include <rte_debug.h>
> > +#include <rte_log.h>
> > +#include <rte_dev.h>
> > +#include <rte_devargs.h>
> > +#include <rte_mbuf.h>
> > +#include <rte_ring.h>
> > +#include <rte_errno.h>
> > +#include <rte_kvargs.h>
> > +#include <rte_malloc.h>
> > +#include <rte_cycles.h>
> > +#include <rte_io.h>
> > +#include <rte_pci.h>
> > +#include <rte_bus_pci.h>
> > +#include <rte_eventdev.h>
> > +#include <rte_eventdev_pmd.h>
> > +#include <rte_eventdev_pmd_pci.h>
> > +#include <rte_memory.h>
> > +#include <rte_string_fns.h>
> > +
> > +#include "../dlb2_priv.h"
> > +#include "../dlb2_iface.h"
> > +#include "../dlb2_inline_fns.h"
> > +#include "dlb2_main.h"
> > +#include "base/dlb2_hw_types.h"
> > +#include "base/dlb2_osdep.h"
> > +#include "base/dlb2_resource.h"
> > +
> > +static const char *event_dlb2_pf_name =
> RTE_STR(EVDEV_DLB2_NAME_PMD);
> > +
> > +static void
> > +dlb2_pf_low_level_io_init(void)
> > +{
> > +       int i;
> > +       /* Addresses will be initialized at port create */
> > +       for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
> > +               /* First directed ports */
> > +               dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
> > +               dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
> > +               dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
> > +
> > +               /* Now load balanced ports */
> > +               dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
> > +               dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
> > +               dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
> > +       }
> > +}
> > +
> > +static int
> > +dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
> > +{
> > +       RTE_SET_USED(handle);
> > +       RTE_SET_USED(name);
> > +
> > +       return 0;
> > +}
> > +
> > +static int
> > +dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
> > +                          uint8_t *revision)
> > +{
> > +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> > +
> > +       *revision = dlb2_dev->revision;
> > +
> > +       return 0;
> > +}
> > +
> > +static void
> > +dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
> > +{
> > +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> > +
> > +       dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
> > +       dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
> > +}
> > +
> > +static int
> > +dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
> > +                         struct dlb2_get_num_resources_args *rsrcs)
> > +{
> > +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> > +
> > +       return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
> > +}
> > +
> > +static int
> > +dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
> > +                        enum dlb2_cq_poll_modes *mode)
> > +{
> > +       RTE_SET_USED(handle);
> > +
> > +       *mode = DLB2_CQ_POLL_MODE_SPARSE;
> > +
> > +       return 0;
> > +}
> > +
> > +static void
> > +dlb2_pf_iface_fn_ptrs_init(void)
> > +{
> > +       dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
> > +       dlb2_iface_open = dlb2_pf_open;
> > +       dlb2_iface_get_device_version = dlb2_pf_get_device_version;
> > +       dlb2_iface_hardware_init = dlb2_pf_hardware_init;
> > +       dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
> > +       dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
> > +}
> > +
> > +/* PCI DEV HOOKS */
> > +static int
> > +dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
> > +{
> > +       int ret = 0;
> > +       struct rte_pci_device *pci_dev;
> > +       struct dlb2_devargs dlb2_args = {
> > +               .socket_id = rte_socket_id(),
> > +               .max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
> > +               .num_dir_credits_override = -1,
> > +               .qid_depth_thresholds = { {0} },
> > +               .cos_id = DLB2_COS_DEFAULT
> > +       };
> > +       struct dlb2_eventdev *dlb2;
> > +
> > +       DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
> > +                    eventdev->data->dev_id, eventdev->data->socket_id);
> > +
> > +       dlb2_pf_iface_fn_ptrs_init();
> > +
> > +       pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
> > +
> > +       if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> > +               dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
> > +
> > +               /* Probe the DLB2 PF layer */
> > +               dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
> > +
> > +               if (dlb2->qm_instance.pf_dev == NULL) {
> > +                       DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
> > +                                    rte_errno);
> > +                       ret = -rte_errno;
> > +                       goto dlb2_probe_failed;
> > +               }
> > +
> > +               /* Were we invoked with runtime parameters? */
> > +               if (pci_dev->device.devargs) {
> > +                       ret = dlb2_parse_params(pci_dev->device.devargs->args,
> > +                                               pci_dev->device.devargs->name,
> > +                                               &dlb2_args);
> > +                       if (ret) {
> > +                               DLB2_LOG_ERR("PFPMD failed to parse args ret=%d,
> errno=%d\n",
> > +                                            ret, rte_errno);
> > +                               goto dlb2_probe_failed;
> > +                       }
> > +               }
> > +
> > +               ret = dlb2_primary_eventdev_probe(eventdev,
> > +                                                 event_dlb2_pf_name,
> > +                                                 &dlb2_args);
> > +       } else {
> > +               ret = dlb2_secondary_eventdev_probe(eventdev,
> > +                                                   event_dlb2_pf_name);
> > +       }
> > +       if (ret)
> > +               goto dlb2_probe_failed;
> > +
> > +       DLB2_LOG_INFO("DLB2 PF Probe success\n");
> > +
> > +       return 0;
> > +
> > +dlb2_probe_failed:
> > +
> > +       DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
> > +
> > +       return ret;
> > +}
> > +
> > +#define EVENTDEV_INTEL_VENDOR_ID 0x8086
> > +
> > +static const struct rte_pci_id pci_id_dlb2_map[] = {
> > +       {
> > +               RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
> > +                              PCI_DEVICE_ID_INTEL_DLB2_PF)
> > +       },
> > +       {
> > +               .vendor_id = 0,
> > +       },
> > +};
> > +
> > +static int
> > +event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
> > +                    struct rte_pci_device *pci_dev)
> > +{
> > +       int ret;
> > +
> > +       ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
> > +                                            sizeof(struct dlb2_eventdev),
> > +                                            dlb2_eventdev_pci_init,
> > +                                            event_dlb2_pf_name);
> > +       if (ret) {
> > +               DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
> > +                               "ret=%d\n", ret);
> > +       }
> > +
> > +       return ret;
> > +}
> > +
> > +static int
> > +event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
> > +{
> > +       int ret;
> > +
> > +       ret = rte_event_pmd_pci_remove(pci_dev, NULL);
> > +
> > +       if (ret) {
> > +               DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
> > +                               "ret=%d\n", ret);
> > +       }
> > +
> > +       return ret;
> > +
> > +}
> > +
> > +static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
> > +       .id_table = pci_id_dlb2_map,
> > +       .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
> > +       .probe = event_dlb2_pci_probe,
> > +       .remove = event_dlb2_pci_remove,
> > +};
> > +
> > +RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
> > +RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
> > --
> > 2.6.4
> >

Will fix in next drop.

[dpdk-dev] [PATCH v2 07/22] event/dlb2: add xstats

[Timothy McDaniel]

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c        |   35 +-
 drivers/event/dlb2/dlb2_xstats.c | 1240 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build   |    1 +
 3 files changed, 1273 insertions(+), 3 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 26985b9..a7f8f68 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -74,6 +74,21 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/*
+ * DUMMY - added so that xstats path will compile/link.
+ * Will be replaced by real version in a subsequent
+ * patch.
+ */
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	RTE_SET_USED(dlb2);
+	RTE_SET_USED(queue);
+
+	return 0;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -341,9 +356,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
-	RTE_SET_USED(dev);
-
-	/* Eventdev PMD entry points */
+	/* Expose PMD's eventdev interface */
+	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dump             = dlb2_eventdev_dump,
+		.xstats_get       = dlb2_eventdev_xstats_get,
+		.xstats_get_names = dlb2_eventdev_xstats_get_names,
+		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
+		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+	};
+
+	dev->dev_ops = &dlb2_eventdev_entry_ops;
 }
 
 int
@@ -395,6 +417,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Complete xtstats runtime initialization */
+	err = dlb2_xstats_init(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
+		return err;
+	}
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
new file mode 100644
index 0000000..21391f7
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_xstats.c
@@ -0,0 +1,1240 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eventdev.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+enum dlb2_xstats_type {
+	/* common to device and port */
+	rx_ok,				/**< Receive an event */
+	rx_drop,                        /**< Error bit set in received QE */
+	rx_interrupt_wait,		/**< Wait on an interrupt */
+	rx_umonitor_umwait,		/**< Block using umwait */
+	tx_ok,				/**< Transmit an event */
+	total_polls,			/**< Call dequeue_burst */
+	zero_polls,			/**< Call dequeue burst and return 0 */
+	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
+	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
+	tx_nospc_inflight_max,		/**< Reach the new_event_threshold */
+	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
+	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
+	/* device specific */
+	nb_events_limit,
+	inflight_events,
+	ldb_pool_size,
+	dir_pool_size,
+	/* port specific */
+	tx_new,				/**< Send an OP_NEW event */
+	tx_fwd,				/**< Send an OP_FORWARD event */
+	tx_rel,				/**< Send an OP_RELEASE event */
+	tx_implicit_rel,		/**< Issue an implicit event release */
+	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED event */
+	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL event */
+	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event */
+	tx_sched_directed,		/**< Send a directed event */
+	tx_invalid,                     /**< Send an event with an invalid op */
+	outstanding_releases,		/**< # of releases a port owes */
+	max_outstanding_releases,	/**< max # of releases a port can owe */
+	rx_sched_ordered,		/**< Dequeue an ordered event */
+	rx_sched_unordered,		/**< Dequeue an unordered event */
+	rx_sched_atomic,		/**< Dequeue an atomic event */
+	rx_sched_directed,		/**< Dequeue an directed event */
+	rx_sched_invalid,               /**< Dequeue event sched type invalid */
+	/* common to port and queue */
+	is_configured,			/**< Port is configured */
+	is_load_balanced,		/**< Port is LDB */
+	hw_id,				/**< Hardware ID */
+	/* queue specific */
+	num_links,			/**< Number of ports linked */
+	sched_type,			/**< Queue sched type */
+	enq_ok,				/**< # events enqueued to the queue */
+	current_depth,			/**< Current queue depth */
+	depth_threshold,		/**< Programmed depth threshold */
+	depth_le50_threshold,
+	/**< Depth LE to 50% of the configured hardware threshold */
+	depth_gt50_le75_threshold,
+	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold */
+	depth_gt75_le100_threshold,
+	/**< Depth GT 75%. but LE to the configured hardware threshold */
+	depth_gt100_threshold
+	/**< Depth GT 100% of the configured hw threshold */
+};
+
+typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
+		uint16_t obj_idx, /* port or queue id */
+		enum dlb2_xstats_type stat, int extra_arg);
+
+enum dlb2_xstats_fn_type {
+	DLB2_XSTATS_FN_DEV,
+	DLB2_XSTATS_FN_PORT,
+	DLB2_XSTATS_FN_QUEUE
+};
+
+struct dlb2_xstats_entry {
+	struct rte_event_dev_xstats_name name;
+	uint64_t reset_value; /* an offset to be taken away to emulate resets */
+	enum dlb2_xstats_fn_type fn_id;
+	enum dlb2_xstats_type stat;
+	enum rte_event_dev_xstats_mode mode;
+	int extra_arg;
+	uint16_t obj_idx;
+	uint8_t reset_allowed; /* when set, this value can be reset */
+};
+
+/* Some device stats are simply a summation of the corresponding port values */
+static uint64_t
+dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
+			     int which_stat)
+{
+	int i;
+	uint64_t val = 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
+
+		if (!port->setup_done)
+			continue;
+
+		switch (which_stat) {
+		case rx_ok:
+			val += port->stats.traffic.rx_ok;
+			break;
+		case rx_drop:
+			val += port->stats.traffic.rx_drop;
+			break;
+		case rx_interrupt_wait:
+			val += port->stats.traffic.rx_interrupt_wait;
+			break;
+		case rx_umonitor_umwait:
+			val += port->stats.traffic.rx_umonitor_umwait;
+			break;
+		case tx_ok:
+			val += port->stats.traffic.tx_ok;
+			break;
+		case total_polls:
+			val += port->stats.traffic.total_polls;
+			break;
+		case zero_polls:
+			val += port->stats.traffic.zero_polls;
+			break;
+		case tx_nospc_ldb_hw_credits:
+			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
+			break;
+		case tx_nospc_dir_hw_credits:
+			val += port->stats.traffic.tx_nospc_dir_hw_credits;
+			break;
+		case tx_nospc_inflight_max:
+			val += port->stats.traffic.tx_nospc_inflight_max;
+			break;
+		case tx_nospc_new_event_limit:
+			val += port->stats.traffic.tx_nospc_new_event_limit;
+			break;
+		case tx_nospc_inflight_credits:
+			val += port->stats.traffic.tx_nospc_inflight_credits;
+			break;
+		default:
+			return -1;
+		}
+	}
+	return val;
+}
+
+static uint64_t
+get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
+	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	switch (type) {
+	case rx_ok:
+	case rx_drop:
+	case rx_interrupt_wait:
+	case rx_umonitor_umwait:
+	case tx_ok:
+	case total_polls:
+	case zero_polls:
+	case tx_nospc_ldb_hw_credits:
+	case tx_nospc_dir_hw_credits:
+	case tx_nospc_inflight_max:
+	case tx_nospc_new_event_limit:
+	case tx_nospc_inflight_credits:
+		return dlb2_device_traffic_stat_get(dlb2, type);
+	case nb_events_limit:
+		return dlb2->new_event_limit;
+	case inflight_events:
+		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
+	case ldb_pool_size:
+		return dlb2->num_ldb_credits;
+	case dir_pool_size:
+		return dlb2->num_dir_credits;
+	default: return -1;
+	}
+}
+
+static uint64_t
+get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
+
+	switch (type) {
+	case rx_ok: return ev_port->stats.traffic.rx_ok;
+
+	case rx_drop: return ev_port->stats.traffic.rx_drop;
+
+	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
+
+	case rx_umonitor_umwait:
+		return ev_port->stats.traffic.rx_umonitor_umwait;
+
+	case tx_ok: return ev_port->stats.traffic.tx_ok;
+
+	case total_polls: return ev_port->stats.traffic.total_polls;
+
+	case zero_polls: return ev_port->stats.traffic.zero_polls;
+
+	case tx_nospc_ldb_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
+
+	case tx_nospc_dir_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
+
+	case tx_nospc_inflight_max:
+		return ev_port->stats.traffic.tx_nospc_inflight_max;
+
+	case tx_nospc_new_event_limit:
+		return ev_port->stats.traffic.tx_nospc_new_event_limit;
+
+	case tx_nospc_inflight_credits:
+		return ev_port->stats.traffic.tx_nospc_inflight_credits;
+
+	case is_configured: return ev_port->setup_done;
+
+	case is_load_balanced: return !ev_port->qm_port.is_directed;
+
+	case hw_id: return ev_port->qm_port.id;
+
+	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
+
+	case tx_fwd: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
+
+	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
+
+	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
+
+	case tx_sched_ordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case tx_sched_unordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case tx_sched_atomic:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case tx_sched_directed:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case tx_invalid: return ev_port->stats.tx_invalid;
+
+	case outstanding_releases: return ev_port->outstanding_releases;
+
+	case max_outstanding_releases:
+		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	case rx_sched_ordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case rx_sched_unordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case rx_sched_atomic:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case rx_sched_directed:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
+
+	default: return -1;
+	}
+}
+
+static uint64_t
+dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
+{
+	int port = 0;
+	uint64_t tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
+
+	return tally;
+}
+
+static uint64_t
+dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
+{
+	int port = 0;
+	uint64_t enq_ok_tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
+
+	return enq_ok_tally;
+}
+
+static uint64_t
+get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_queue *ev_queue =
+		&dlb2->ev_queues[obj_idx];
+
+	switch (type) {
+	case is_configured: return ev_queue->setup_done;
+
+	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
+
+	case hw_id: return ev_queue->qm_queue.id;
+
+	case num_links: return ev_queue->num_links;
+
+	case sched_type: return ev_queue->qm_queue.sched_type;
+
+	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
+
+	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
+
+	case depth_threshold: return ev_queue->depth_threshold;
+
+	case depth_le50_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_LE50);
+
+	case depth_gt50_le75_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT50_LE75);
+
+	case depth_gt75_le100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT75_LE100);
+
+	case depth_gt100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT100);
+
+	default: return -1;
+	}
+}
+
+int
+dlb2_xstats_init(struct dlb2_eventdev *dlb2)
+{
+	/*
+	 * define the stats names and types. Used to build up the device
+	 * xstats array
+	 * There are multiple set of stats:
+	 *   - device-level,
+	 *   - per-port,
+	 *   - per-qid,
+	 *
+	 * For each of these sets, we have three parallel arrays, one for the
+	 * names, the other for the stat type parameter to be passed in the fn
+	 * call to get that stat. The third array allows resetting or not.
+	 * All these arrays must be kept in sync
+	 */
+	static const char * const dev_stats[] = {
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"nb_events_limit",
+		"inflight_events",
+		"ldb_pool_size",
+		"dir_pool_size",
+	};
+	static const enum dlb2_xstats_type dev_types[] = {
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		nb_events_limit,
+		inflight_events,
+		ldb_pool_size,
+		dir_pool_size,
+	};
+	/* Note: generated device stats are not allowed to be reset. */
+	static const uint8_t dev_reset_allowed[] = {
+		0, /* rx_ok */
+		0, /* rx_drop */
+		0, /* rx_interrupt_wait */
+		0, /* rx_umonitor_umwait */
+		0, /* tx_ok */
+		0, /* total_polls */
+		0, /* zero_polls */
+		0, /* tx_nospc_ldb_hw_credits */
+		0, /* tx_nospc_dir_hw_credits */
+		0, /* tx_nospc_inflight_max */
+		0, /* tx_nospc_new_event_limit */
+		0, /* tx_nospc_inflight_credits */
+		0, /* nb_events_limit */
+		0, /* inflight_events */
+		0, /* ldb_pool_size */
+		0, /* dir_pool_size */
+	};
+	static const char * const port_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"tx_new",
+		"tx_fwd",
+		"tx_rel",
+		"tx_implicit_rel",
+		"tx_sched_ordered",
+		"tx_sched_unordered",
+		"tx_sched_atomic",
+		"tx_sched_directed",
+		"tx_invalid",
+		"outstanding_releases",
+		"max_outstanding_releases",
+		"rx_sched_ordered",
+		"rx_sched_unordered",
+		"rx_sched_atomic",
+		"rx_sched_directed",
+		"rx_sched_invalid"
+	};
+	static const enum dlb2_xstats_type port_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		tx_new,
+		tx_fwd,
+		tx_rel,
+		tx_implicit_rel,
+		tx_sched_ordered,
+		tx_sched_unordered,
+		tx_sched_atomic,
+		tx_sched_directed,
+		tx_invalid,
+		outstanding_releases,
+		max_outstanding_releases,
+		rx_sched_ordered,
+		rx_sched_unordered,
+		rx_sched_atomic,
+		rx_sched_directed,
+		rx_sched_invalid
+	};
+	static const uint8_t port_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		1, /* rx_ok */
+		1, /* rx_drop */
+		1, /* rx_interrupt_wait */
+		1, /* rx_umonitor_umwait */
+		1, /* tx_ok */
+		1, /* total_polls */
+		1, /* zero_polls */
+		1, /* tx_nospc_ldb_hw_credits */
+		1, /* tx_nospc_dir_hw_credits */
+		1, /* tx_nospc_inflight_max */
+		1, /* tx_nospc_new_event_limit */
+		1, /* tx_nospc_inflight_credits */
+		1, /* tx_new */
+		1, /* tx_fwd */
+		1, /* tx_rel */
+		1, /* tx_implicit_rel */
+		1, /* tx_sched_ordered */
+		1, /* tx_sched_unordered */
+		1, /* tx_sched_atomic */
+		1, /* tx_sched_directed */
+		1, /* tx_invalid */
+		0, /* outstanding_releases */
+		0, /* max_outstanding_releases */
+		1, /* rx_sched_ordered */
+		1, /* rx_sched_unordered */
+		1, /* rx_sched_atomic */
+		1, /* rx_sched_directed */
+		1  /* rx_sched_invalid */
+	};
+
+	/* QID specific stats */
+	static const char * const qid_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"num_links",
+		"sched_type",
+		"enq_ok",
+		"current_depth",
+		"depth_threshold",
+		"depth_le50_threshold",
+		"depth_gt50_le75_threshold",
+		"depth_gt75_le100_threshold",
+		"depth_gt100_threshold",
+	};
+	static const enum dlb2_xstats_type qid_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		num_links,
+		sched_type,
+		enq_ok,
+		current_depth,
+		depth_threshold,
+		depth_le50_threshold,
+		depth_gt50_le75_threshold,
+		depth_gt75_le100_threshold,
+		depth_gt100_threshold,
+	};
+	static const uint8_t qid_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		0, /* num_links */
+		0, /* sched_type */
+		1, /* enq_ok */
+		0, /* current_depth */
+		0, /* depth_threshold */
+		1, /* depth_le50_threshold */
+		1, /* depth_gt50_le75_threshold */
+		1, /* depth_gt75_le100_threshold */
+		1, /* depth_gt100_threshold */
+	};
+
+	/* ---- end of stat definitions ---- */
+
+	/* check sizes, since a missed comma can lead to strings being
+	 * joined by the compiler.
+	 */
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
+
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_reset_allowed));
+
+	/* other vars */
+	const unsigned int count = RTE_DIM(dev_stats) +
+			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
+			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
+	unsigned int i, port, qid, stat_id = 0;
+
+	dlb2->xstats = rte_zmalloc_socket(NULL,
+			sizeof(dlb2->xstats[0]) * count, 0,
+			dlb2->qm_instance.info.socket_id);
+	if (dlb2->xstats == NULL)
+		return -ENOMEM;
+
+#define sname dlb2->xstats[stat_id].name.name
+	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
+		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
+			.fn_id = DLB2_XSTATS_FN_DEV,
+			.stat = dev_types[i],
+			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
+			.reset_allowed = dev_reset_allowed[i],
+		};
+		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
+	}
+	dlb2->xstats_count_mode_dev = stat_id;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
+		dlb2->xstats_offset_for_port[port] = stat_id;
+
+		uint32_t count_offset = stat_id;
+
+		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_PORT,
+				.obj_idx = port,
+				.stat = port_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_PORT,
+				.reset_allowed = port_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "port_%u_%s",
+				 port, port_stats[i]);
+		}
+
+		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_port = stat_id - dlb2->xstats_count_mode_dev;
+
+	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
+		uint32_t count_offset = stat_id;
+
+		dlb2->xstats_offset_for_qid[qid] = stat_id;
+
+		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_QUEUE,
+				.obj_idx = qid,
+				.stat = qid_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
+				.reset_allowed = qid_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "qid_%u_%s",
+				 qid, qid_stats[i]);
+		}
+
+		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_queue = stat_id -
+		(dlb2->xstats_count_mode_dev + dlb2->xstats_count_mode_port);
+#undef sname
+
+	dlb2->xstats_count = stat_id;
+
+	return 0;
+}
+
+void
+dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
+{
+	rte_free(dlb2->xstats);
+	dlb2->xstats_count = 0;
+}
+
+int
+dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstats_names,
+		unsigned int *ids, unsigned int size)
+{
+	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+	uint32_t start_offset = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			break;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			break;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
+		break;
+	default:
+		return -EINVAL;
+	};
+
+	if (xstats_mode_count > size || ids == NULL || xstats_names == NULL)
+		return xstats_mode_count;
+
+	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
+		if (dlb2->xstats[i].mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != dlb2->xstats[i].obj_idx)
+			continue;
+
+		xstats_names[xidx] = dlb2->xstats[i].name;
+		if (ids)
+			ids[xidx] = start_offset + xidx;
+		xidx++;
+	}
+	return xidx;
+}
+
+static int
+dlb2_xstats_update(struct dlb2_eventdev *dlb2,
+		enum rte_event_dev_xstats_mode mode,
+		uint8_t queue_port_id, const unsigned int ids[],
+		uint64_t values[], unsigned int n, const uint32_t reset)
+{
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			goto invalid_value;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			goto invalid_value;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		break;
+	default:
+		goto invalid_value;
+	};
+
+	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
+		dlb2_xstats_fn fn;
+
+		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != xs->obj_idx)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			goto invalid_value;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+
+		if (values)
+			values[xidx] = val;
+
+		if (xs->reset_allowed && reset)
+			xs->reset_value += val;
+
+		xidx++;
+	}
+
+	return xidx;
+
+invalid_value:
+	return -EINVAL;
+}
+
+int
+dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	const uint32_t reset = 0;
+
+	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
+				  reset);
+}
+
+uint64_t
+dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+				 const char *name, unsigned int *id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	dlb2_xstats_fn fn;
+
+	for (i = 0; i < dlb2->xstats_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (strncmp(xs->name.name, name,
+			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
+			if (id != NULL)
+				*id = i;
+
+			switch (xs->fn_id) {
+			case DLB2_XSTATS_FN_DEV:
+				fn = get_dev_stat;
+				break;
+			case DLB2_XSTATS_FN_PORT:
+				fn = get_port_stat;
+				break;
+			case DLB2_XSTATS_FN_QUEUE:
+				fn = get_queue_stat;
+				break;
+			default:
+				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
+					  xs->fn_id);
+				return (uint64_t)-1;
+			}
+
+			return fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+		}
+	}
+	if (id != NULL)
+		*id = (uint32_t)-1;
+	return (uint64_t)-1;
+}
+
+static void
+dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
+			uint32_t num)
+{
+	uint32_t i;
+	dlb2_xstats_fn fn;
+
+	for (i = start; i < start + num; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (!xs->reset_allowed)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			return;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
+		xs->reset_value = val;
+	}
+}
+
+static int
+dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
+			const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_QUEUE,
+					queue_id, ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	if (ids == NULL)
+		dlb2_xstats_reset_range(dlb2,
+			dlb2->xstats_offset_for_qid[queue_id],
+			dlb2->xstats_count_per_qid[queue_id]);
+
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
+		       const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+	int offset = dlb2->xstats_offset_for_port[port_id];
+	int nb_stat = dlb2->xstats_count_per_port[port_id];
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_PORT, port_id,
+					ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
+		      uint32_t nb_ids)
+{
+	uint32_t i;
+
+	if (ids) {
+		for (i = 0; i < nb_ids; i++) {
+			uint32_t id = ids[i];
+
+			if (id >= dlb2->xstats_count_mode_dev)
+				return -EINVAL;
+			dlb2_xstats_reset_range(dlb2, id, 1);
+		}
+	} else {
+		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
+			dlb2_xstats_reset_range(dlb2, i, 1);
+	}
+
+	return 0;
+}
+
+int
+dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+			   enum rte_event_dev_xstats_mode mode,
+			   int16_t queue_port_id,
+			   const uint32_t ids[],
+			   uint32_t nb_ids)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	uint32_t i;
+
+	/* handle -1 for queue_port_id here, looping over all ports/queues */
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
+			return -EINVAL;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+				if (dlb2_xstats_reset_port(dlb2, i,
+							   ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
+			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
+						   ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
+				if (dlb2_xstats_reset_queue(dlb2, i,
+							    ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
+			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
+						    ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	};
+
+	return 0;
+}
+
+void
+dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	int i;
+
+	if (f == NULL) {
+		printf("Invalid file pointer\n");
+		return;
+	}
+
+	if (dev == NULL) {
+		fprintf(f, "Invalid event device\n");
+		return;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (dlb2 == NULL) {
+		fprintf(f, "DLB2 Event device cannot be dumped!\n");
+		return;
+	}
+
+	if (!dlb2->configured)
+		fprintf(f, "DLB2 Event device is not configured\n");
+
+	handle = &dlb2->qm_instance;
+
+	fprintf(f, "================\n");
+	fprintf(f, "DLB2 Device Dump\n");
+	fprintf(f, "================\n");
+
+	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
+		dlb2->umwait_allowed ? "yes" : "no");
+
+	/* Generic top level device information */
+
+	fprintf(f, "device is configured and run state =");
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		fprintf(f, "STOPPED\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
+		fprintf(f, "STOPPING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
+		fprintf(f, "STARTING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
+		fprintf(f, "STARTED\n");
+	else
+		fprintf(f, "UNEXPECTED\n");
+
+	fprintf(f, "domain ID=%u, socket_id=%u, evdev=%p\n",
+		handle->domain_id, handle->info.socket_id, dlb2->event_dev);
+
+	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
+		dlb2->num_dir_ports, dlb2->num_dir_queues);
+
+	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
+		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
+
+	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
+		handle->cfg.resources.num_atomic_inflights,
+		handle->cfg.resources.num_hist_list_entries);
+
+	fprintf(f, "results from most recent hw resource query:\n");
+
+	fprintf(f, "\tnum_sched_domains = %u\n",
+		dlb2->hw_rsrc_query_results.num_sched_domains);
+
+	fprintf(f, "\tnum_ldb_queues = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_queues);
+
+	fprintf(f, "\tnum_ldb_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_ports);
+
+	fprintf(f, "\tnum_dir_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_ports);
+
+	fprintf(f, "\tnum_atomic_inflights = %u\n",
+		dlb2->hw_rsrc_query_results.num_atomic_inflights);
+
+	fprintf(f, "\tnum_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.num_hist_list_entries);
+
+	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
+
+	fprintf(f, "\tnum_ldb_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_credits);
+
+	fprintf(f, "\tnum_dir_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_credits);
+
+	/* Port level information */
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
+		int j;
+
+		if (!p->enq_configured)
+			fprintf(f, "Port_%d is not configured\n", i);
+
+		fprintf(f, "Port_%d\n", i);
+		fprintf(f, "=======\n");
+
+		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
+			p->id, p->setup_done);
+
+		fprintf(f, "\tconfig state=%d, port state=%d\n",
+			p->qm_port.config_state, p->qm_port.state);
+
+		fprintf(f, "\tport is %s\n",
+			p->qm_port.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\toutstanding releases=%u\n",
+			p->outstanding_releases);
+
+		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
+			p->inflight_max, p->inflight_credits);
+
+		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
+			p->credit_update_quanta, p->implicit_release);
+
+		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (p->link[j].valid)
+				fprintf(f, "id=%u prio=%u ",
+					p->link[j].queue_id,
+					p->link[j].priority);
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
+
+		fprintf(f, "\tcached_ldb_credits=%u\n",
+			p->qm_port.cached_ldb_credits);
+
+		fprintf(f, "\tldb_credits = %u\n",
+			p->qm_port.ldb_credits);
+
+		fprintf(f, "\tcached_dir_credits = %u\n",
+			p->qm_port.cached_dir_credits);
+
+		fprintf(f, "\tdir_credits = %u\n",
+			p->qm_port.dir_credits);
+
+		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
+			p->qm_port.gen_bit,
+			p->qm_port.cq_idx,
+			p->qm_port.cq_depth);
+
+		fprintf(f, "\tinterrupt armed=%d\n",
+			p->qm_port.int_armed);
+
+		fprintf(f, "\tPort statistics\n");
+
+		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
+			p->stats.traffic.rx_ok);
+
+		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
+			p->stats.traffic.rx_drop);
+
+		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
+			p->stats.traffic.rx_interrupt_wait);
+
+		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
+			p->stats.traffic.rx_umonitor_umwait);
+
+		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
+			p->stats.traffic.tx_ok);
+
+		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
+			p->stats.traffic.total_polls);
+
+		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
+			p->stats.traffic.zero_polls);
+
+		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_ldb_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_dir_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_max);
+
+		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_new_event_limit);
+
+		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_credits);
+
+		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
+
+		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
+
+		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
+
+		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
+			p->stats.tx_implicit_rel);
+
+		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
+			p->stats.tx_invalid);
+
+		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
+			p->stats.rx_sched_invalid);
+	}
+
+	/* Queue level information */
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
+		int j, k;
+
+		if (!q->setup_done)
+			fprintf(f, "Queue_%d is not configured\n", i);
+
+		fprintf(f, "Queue_%d\n", i);
+		fprintf(f, "========\n");
+
+		fprintf(f, "\tevqueue_%u is set up\n", q->id);
+
+		fprintf(f, "\tqueue is %s\n",
+			q->qm_queue.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
+
+		for (j = 0; j < dlb2->num_ports; j++) {
+			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
+
+			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++) {
+				if (p->link[k].valid &&
+				    p->link[k].queue_id == q->id)
+					fprintf(f, "id=%u prio=%u ",
+						p->id, p->link[k].priority);
+			}
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\tcurrent depth: %u events\n",
+			dlb2_get_queue_depth(dlb2, q));
+
+		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
+			q->qm_queue.num_qid_inflights, q->qm_queue.sched_type);
+	}
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 99b71f9..6bf8adf 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -3,6 +3,7 @@
 
 sources = files('dlb2.c',
 		'dlb2_iface.c',
+		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c'
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 07/22] event/dlb2: add xstats

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Saturday, October 17, 2020 1:21 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH v2 07/22] event/dlb2: add xstats
> 
> Add support for DLB2 xstats.  Perform initialization and add
> standard xstats entry points.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

(Looks like you're missing Mike's Reviewed-by, also)

> ---
>  drivers/event/dlb2/dlb2.c        |   35 +-
>  drivers/event/dlb2/dlb2_xstats.c | 1240
> ++++++++++++++++++++++++++++++++++++++
>  drivers/event/dlb2/meson.build   |    1 +
>  3 files changed, 1273 insertions(+), 3 deletions(-)
>  create mode 100644 drivers/event/dlb2/dlb2_xstats.c
> 
> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> index 26985b9..a7f8f68 100644
> --- a/drivers/event/dlb2/dlb2.c
> +++ b/drivers/event/dlb2/dlb2.c
> @@ -74,6 +74,21 @@ static struct rte_event_dev_info evdev_dlb2_default_info
> = {
>  struct process_local_port_data
>  dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
> 
> +/*
> + * DUMMY - added so that xstats path will compile/link.
> + * Will be replaced by real version in a subsequent
> + * patch.
> + */
> +uint32_t
> +dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
> +		     struct dlb2_eventdev_queue *queue)
> +{
> +	RTE_SET_USED(dlb2);
> +	RTE_SET_USED(queue);
> +
> +	return 0;
> +}
> +
>  /* override defaults with value(s) provided on command line */
>  static void
>  dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
> @@ -341,9 +356,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
>  static void
>  dlb2_entry_points_init(struct rte_eventdev *dev)
>  {
> -	RTE_SET_USED(dev);
> -
> -	/* Eventdev PMD entry points */
> +	/* Expose PMD's eventdev interface */
> +	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
> +		.dump             = dlb2_eventdev_dump,
> +		.xstats_get       = dlb2_eventdev_xstats_get,
> +		.xstats_get_names = dlb2_eventdev_xstats_get_names,
> +		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
> +		.xstats_reset	    = dlb2_eventdev_xstats_reset,
> +	};
> +
> +	dev->dev_ops = &dlb2_eventdev_entry_ops;
>  }
> 
>  int
> @@ -395,6 +417,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev
> *dev,
>  		return err;
>  	}
> 
> +	/* Complete xtstats runtime initialization */
> +	err = dlb2_xstats_init(dlb2);
> +	if (err) {
> +		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
> +		return err;
> +	}
> +
>  	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
> 
>  	dlb2_iface_low_level_io_init();
> diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
> new file mode 100644
> index 0000000..21391f7
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2_xstats.c
> @@ -0,0 +1,1240 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <inttypes.h>
> +
> +#include <rte_malloc.h>
> +#include <rte_eventdev.h>
> +
> +#include "dlb2_priv.h"
> +#include "dlb2_inline_fns.h"
> +
> +enum dlb2_xstats_type {
> +	/* common to device and port */
> +	rx_ok,				/**< Receive an event */
> +	rx_drop,                        /**< Error bit set in received QE */
> +	rx_interrupt_wait,		/**< Wait on an interrupt */
> +	rx_umonitor_umwait,		/**< Block using umwait */
> +	tx_ok,				/**< Transmit an event */
> +	total_polls,			/**< Call dequeue_burst */
> +	zero_polls,			/**< Call dequeue burst and return 0 */
> +	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
> +	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
> +	tx_nospc_inflight_max,		/**< Reach the
> new_event_threshold */
> +	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
> +	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
> +	/* device specific */
> +	nb_events_limit,
> +	inflight_events,
> +	ldb_pool_size,
> +	dir_pool_size,
> +	/* port specific */
> +	tx_new,				/**< Send an OP_NEW event */
> +	tx_fwd,				/**< Send an OP_FORWARD
> event */
> +	tx_rel,				/**< Send an OP_RELEASE event */
> +	tx_implicit_rel,		/**< Issue an implicit event release */
> +	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED
> event */
> +	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL
> event */
> +	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event
> */
> +	tx_sched_directed,		/**< Send a directed event */
> +	tx_invalid,                     /**< Send an event with an invalid op */
> +	outstanding_releases,		/**< # of releases a port owes */
> +	max_outstanding_releases,	/**< max # of releases a port can owe */
> +	rx_sched_ordered,		/**< Dequeue an ordered event */
> +	rx_sched_unordered,		/**< Dequeue an unordered event */
> +	rx_sched_atomic,		/**< Dequeue an atomic event */
> +	rx_sched_directed,		/**< Dequeue an directed event */
> +	rx_sched_invalid,               /**< Dequeue event sched type invalid */
> +	/* common to port and queue */
> +	is_configured,			/**< Port is configured */
> +	is_load_balanced,		/**< Port is LDB */
> +	hw_id,				/**< Hardware ID */
> +	/* queue specific */
> +	num_links,			/**< Number of ports linked */
> +	sched_type,			/**< Queue sched type */
> +	enq_ok,				/**< # events enqueued to the
> queue */
> +	current_depth,			/**< Current queue depth */
> +	depth_threshold,		/**< Programmed depth threshold */
> +	depth_le50_threshold,
> +	/**< Depth LE to 50% of the configured hardware threshold */
> +	depth_gt50_le75_threshold,
> +	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold
> */
> +	depth_gt75_le100_threshold,
> +	/**< Depth GT 75%. but LE to the configured hardware threshold */
> +	depth_gt100_threshold
> +	/**< Depth GT 100% of the configured hw threshold */
> +};
> +
> +typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
> +		uint16_t obj_idx, /* port or queue id */
> +		enum dlb2_xstats_type stat, int extra_arg);
> +
> +enum dlb2_xstats_fn_type {
> +	DLB2_XSTATS_FN_DEV,
> +	DLB2_XSTATS_FN_PORT,
> +	DLB2_XSTATS_FN_QUEUE
> +};
> +
> +struct dlb2_xstats_entry {
> +	struct rte_event_dev_xstats_name name;
> +	uint64_t reset_value; /* an offset to be taken away to emulate resets */
> +	enum dlb2_xstats_fn_type fn_id;
> +	enum dlb2_xstats_type stat;
> +	enum rte_event_dev_xstats_mode mode;
> +	int extra_arg;
> +	uint16_t obj_idx;
> +	uint8_t reset_allowed; /* when set, this value can be reset */
> +};
> +
> +/* Some device stats are simply a summation of the corresponding port values
> */
> +static uint64_t
> +dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
> +			     int which_stat)
> +{
> +	int i;
> +	uint64_t val = 0;
> +
> +	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
> +		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
> +
> +		if (!port->setup_done)
> +			continue;
> +
> +		switch (which_stat) {
> +		case rx_ok:
> +			val += port->stats.traffic.rx_ok;
> +			break;
> +		case rx_drop:
> +			val += port->stats.traffic.rx_drop;
> +			break;
> +		case rx_interrupt_wait:
> +			val += port->stats.traffic.rx_interrupt_wait;
> +			break;
> +		case rx_umonitor_umwait:
> +			val += port->stats.traffic.rx_umonitor_umwait;
> +			break;
> +		case tx_ok:
> +			val += port->stats.traffic.tx_ok;
> +			break;
> +		case total_polls:
> +			val += port->stats.traffic.total_polls;
> +			break;
> +		case zero_polls:
> +			val += port->stats.traffic.zero_polls;
> +			break;
> +		case tx_nospc_ldb_hw_credits:
> +			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
> +			break;
> +		case tx_nospc_dir_hw_credits:
> +			val += port->stats.traffic.tx_nospc_dir_hw_credits;
> +			break;
> +		case tx_nospc_inflight_max:
> +			val += port->stats.traffic.tx_nospc_inflight_max;
> +			break;
> +		case tx_nospc_new_event_limit:
> +			val += port->stats.traffic.tx_nospc_new_event_limit;
> +			break;
> +		case tx_nospc_inflight_credits:
> +			val += port->stats.traffic.tx_nospc_inflight_credits;
> +			break;
> +		default:
> +			return -1;
> +		}
> +	}
> +	return val;
> +}
> +
> +static uint64_t
> +get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
> +	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
> +{
> +	switch (type) {
> +	case rx_ok:
> +	case rx_drop:
> +	case rx_interrupt_wait:
> +	case rx_umonitor_umwait:
> +	case tx_ok:
> +	case total_polls:
> +	case zero_polls:
> +	case tx_nospc_ldb_hw_credits:
> +	case tx_nospc_dir_hw_credits:
> +	case tx_nospc_inflight_max:
> +	case tx_nospc_new_event_limit:
> +	case tx_nospc_inflight_credits:
> +		return dlb2_device_traffic_stat_get(dlb2, type);
> +	case nb_events_limit:
> +		return dlb2->new_event_limit;
> +	case inflight_events:
> +		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
> +	case ldb_pool_size:
> +		return dlb2->num_ldb_credits;
> +	case dir_pool_size:
> +		return dlb2->num_dir_credits;
> +	default: return -1;
> +	}
> +}
> +
> +static uint64_t
> +get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
> +	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
> +{
> +	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
> +
> +	switch (type) {
> +	case rx_ok: return ev_port->stats.traffic.rx_ok;
> +
> +	case rx_drop: return ev_port->stats.traffic.rx_drop;
> +
> +	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
> +
> +	case rx_umonitor_umwait:
> +		return ev_port->stats.traffic.rx_umonitor_umwait;
> +
> +	case tx_ok: return ev_port->stats.traffic.tx_ok;
> +
> +	case total_polls: return ev_port->stats.traffic.total_polls;
> +
> +	case zero_polls: return ev_port->stats.traffic.zero_polls;
> +
> +	case tx_nospc_ldb_hw_credits:
> +		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
> +
> +	case tx_nospc_dir_hw_credits:
> +		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
> +
> +	case tx_nospc_inflight_max:
> +		return ev_port->stats.traffic.tx_nospc_inflight_max;
> +
> +	case tx_nospc_new_event_limit:
> +		return ev_port->stats.traffic.tx_nospc_new_event_limit;
> +
> +	case tx_nospc_inflight_credits:
> +		return ev_port->stats.traffic.tx_nospc_inflight_credits;
> +
> +	case is_configured: return ev_port->setup_done;
> +
> +	case is_load_balanced: return !ev_port->qm_port.is_directed;
> +
> +	case hw_id: return ev_port->qm_port.id;
> +
> +	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
> +
> +	case tx_fwd: return ev_port-
> >stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
> +
> +	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
> +
> +	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
> +
> +	case tx_sched_ordered:
> +		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
> +
> +	case tx_sched_unordered:
> +		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
> +
> +	case tx_sched_atomic:
> +		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
> +
> +	case tx_sched_directed:
> +		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
> +
> +	case tx_invalid: return ev_port->stats.tx_invalid;
> +
> +	case outstanding_releases: return ev_port->outstanding_releases;
> +
> +	case max_outstanding_releases:
> +		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
> +
> +	case rx_sched_ordered:
> +		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
> +
> +	case rx_sched_unordered:
> +		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
> +
> +	case rx_sched_atomic:
> +		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
> +
> +	case rx_sched_directed:
> +		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
> +
> +	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
> +
> +	default: return -1;
> +	}
> +}
> +
> +static uint64_t
> +dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
> +{
> +	int port = 0;
> +	uint64_t tally = 0;
> +
> +	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
> +		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
> +
> +	return tally;
> +}
> +
> +static uint64_t
> +dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
> +{
> +	int port = 0;
> +	uint64_t enq_ok_tally = 0;
> +
> +	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
> +		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
> +
> +	return enq_ok_tally;
> +}
> +
> +static uint64_t
> +get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
> +	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
> +{
> +	struct dlb2_eventdev_queue *ev_queue =
> +		&dlb2->ev_queues[obj_idx];
> +
> +	switch (type) {
> +	case is_configured: return ev_queue->setup_done;
> +
> +	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
> +
> +	case hw_id: return ev_queue->qm_queue.id;
> +
> +	case num_links: return ev_queue->num_links;
> +
> +	case sched_type: return ev_queue->qm_queue.sched_type;
> +
> +	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
> +
> +	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
> +
> +	case depth_threshold: return ev_queue->depth_threshold;
> +
> +	case depth_le50_threshold:
> +		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
> +					       DLB2_QID_DEPTH_LE50);
> +
> +	case depth_gt50_le75_threshold:
> +		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
> +					       DLB2_QID_DEPTH_GT50_LE75);
> +
> +	case depth_gt75_le100_threshold:
> +		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
> +					       DLB2_QID_DEPTH_GT75_LE100);
> +
> +	case depth_gt100_threshold:
> +		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
> +					       DLB2_QID_DEPTH_GT100);
> +
> +	default: return -1;
> +	}
> +}
> +
> +int
> +dlb2_xstats_init(struct dlb2_eventdev *dlb2)
> +{
> +	/*
> +	 * define the stats names and types. Used to build up the device
> +	 * xstats array
> +	 * There are multiple set of stats:
> +	 *   - device-level,
> +	 *   - per-port,
> +	 *   - per-qid,
> +	 *
> +	 * For each of these sets, we have three parallel arrays, one for the
> +	 * names, the other for the stat type parameter to be passed in the fn
> +	 * call to get that stat. The third array allows resetting or not.
> +	 * All these arrays must be kept in sync
> +	 */
> +	static const char * const dev_stats[] = {
> +		"rx_ok",
> +		"rx_drop",
> +		"rx_interrupt_wait",
> +		"rx_umonitor_umwait",
> +		"tx_ok",
> +		"total_polls",
> +		"zero_polls",
> +		"tx_nospc_ldb_hw_credits",
> +		"tx_nospc_dir_hw_credits",
> +		"tx_nospc_inflight_max",
> +		"tx_nospc_new_event_limit",
> +		"tx_nospc_inflight_credits",
> +		"nb_events_limit",
> +		"inflight_events",
> +		"ldb_pool_size",
> +		"dir_pool_size",
> +	};
> +	static const enum dlb2_xstats_type dev_types[] = {
> +		rx_ok,
> +		rx_drop,
> +		rx_interrupt_wait,
> +		rx_umonitor_umwait,
> +		tx_ok,
> +		total_polls,
> +		zero_polls,
> +		tx_nospc_ldb_hw_credits,
> +		tx_nospc_dir_hw_credits,
> +		tx_nospc_inflight_max,
> +		tx_nospc_new_event_limit,
> +		tx_nospc_inflight_credits,
> +		nb_events_limit,
> +		inflight_events,
> +		ldb_pool_size,
> +		dir_pool_size,
> +	};
> +	/* Note: generated device stats are not allowed to be reset. */
> +	static const uint8_t dev_reset_allowed[] = {
> +		0, /* rx_ok */
> +		0, /* rx_drop */
> +		0, /* rx_interrupt_wait */
> +		0, /* rx_umonitor_umwait */
> +		0, /* tx_ok */
> +		0, /* total_polls */
> +		0, /* zero_polls */
> +		0, /* tx_nospc_ldb_hw_credits */
> +		0, /* tx_nospc_dir_hw_credits */
> +		0, /* tx_nospc_inflight_max */
> +		0, /* tx_nospc_new_event_limit */
> +		0, /* tx_nospc_inflight_credits */
> +		0, /* nb_events_limit */
> +		0, /* inflight_events */
> +		0, /* ldb_pool_size */
> +		0, /* dir_pool_size */
> +	};
> +	static const char * const port_stats[] = {
> +		"is_configured",
> +		"is_load_balanced",
> +		"hw_id",
> +		"rx_ok",
> +		"rx_drop",
> +		"rx_interrupt_wait",
> +		"rx_umonitor_umwait",
> +		"tx_ok",
> +		"total_polls",
> +		"zero_polls",
> +		"tx_nospc_ldb_hw_credits",
> +		"tx_nospc_dir_hw_credits",
> +		"tx_nospc_inflight_max",
> +		"tx_nospc_new_event_limit",
> +		"tx_nospc_inflight_credits",
> +		"tx_new",
> +		"tx_fwd",
> +		"tx_rel",
> +		"tx_implicit_rel",
> +		"tx_sched_ordered",
> +		"tx_sched_unordered",
> +		"tx_sched_atomic",
> +		"tx_sched_directed",
> +		"tx_invalid",
> +		"outstanding_releases",
> +		"max_outstanding_releases",
> +		"rx_sched_ordered",
> +		"rx_sched_unordered",
> +		"rx_sched_atomic",
> +		"rx_sched_directed",
> +		"rx_sched_invalid"
> +	};
> +	static const enum dlb2_xstats_type port_types[] = {
> +		is_configured,
> +		is_load_balanced,
> +		hw_id,
> +		rx_ok,
> +		rx_drop,
> +		rx_interrupt_wait,
> +		rx_umonitor_umwait,
> +		tx_ok,
> +		total_polls,
> +		zero_polls,
> +		tx_nospc_ldb_hw_credits,
> +		tx_nospc_dir_hw_credits,
> +		tx_nospc_inflight_max,
> +		tx_nospc_new_event_limit,
> +		tx_nospc_inflight_credits,
> +		tx_new,
> +		tx_fwd,
> +		tx_rel,
> +		tx_implicit_rel,
> +		tx_sched_ordered,
> +		tx_sched_unordered,
> +		tx_sched_atomic,
> +		tx_sched_directed,
> +		tx_invalid,
> +		outstanding_releases,
> +		max_outstanding_releases,
> +		rx_sched_ordered,
> +		rx_sched_unordered,
> +		rx_sched_atomic,
> +		rx_sched_directed,
> +		rx_sched_invalid
> +	};
> +	static const uint8_t port_reset_allowed[] = {
> +		0, /* is_configured */
> +		0, /* is_load_balanced */
> +		0, /* hw_id */
> +		1, /* rx_ok */
> +		1, /* rx_drop */
> +		1, /* rx_interrupt_wait */
> +		1, /* rx_umonitor_umwait */
> +		1, /* tx_ok */
> +		1, /* total_polls */
> +		1, /* zero_polls */
> +		1, /* tx_nospc_ldb_hw_credits */
> +		1, /* tx_nospc_dir_hw_credits */
> +		1, /* tx_nospc_inflight_max */
> +		1, /* tx_nospc_new_event_limit */
> +		1, /* tx_nospc_inflight_credits */
> +		1, /* tx_new */
> +		1, /* tx_fwd */
> +		1, /* tx_rel */
> +		1, /* tx_implicit_rel */
> +		1, /* tx_sched_ordered */
> +		1, /* tx_sched_unordered */
> +		1, /* tx_sched_atomic */
> +		1, /* tx_sched_directed */
> +		1, /* tx_invalid */
> +		0, /* outstanding_releases */
> +		0, /* max_outstanding_releases */
> +		1, /* rx_sched_ordered */
> +		1, /* rx_sched_unordered */
> +		1, /* rx_sched_atomic */
> +		1, /* rx_sched_directed */
> +		1  /* rx_sched_invalid */
> +	};
> +
> +	/* QID specific stats */
> +	static const char * const qid_stats[] = {
> +		"is_configured",
> +		"is_load_balanced",
> +		"hw_id",
> +		"num_links",
> +		"sched_type",
> +		"enq_ok",
> +		"current_depth",
> +		"depth_threshold",
> +		"depth_le50_threshold",
> +		"depth_gt50_le75_threshold",
> +		"depth_gt75_le100_threshold",
> +		"depth_gt100_threshold",
> +	};
> +	static const enum dlb2_xstats_type qid_types[] = {
> +		is_configured,
> +		is_load_balanced,
> +		hw_id,
> +		num_links,
> +		sched_type,
> +		enq_ok,
> +		current_depth,
> +		depth_threshold,
> +		depth_le50_threshold,
> +		depth_gt50_le75_threshold,
> +		depth_gt75_le100_threshold,
> +		depth_gt100_threshold,
> +	};
> +	static const uint8_t qid_reset_allowed[] = {
> +		0, /* is_configured */
> +		0, /* is_load_balanced */
> +		0, /* hw_id */
> +		0, /* num_links */
> +		0, /* sched_type */
> +		1, /* enq_ok */
> +		0, /* current_depth */
> +		0, /* depth_threshold */
> +		1, /* depth_le50_threshold */
> +		1, /* depth_gt50_le75_threshold */
> +		1, /* depth_gt75_le100_threshold */
> +		1, /* depth_gt100_threshold */
> +	};
> +
> +	/* ---- end of stat definitions ---- */
> +
> +	/* check sizes, since a missed comma can lead to strings being
> +	 * joined by the compiler.
> +	 */
> +	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
> +	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
> +	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
> +
> +	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) !=
> RTE_DIM(dev_reset_allowed));
> +	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) !=
> RTE_DIM(port_reset_allowed));
> +	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) !=
> RTE_DIM(qid_reset_allowed));
> +
> +	/* other vars */
> +	const unsigned int count = RTE_DIM(dev_stats) +
> +			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
> +			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
> +	unsigned int i, port, qid, stat_id = 0;
> +
> +	dlb2->xstats = rte_zmalloc_socket(NULL,
> +			sizeof(dlb2->xstats[0]) * count, 0,
> +			dlb2->qm_instance.info.socket_id);
> +	if (dlb2->xstats == NULL)
> +		return -ENOMEM;
> +
> +#define sname dlb2->xstats[stat_id].name.name
> +	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
> +		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
> +			.fn_id = DLB2_XSTATS_FN_DEV,
> +			.stat = dev_types[i],
> +			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
> +			.reset_allowed = dev_reset_allowed[i],
> +		};
> +		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
> +	}
> +	dlb2->xstats_count_mode_dev = stat_id;
> +
> +	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
> +		dlb2->xstats_offset_for_port[port] = stat_id;
> +
> +		uint32_t count_offset = stat_id;
> +
> +		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
> +			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
> +				.fn_id = DLB2_XSTATS_FN_PORT,
> +				.obj_idx = port,
> +				.stat = port_types[i],
> +				.mode = RTE_EVENT_DEV_XSTATS_PORT,
> +				.reset_allowed = port_reset_allowed[i],
> +			};
> +			snprintf(sname, sizeof(sname), "port_%u_%s",
> +				 port, port_stats[i]);
> +		}
> +
> +		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
> +	}
> +
> +	dlb2->xstats_count_mode_port = stat_id - dlb2-
> >xstats_count_mode_dev;
> +
> +	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
> +		uint32_t count_offset = stat_id;
> +
> +		dlb2->xstats_offset_for_qid[qid] = stat_id;
> +
> +		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
> +			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
> +				.fn_id = DLB2_XSTATS_FN_QUEUE,
> +				.obj_idx = qid,
> +				.stat = qid_types[i],
> +				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
> +				.reset_allowed = qid_reset_allowed[i],
> +			};
> +			snprintf(sname, sizeof(sname), "qid_%u_%s",
> +				 qid, qid_stats[i]);
> +		}
> +
> +		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
> +	}
> +
> +	dlb2->xstats_count_mode_queue = stat_id -
> +		(dlb2->xstats_count_mode_dev + dlb2-
> >xstats_count_mode_port);
> +#undef sname
> +
> +	dlb2->xstats_count = stat_id;
> +
> +	return 0;
> +}
> +
> +void
> +dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
> +{
> +	rte_free(dlb2->xstats);
> +	dlb2->xstats_count = 0;
> +}
> +
> +int
> +dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
> +		enum rte_event_dev_xstats_mode mode, uint8_t
> queue_port_id,
> +		struct rte_event_dev_xstats_name *xstats_names,
> +		unsigned int *ids, unsigned int size)
> +{
> +	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	unsigned int i;
> +	unsigned int xidx = 0;
> +	uint32_t xstats_mode_count = 0;
> +	uint32_t start_offset = 0;
> +
> +	switch (mode) {
> +	case RTE_EVENT_DEV_XSTATS_DEVICE:
> +		xstats_mode_count = dlb2->xstats_count_mode_dev;
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_PORT:
> +		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
> +			break;
> +		xstats_mode_count = dlb2-
> >xstats_count_per_port[queue_port_id];
> +		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_QUEUE:
> +#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
> +		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
> +			break;
> +#endif
> +		xstats_mode_count = dlb2-
> >xstats_count_per_qid[queue_port_id];
> +		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
> +		break;
> +	default:
> +		return -EINVAL;
> +	};
> +
> +	if (xstats_mode_count > size || ids == NULL || xstats_names == NULL)
> +		return xstats_mode_count;
> +
> +	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
> +		if (dlb2->xstats[i].mode != mode)
> +			continue;
> +
> +		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
> +		    queue_port_id != dlb2->xstats[i].obj_idx)
> +			continue;
> +
> +		xstats_names[xidx] = dlb2->xstats[i].name;
> +		if (ids)
> +			ids[xidx] = start_offset + xidx;
> +		xidx++;
> +	}
> +	return xidx;
> +}
> +
> +static int
> +dlb2_xstats_update(struct dlb2_eventdev *dlb2,
> +		enum rte_event_dev_xstats_mode mode,
> +		uint8_t queue_port_id, const unsigned int ids[],
> +		uint64_t values[], unsigned int n, const uint32_t reset)
> +{
> +	unsigned int i;
> +	unsigned int xidx = 0;
> +	uint32_t xstats_mode_count = 0;
> +
> +	switch (mode) {
> +	case RTE_EVENT_DEV_XSTATS_DEVICE:
> +		xstats_mode_count = dlb2->xstats_count_mode_dev;
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_PORT:
> +		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
> +			goto invalid_value;
> +		xstats_mode_count = dlb2-
> >xstats_count_per_port[queue_port_id];
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_QUEUE:
> +#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
> +		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
> +			goto invalid_value;
> +#endif
> +		xstats_mode_count = dlb2-
> >xstats_count_per_qid[queue_port_id];
> +		break;
> +	default:
> +		goto invalid_value;
> +	};
> +
> +	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
> +		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
> +		dlb2_xstats_fn fn;
> +
> +		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
> +			continue;
> +
> +		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
> +		    queue_port_id != xs->obj_idx)
> +			continue;
> +
> +		switch (xs->fn_id) {
> +		case DLB2_XSTATS_FN_DEV:
> +			fn = get_dev_stat;
> +			break;
> +		case DLB2_XSTATS_FN_PORT:
> +			fn = get_port_stat;
> +			break;
> +		case DLB2_XSTATS_FN_QUEUE:
> +			fn = get_queue_stat;
> +			break;
> +		default:
> +			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs-
> >fn_id);
> +			goto invalid_value;
> +		}
> +
> +		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
> +				  xs->extra_arg) - xs->reset_value;
> +
> +		if (values)
> +			values[xidx] = val;
> +
> +		if (xs->reset_allowed && reset)
> +			xs->reset_value += val;
> +
> +		xidx++;
> +	}
> +
> +	return xidx;
> +
> +invalid_value:
> +	return -EINVAL;
> +}
> +
> +int
> +dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
> +		enum rte_event_dev_xstats_mode mode, uint8_t
> queue_port_id,
> +		const unsigned int ids[], uint64_t values[], unsigned int n)
> +{
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	const uint32_t reset = 0;
> +
> +	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
> +				  reset);
> +}
> +
> +uint64_t
> +dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
> +				 const char *name, unsigned int *id)
> +{
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	unsigned int i;
> +	dlb2_xstats_fn fn;
> +
> +	for (i = 0; i < dlb2->xstats_count; i++) {
> +		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
> +
> +		if (strncmp(xs->name.name, name,
> +			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
> +			if (id != NULL)
> +				*id = i;
> +
> +			switch (xs->fn_id) {
> +			case DLB2_XSTATS_FN_DEV:
> +				fn = get_dev_stat;
> +				break;
> +			case DLB2_XSTATS_FN_PORT:
> +				fn = get_port_stat;
> +				break;
> +			case DLB2_XSTATS_FN_QUEUE:
> +				fn = get_queue_stat;
> +				break;
> +			default:
> +				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
> +					  xs->fn_id);
> +				return (uint64_t)-1;
> +			}
> +
> +			return fn(dlb2, xs->obj_idx, xs->stat,
> +				  xs->extra_arg) - xs->reset_value;
> +		}
> +	}
> +	if (id != NULL)
> +		*id = (uint32_t)-1;
> +	return (uint64_t)-1;
> +}
> +
> +static void
> +dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
> +			uint32_t num)
> +{
> +	uint32_t i;
> +	dlb2_xstats_fn fn;
> +
> +	for (i = start; i < start + num; i++) {
> +		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
> +
> +		if (!xs->reset_allowed)
> +			continue;
> +
> +		switch (xs->fn_id) {
> +		case DLB2_XSTATS_FN_DEV:
> +			fn = get_dev_stat;
> +			break;
> +		case DLB2_XSTATS_FN_PORT:
> +			fn = get_port_stat;
> +			break;
> +		case DLB2_XSTATS_FN_QUEUE:
> +			fn = get_queue_stat;
> +			break;
> +		default:
> +			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs-
> >fn_id);
> +			return;
> +		}
> +
> +		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
> +		xs->reset_value = val;
> +	}
> +}
> +
> +static int
> +dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
> +			const uint32_t ids[], uint32_t nb_ids)
> +{
> +	const uint32_t reset = 1;
> +
> +	if (ids) {
> +		uint32_t nb_reset = dlb2_xstats_update(dlb2,
> +					RTE_EVENT_DEV_XSTATS_QUEUE,
> +					queue_id, ids, NULL, nb_ids,
> +					reset);
> +		return nb_reset == nb_ids ? 0 : -EINVAL;
> +	}
> +
> +	if (ids == NULL)
> +		dlb2_xstats_reset_range(dlb2,
> +			dlb2->xstats_offset_for_qid[queue_id],
> +			dlb2->xstats_count_per_qid[queue_id]);
> +
> +	return 0;
> +}
> +
> +static int
> +dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
> +		       const uint32_t ids[], uint32_t nb_ids)
> +{
> +	const uint32_t reset = 1;
> +	int offset = dlb2->xstats_offset_for_port[port_id];
> +	int nb_stat = dlb2->xstats_count_per_port[port_id];
> +
> +	if (ids) {
> +		uint32_t nb_reset = dlb2_xstats_update(dlb2,
> +					RTE_EVENT_DEV_XSTATS_PORT, port_id,
> +					ids, NULL, nb_ids,
> +					reset);
> +		return nb_reset == nb_ids ? 0 : -EINVAL;
> +	}
> +
> +	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
> +	return 0;
> +}
> +
> +static int
> +dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
> +		      uint32_t nb_ids)
> +{
> +	uint32_t i;
> +
> +	if (ids) {
> +		for (i = 0; i < nb_ids; i++) {
> +			uint32_t id = ids[i];
> +
> +			if (id >= dlb2->xstats_count_mode_dev)
> +				return -EINVAL;
> +			dlb2_xstats_reset_range(dlb2, id, 1);
> +		}
> +	} else {
> +		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
> +			dlb2_xstats_reset_range(dlb2, i, 1);
> +	}
> +
> +	return 0;
> +}
> +
> +int
> +dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
> +			   enum rte_event_dev_xstats_mode mode,
> +			   int16_t queue_port_id,
> +			   const uint32_t ids[],
> +			   uint32_t nb_ids)
> +{
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	uint32_t i;
> +
> +	/* handle -1 for queue_port_id here, looping over all ports/queues */
> +	switch (mode) {
> +	case RTE_EVENT_DEV_XSTATS_DEVICE:
> +		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
> +			return -EINVAL;
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_PORT:
> +		if (queue_port_id == -1) {
> +			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
> +				if (dlb2_xstats_reset_port(dlb2, i,
> +							   ids, nb_ids))
> +					return -EINVAL;
> +			}
> +		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
> +			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
> +						   ids, nb_ids))
> +				return -EINVAL;
> +		}
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_QUEUE:
> +		if (queue_port_id == -1) {
> +			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
> +				if (dlb2_xstats_reset_queue(dlb2, i,
> +							    ids, nb_ids))
> +					return -EINVAL;
> +			}
> +		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
> +			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
> +						    ids, nb_ids))
> +				return -EINVAL;
> +		}
> +		break;
> +	};
> +
> +	return 0;
> +}
> +
> +void
> +dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
> +{
> +	struct dlb2_eventdev *dlb2;
> +	struct dlb2_hw_dev *handle;
> +	int i;
> +
> +	if (f == NULL) {
> +		printf("Invalid file pointer\n");
> +		return;
> +	}
> +
> +	if (dev == NULL) {
> +		fprintf(f, "Invalid event device\n");
> +		return;
> +	}
> +
> +	dlb2 = dlb2_pmd_priv(dev);
> +
> +	if (dlb2 == NULL) {
> +		fprintf(f, "DLB2 Event device cannot be dumped!\n");
> +		return;
> +	}
> +
> +	if (!dlb2->configured)
> +		fprintf(f, "DLB2 Event device is not configured\n");
> +
> +	handle = &dlb2->qm_instance;
> +
> +	fprintf(f, "================\n");
> +	fprintf(f, "DLB2 Device Dump\n");
> +	fprintf(f, "================\n");
> +
> +	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
> +		dlb2->umwait_allowed ? "yes" : "no");
> +
> +	/* Generic top level device information */
> +
> +	fprintf(f, "device is configured and run state =");
> +	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
> +		fprintf(f, "STOPPED\n");
> +	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
> +		fprintf(f, "STOPPING\n");
> +	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
> +		fprintf(f, "STARTING\n");
> +	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
> +		fprintf(f, "STARTED\n");
> +	else
> +		fprintf(f, "UNEXPECTED\n");
> +
> +	fprintf(f, "domain ID=%u, socket_id=%u, evdev=%p\n",
> +		handle->domain_id, handle->info.socket_id, dlb2->event_dev);
> +
> +	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
> +		dlb2->num_dir_ports, dlb2->num_dir_queues);
> +
> +	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
> +		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
> +
> +	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
> +		handle->cfg.resources.num_atomic_inflights,
> +		handle->cfg.resources.num_hist_list_entries);
> +
> +	fprintf(f, "results from most recent hw resource query:\n");
> +
> +	fprintf(f, "\tnum_sched_domains = %u\n",
> +		dlb2->hw_rsrc_query_results.num_sched_domains);
> +
> +	fprintf(f, "\tnum_ldb_queues = %u\n",
> +		dlb2->hw_rsrc_query_results.num_ldb_queues);
> +
> +	fprintf(f, "\tnum_ldb_ports = %u\n",
> +		dlb2->hw_rsrc_query_results.num_ldb_ports);
> +
> +	fprintf(f, "\tnum_dir_ports = %u\n",
> +		dlb2->hw_rsrc_query_results.num_dir_ports);
> +
> +	fprintf(f, "\tnum_atomic_inflights = %u\n",
> +		dlb2->hw_rsrc_query_results.num_atomic_inflights);
> +
> +	fprintf(f, "\tnum_hist_list_entries = %u\n",
> +		dlb2->hw_rsrc_query_results.num_hist_list_entries);
> +
> +	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
> +		dlb2-
> >hw_rsrc_query_results.max_contiguous_hist_list_entries);
> +
> +	fprintf(f, "\tnum_ldb_credits = %u\n",
> +		dlb2->hw_rsrc_query_results.num_ldb_credits);
> +
> +	fprintf(f, "\tnum_dir_credits = %u\n",
> +		dlb2->hw_rsrc_query_results.num_dir_credits);
> +
> +	/* Port level information */
> +
> +	for (i = 0; i < dlb2->num_ports; i++) {
> +		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
> +		int j;
> +
> +		if (!p->enq_configured)
> +			fprintf(f, "Port_%d is not configured\n", i);
> +
> +		fprintf(f, "Port_%d\n", i);
> +		fprintf(f, "=======\n");
> +
> +		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
> +			p->id, p->setup_done);
> +
> +		fprintf(f, "\tconfig state=%d, port state=%d\n",
> +			p->qm_port.config_state, p->qm_port.state);
> +
> +		fprintf(f, "\tport is %s\n",
> +			p->qm_port.is_directed ? "directed" : "load balanced");
> +
> +		fprintf(f, "\toutstanding releases=%u\n",
> +			p->outstanding_releases);
> +
> +		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
> +			p->inflight_max, p->inflight_credits);
> +
> +		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
> +			p->credit_update_quanta, p->implicit_release);
> +
> +		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
> +
> +		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
> +			if (p->link[j].valid)
> +				fprintf(f, "id=%u prio=%u ",
> +					p->link[j].queue_id,
> +					p->link[j].priority);
> +		}
> +		fprintf(f, "\n");
> +
> +		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
> +
> +		fprintf(f, "\tcached_ldb_credits=%u\n",
> +			p->qm_port.cached_ldb_credits);
> +
> +		fprintf(f, "\tldb_credits = %u\n",
> +			p->qm_port.ldb_credits);
> +
> +		fprintf(f, "\tcached_dir_credits = %u\n",
> +			p->qm_port.cached_dir_credits);
> +
> +		fprintf(f, "\tdir_credits = %u\n",
> +			p->qm_port.dir_credits);
> +
> +		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
> +			p->qm_port.gen_bit,
> +			p->qm_port.cq_idx,
> +			p->qm_port.cq_depth);
> +
> +		fprintf(f, "\tinterrupt armed=%d\n",
> +			p->qm_port.int_armed);
> +
> +		fprintf(f, "\tPort statistics\n");
> +
> +		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
> +			p->stats.traffic.rx_ok);
> +
> +		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
> +			p->stats.traffic.rx_drop);
> +
> +		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
> +			p->stats.traffic.rx_interrupt_wait);
> +
> +		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
> +			p->stats.traffic.rx_umonitor_umwait);
> +
> +		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
> +			p->stats.traffic.tx_ok);
> +
> +		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
> +			p->stats.traffic.total_polls);
> +
> +		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
> +			p->stats.traffic.zero_polls);
> +
> +		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
> +			p->stats.traffic.tx_nospc_ldb_hw_credits);
> +
> +		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
> +			p->stats.traffic.tx_nospc_dir_hw_credits);
> +
> +		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
> +			p->stats.traffic.tx_nospc_inflight_max);
> +
> +		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
> +			p->stats.traffic.tx_nospc_new_event_limit);
> +
> +		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
> +			p->stats.traffic.tx_nospc_inflight_credits);
> +
> +		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
> +			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
> +
> +		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
> +			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
> +
> +		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
> +			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
> +
> +		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
> +			p->stats.tx_implicit_rel);
> +
> +		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
> +			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
> +
> +		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
> +			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
> +
> +		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
> +			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
> +
> +		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
> +			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
> +
> +		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
> +			p->stats.tx_invalid);
> +
> +		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
> +			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
> +
> +		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
> +			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
> +
> +		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
> +			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
> +
> +		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
> +			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
> +
> +		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
> +			p->stats.rx_sched_invalid);
> +	}
> +
> +	/* Queue level information */
> +
> +	for (i = 0; i < dlb2->num_queues; i++) {
> +		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
> +		int j, k;
> +
> +		if (!q->setup_done)
> +			fprintf(f, "Queue_%d is not configured\n", i);
> +
> +		fprintf(f, "Queue_%d\n", i);
> +		fprintf(f, "========\n");
> +
> +		fprintf(f, "\tevqueue_%u is set up\n", q->id);
> +
> +		fprintf(f, "\tqueue is %s\n",
> +			q->qm_queue.is_directed ? "directed" : "load balanced");
> +
> +		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
> +
> +		for (j = 0; j < dlb2->num_ports; j++) {
> +			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
> +
> +			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++)
> {
> +				if (p->link[k].valid &&
> +				    p->link[k].queue_id == q->id)
> +					fprintf(f, "id=%u prio=%u ",
> +						p->id, p->link[k].priority);
> +			}
> +		}
> +		fprintf(f, "\n");
> +
> +		fprintf(f, "\tcurrent depth: %u events\n",
> +			dlb2_get_queue_depth(dlb2, q));
> +
> +		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
> +			q->qm_queue.num_qid_inflights, q-
> >qm_queue.sched_type);
> +	}
> +}
> diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
> index 99b71f9..6bf8adf 100644
> --- a/drivers/event/dlb2/meson.build
> +++ b/drivers/event/dlb2/meson.build
> @@ -3,6 +3,7 @@
> 
>  sources = files('dlb2.c',
>  		'dlb2_iface.c',
> +		'dlb2_xstats.c',
>  		'pf/dlb2_main.c',
>  		'pf/dlb2_pf.c',
>  		'pf/base/dlb2_resource.c'
> --
> 2.6.4

[dpdk-dev] [PATCH v2 08/22] event/dlb2: add infos get and configure

[Timothy McDaniel]

Add support for configuring the DLB2 hardware.
In particular, this patch configures the DLB2
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  |  310 +++
 drivers/event/dlb2/dlb2_iface.c            |    7 +-
 drivers/event/dlb2/dlb2_iface.h            |    5 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 3234 ++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |   15 +
 drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
 6 files changed, 3614 insertions(+), 1 deletion(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index a7f8f68..79b23da 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -89,6 +89,22 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
 	return 0;
 }
 
+static void
+dlb2_free_qe_mem(struct dlb2_port *qm_port)
+{
+	if (qm_port == NULL)
+		return;
+
+	rte_free(qm_port->qe4);
+	qm_port->qe4 = NULL;
+
+	rte_free(qm_port->int_arm_qe);
+	qm_port->int_arm_qe = NULL;
+
+	rte_free(qm_port->consume_qe);
+	qm_port->consume_qe = NULL;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -354,10 +370,304 @@ set_qid_depth_thresh(const char *key __rte_unused,
 }
 
 static void
+dlb2_eventdev_info_get(struct rte_eventdev *dev,
+		       struct rte_event_dev_info *dev_info)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret;
+
+	ret = dlb2_hw_query_resources(dlb2);
+	if (ret) {
+		const struct rte_eventdev_data *data = dev->data;
+
+		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+			     ret, data->dev_id);
+		/* fn is void, so fall through and return values set up in
+		 * probe
+		 */
+	}
+
+	/* Add num resources currently owned by this domain.
+	 * These would become available if the scheduling domain were reset due
+	 * to the application recalling eventdev_configure to *reconfigure* the
+	 * domain.
+	 */
+	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
+	evdev_dlb2_default_info.max_event_queues += dlb2->num_ldb_queues;
+	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
+
+	evdev_dlb2_default_info.max_event_queues =
+		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
+			RTE_EVENT_MAX_QUEUES_PER_DEV);
+
+	evdev_dlb2_default_info.max_num_events =
+		RTE_MIN(evdev_dlb2_default_info.max_num_events,
+			dlb2->max_num_events_override);
+
+	*dev_info = evdev_dlb2_default_info;
+}
+
+static int
+dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
+			    const struct dlb2_hw_rsrcs *resources_asked)
+{
+	int ret = 0;
+	struct dlb2_create_sched_domain_args *cfg;
+
+	if (resources_asked == NULL) {
+		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
+		ret = EINVAL;
+		goto error_exit;
+	}
+
+	/* Map generic qm resources to dlb2 resources */
+	cfg = &handle->cfg.resources;
+
+	/* DIR ports and queues */
+
+	cfg->num_dir_ports = resources_asked->num_dir_ports;
+
+	cfg->num_dir_credits = resources_asked->num_dir_credits;
+
+	/* LDB queues */
+
+	cfg->num_ldb_queues = resources_asked->num_ldb_queues;
+
+	/* LDB ports */
+
+	cfg->cos_strict = 0; /* Best effort */
+	cfg->num_cos_ldb_ports[0] = 0;
+	cfg->num_cos_ldb_ports[1] = 0;
+	cfg->num_cos_ldb_ports[2] = 0;
+	cfg->num_cos_ldb_ports[3] = 0;
+
+	switch (handle->cos_id) {
+	case DLB2_COS_0:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[0] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_1:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[1] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_2:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[2] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_3:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[3] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_DEFAULT:
+		/* all ldb ports are don't care ports from a cos perspective */
+		cfg->num_ldb_ports =
+			resources_asked->num_ldb_ports;
+		break;
+	}
+
+	cfg->num_ldb_credits =
+		resources_asked->num_ldb_credits;
+
+	cfg->num_atomic_inflights =
+		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
+		cfg->num_ldb_queues;
+
+	cfg->num_hist_list_entries = resources_asked->num_ldb_ports *
+		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d\n",
+		     cfg->num_ldb_queues,
+		     resources_asked->num_ldb_ports,
+		     cfg->num_dir_ports,
+		     cfg->num_atomic_inflights,
+		     cfg->num_hist_list_entries,
+		     cfg->num_ldb_credits,
+		     cfg->num_dir_credits);
+
+	/* Configure the QM */
+
+	ret = dlb2_iface_sched_domain_create(handle, cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: domain create failed, ret = %d, extra status: %s\n",
+			     ret,
+			     dlb2_error_strings[cfg->response.status]);
+
+		goto error_exit;
+	}
+
+	handle->domain_id = cfg->response.id;
+	handle->cfg.configured = true;
+
+error_exit:
+
+	return ret;
+}
+
+static void
+dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	enum dlb2_configuration_state config_state;
+	int i, j;
+
+	dlb2_iface_domain_reset(dlb2);
+
+	/* Free all dynamically allocated port memory */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
+
+	/* If reconfiguring, mark the device's queues and ports as "previously
+	 * configured." If the user doesn't reconfigure them, the PMD will
+	 * reapply their previous configuration when the device is started.
+	 */
+	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
+		DLB2_NOT_CONFIGURED;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		dlb2->ev_ports[i].qm_port.config_state = config_state;
+		/* Reset setup_done so ports can be reconfigured */
+		dlb2->ev_ports[i].setup_done = false;
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			dlb2->ev_ports[i].link[j].mapped = false;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++)
+		dlb2->ev_queues[i].qm_queue.config_state = config_state;
+
+	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
+		dlb2->ev_queues[i].setup_done = false;
+
+	dlb2->num_ports = 0;
+	dlb2->num_ldb_ports = 0;
+	dlb2->num_dir_ports = 0;
+	dlb2->num_queues = 0;
+	dlb2->num_ldb_queues = 0;
+	dlb2->num_dir_queues = 0;
+	dlb2->configured = false;
+}
+
+/* Note: 1 QM instance per QM device, QM instance/device == event device */
+static int
+dlb2_eventdev_configure(const struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
+	const struct rte_eventdev_data *data = dev->data;
+	const struct rte_event_dev_config *config = &data->dev_conf;
+	int ret;
+
+	/* If this eventdev is already configured, we must release the current
+	 * scheduling domain before attempting to configure a new one.
+	 */
+	if (dlb2->configured) {
+		dlb2_hw_reset_sched_domain(dev, true);
+
+		ret = dlb2_hw_query_resources(dlb2);
+		if (ret) {
+			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+				     ret, data->dev_id);
+			return ret;
+		}
+	}
+
+	if (config->nb_event_queues > rsrcs->num_queues) {
+		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_queues,
+			     rsrcs->num_queues);
+		return -EINVAL;
+	}
+	if (config->nb_event_ports > (rsrcs->num_ldb_ports
+			+ rsrcs->num_dir_ports)) {
+		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_ports,
+			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
+		return -EINVAL;
+	}
+	if (config->nb_events_limit > rsrcs->nb_events_limit) {
+		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_events_limit,
+			     rsrcs->nb_events_limit);
+		return -EINVAL;
+	}
+
+	if (config->event_dev_cfg & RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT)
+		dlb2->global_dequeue_wait = false;
+	else {
+		uint32_t timeout32;
+
+		dlb2->global_dequeue_wait = true;
+
+		/* note size mismatch of timeout vals in eventdev lib. */
+		timeout32 = config->dequeue_timeout_ns;
+
+		dlb2->global_dequeue_wait_ticks =
+			timeout32 * (rte_get_timer_hz() / 1E9);
+	}
+
+	/* Does this platform support umonitor/umwait? */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
+		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
+			DLB2_LOG_ERR("invalid value (%d) for RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
+				     RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
+			return -EINVAL;
+		}
+		dlb2->umwait_allowed = true;
+	}
+
+	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
+	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs->num_dir_ports;
+	/* 1 dir queue per dir port */
+	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports;
+
+	/* Scale down nb_events_limit by 4 for directed credits, since there
+	 * are 4x as many load-balanced credits.
+	 */
+	rsrcs->num_ldb_credits = 0;
+	rsrcs->num_dir_credits = 0;
+
+	if (rsrcs->num_ldb_queues)
+		rsrcs->num_ldb_credits = config->nb_events_limit;
+	if (rsrcs->num_dir_ports)
+		rsrcs->num_dir_credits = config->nb_events_limit / 4;
+	if (dlb2->num_dir_credits_override != -1)
+		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
+
+	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
+		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
+		return -ENODEV;
+	}
+
+	dlb2->new_event_limit = config->nb_events_limit;
+	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
+
+	/* Save number of ports/queues for this event dev */
+	dlb2->num_ports = config->nb_event_ports;
+	dlb2->num_queues = config->nb_event_queues;
+	dlb2->num_dir_ports = rsrcs->num_dir_ports;
+	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
+	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
+	dlb2->num_dir_queues = dlb2->num_dir_ports;
+	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
+	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
+	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
+	dlb2->max_dir_credits = rsrcs->num_dir_credits;
+
+	dlb2->configured = true;
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dev_infos_get    = dlb2_eventdev_info_get,
+		.dev_configure    = dlb2_eventdev_configure,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 0d93faf..a829b9b 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -25,4 +25,9 @@ int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 				   enum dlb2_cq_poll_modes *mode);
 
 int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
-				    struct dlb2_get_num_resources_args *rsrcs);
+				struct dlb2_get_num_resources_args *rsrcs);
+
+int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				struct dlb2_create_sched_domain_args *args);
+
+void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4fb416e..576c1c3 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -26,4 +26,9 @@ extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
 
+extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				 struct dlb2_create_sched_domain_args *args);
+
+extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 6de8b95..fe41e35 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -12,6 +12,24 @@
 #include "dlb2_regs.h"
 #include "dlb2_resource.h"
 
+#define DLB2_DOM_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, domain_list, iter)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, func_list, iter)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, domain_list, it, it_tmp)
+
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, func_list, it, it_tmp)
+
 static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
 {
 	int i;
@@ -272,3 +290,3219 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
 
 	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
 }
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+	union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+	r0.field.count = domain->num_ldb_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+	r1.field.count = domain->num_dir_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+		       struct dlb2_function_resources *rsrcs,
+		       u32 domain_id,
+		       u32 cos_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	RTE_SET_USED(iter);
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+				   typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				   struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_ports,
+				   u32 cos_id,
+				   struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_ldb_port *port;
+
+		port = dlb2_get_next_ldb_port(hw, rsrcs,
+					      domain->id.phys_id, cos_id);
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_ports[cos_id],
+			      &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_ports[cos_id],
+			      &port->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_create_sched_domain_args *args,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      num,
+						      i,
+						      resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+					ret = __dlb2_attach_ldb_ports(hw,
+								      rsrcs,
+								      domain,
+								      1,
+								      cos_id,
+								      resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret < 0)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      1,
+						      j,
+						      resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 u32 num_ports,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+					   typeof(*port));
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_dir_pq_pairs, &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_dir_pq_pairs, &port->domain_list);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+					struct dlb2_hw_domain *domain,
+					u32 num_atomic_inflights,
+					struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+				     struct dlb2_hw_domain *domain,
+				     u32 num_hist_list_entries,
+				     struct dlb2_cmd_response *resp)
+{
+	struct dlb2_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb2_bitmap_find_set_bit_range(bitmap,
+						      num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+				  struct dlb2_function_resources *rsrcs,
+				  struct dlb2_hw_domain *domain,
+				  u32 num_queues,
+				  struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb2_ldb_queue *queue;
+
+		queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+					    typeof(*queue));
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_queues, &queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_queues, &queue->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+			     struct dlb2_function_resources *rsrcs,
+			     struct dlb2_hw_domain *domain,
+			     struct dlb2_create_sched_domain_args *args,
+			     struct dlb2_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb2_attach_ldb_queues(hw,
+				     rsrcs,
+				     domain,
+				     args->num_ldb_queues,
+				     resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_ports(hw,
+				    rsrcs,
+				    domain,
+				    args,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_ports(hw,
+				    rsrcs,
+				    domain,
+				    args->num_dir_ports,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_credits(rsrcs,
+				      domain,
+				      args->num_ldb_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_credits(rsrcs,
+				      domain,
+				      args->num_dir_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+						   domain,
+						   args->num_hist_list_entries,
+						   resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_atomic_inflights(rsrcs,
+					   domain,
+					   args->num_atomic_inflights,
+					   resp);
+	if (ret < 0)
+		return ret;
+
+	dlb2_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+				  struct dlb2_create_sched_domain_args *args,
+				  struct dlb2_cmd_response *resp)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb2_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+				  struct dlb2_create_sched_domain_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		    args->num_ldb_queues);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		    args->num_ldb_ports);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		    args->num_cos_ldb_ports[0]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		    args->cos_strict);
+	DLB2_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		    args->num_dir_ports);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		    args->num_atomic_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		    args->num_hist_list_entries);
+	DLB2_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		    args->num_ldb_credits);
+	DLB2_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		    args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ *	domain and its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+	if (ret)
+		return ret;
+
+	domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available domains\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (domain->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_domains contains configured domains.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	dlb2_init_domain_rsrc_lists(domain);
+
+	ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to verify args.\n",
+			    __func__);
+
+		return ret;
+	}
+
+	dlb2_list_del(&rsrcs->avail_domains, &domain->func_list);
+
+	dlb2_list_add(&rsrcs->used_domains, &domain->func_list);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+	DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+			     struct dlb2_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb2_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		os_enqueue_four_hcws(hw, hcw, pp_addr);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	int ret;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_disable(hw, port);
+
+		ret = dlb2_drain_dir_cq(hw, port);
+		if (ret < 0)
+			return ret;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_dir_pq_pair *queue)
+{
+	union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *queue)
+{
+	return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+				      struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		os_enqueue_four_hcws(hw, hcw, pp_addr);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			os_enqueue_four_hcws(hw, hcw, pp_addr);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int ret, i;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			if (toggle_port)
+				dlb2_ldb_port_cq_disable(hw, port);
+
+			ret = dlb2_drain_ldb_cq(hw, port);
+			if (ret < 0)
+				return ret;
+
+			if (toggle_port)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+
+	return 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_atm_active r1;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_ldb_queue *queue)
+{
+	return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb2_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+					   struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to unmap domain queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = true;
+
+			dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+			   u32 id,
+			   bool vdev_req,
+			   unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iter1) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter2)
+			if (queue->id.virt_id == id)
+				return queue;
+	}
+
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue, iter1)
+		if (queue->id.virt_id == id)
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+						      u32 id,
+						      bool vdev_req,
+						      unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iteration;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	RTE_SET_USED(iteration);
+
+	if (id >= DLB2_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iteration)
+		if (domain->id.virt_id == id)
+			return domain;
+
+	return NULL;
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot,
+					   enum dlb2_qid_map_state new_state)
+{
+	enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb2_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB2_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB2_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB2_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB2_HW_DBG(hw,
+		    "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB2_HW_ERR(hw,
+		    "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+				enum dlb2_qid_map_state state,
+				int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+				      enum dlb2_qid_map_state state,
+				      struct dlb2_ldb_queue *queue,
+				      int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+					      struct dlb2_hw_domain *domain,
+					      struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain,
+					     struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+						struct dlb2_ldb_port *port,
+						int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 0;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 1;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+					struct dlb2_ldb_port *p,
+					struct dlb2_ldb_queue *q,
+					u8 priority)
+{
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_lsp_cq2qid0 r1;
+	union dlb2_atm_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix_00 r3;
+	union dlb2_lsp_qid2cqidix2_00 r4;
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+	    !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+	r0.field.v |= 1 << i;
+	r0.field.prio |= (priority & 0x7) << i * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+	else
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		r1.field.qid_p0 = q->id.phys_id;
+	if (i == 1 || i == 5)
+		r1.field.qid_p1 = q->id.phys_id;
+	if (i == 2 || i == 6)
+		r1.field.qid_p2 = q->id.phys_id;
+	if (i == 3 || i == 7)
+		r1.field.qid_p3 = q->id.phys_id;
+
+	if (i < 4)
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+	else
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r4.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+						  p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		r2.field.cq_p0 |= 1 << i;
+		r3.field.cq_p0 |= 1 << i;
+		r4.field.cq_p0 |= 1 << i;
+		break;
+
+	case 1:
+		r2.field.cq_p1 |= 1 << i;
+		r3.field.cq_p1 |= 1 << i;
+		r4.field.cq_p1 |= 1 << i;
+		break;
+
+	case 2:
+		r2.field.cq_p2 |= 1 << i;
+		r3.field.cq_p2 |= 1 << i;
+		r4.field.cq_p2 |= 1 << i;
+		break;
+
+	case 3:
+		r2.field.cq_p3 |= 1 << i;
+		r3.field.cq_p3 |= 1 << i;
+		r4.field.cq_p3 |= 1 << i;
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r3.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		    r4.val);
+
+	dlb2_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB2_QUEUE_MAPPED;
+
+	return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	/* Set the atomic scheduling haswork bit */
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.rlist_haswork_v = r0.field.count > 0;
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      u8 slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.rlist_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.nalb_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+					      struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+	r0.field.limit = queue->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+						struct dlb2_ldb_queue *queue)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+						struct dlb2_hw_domain *domain,
+						struct dlb2_ldb_port *port,
+						struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_lsp_qid_ldb_infl_cnt r0;
+	enum dlb2_qid_map_state state;
+	int slot, ret, i;
+	u8 prio;
+	RTE_SET_USED(iter);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: non-zero QID inflight count\n",
+			    __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			state = DLB2_QUEUE_MAPPED;
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+					 struct dlb2_ldb_port *port,
+					 struct dlb2_ldb_queue *queue,
+					 u8 priority)
+{
+	union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	int domain_id, slot, ret;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+		DLB2_HW_ERR(hw,
+			    "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		if (port->enabled)
+			dlb2_ldb_port_cq_enable(hw, port);
+
+		dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain,
+					struct dlb2_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		union dlb2_lsp_qid_ldb_infl_cnt r0;
+		struct dlb2_ldb_queue *queue;
+		int qid;
+
+		if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: unable to find queue %d\n",
+				    __func__, qid);
+			continue;
+		}
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count)
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb2_ldb_port_cq_disable(hw, port);
+
+		dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count) {
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+
+			dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port,
+				   struct dlb2_ldb_queue *queue)
+{
+	enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_atm_qid2cqidix_00 r1;
+	union dlb2_lsp_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix2_00 r3;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB2_QUEUE_MAPPED;
+	in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: QID %d isn't mapped\n",
+			    __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+	r0.field.v &= ~(1 << i);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		r1.field.cq_p0 &= ~(1 << i);
+		r2.field.cq_p0 &= ~(1 << i);
+		r3.field.cq_p0 &= ~(1 << i);
+		break;
+
+	case 1:
+		r1.field.cq_p1 &= ~(1 << i);
+		r2.field.cq_p1 &= ~(1 << i);
+		r3.field.cq_p1 &= ~(1 << i);
+		break;
+
+	case 2:
+		r1.field.cq_p2 &= ~(1 << i);
+		r2.field.cq_p2 &= ~(1 << i);
+		r3.field.cq_p2 &= ~(1 << i);
+		break;
+
+	case 3:
+		r1.field.cq_p3 &= ~(1 << i);
+		r2.field.cq_p3 &= ~(1 << i);
+		r3.field.cq_p3 &= ~(1 << i);
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+		    r1.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+		    r3.val);
+
+	dlb2_flush_csr(hw);
+
+	unmapped = DLB2_QUEUE_UNMAPPED;
+
+	return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_ldb_port *port,
+				 struct dlb2_ldb_queue *queue,
+				 u8 prio)
+{
+	if (domain->started)
+		return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   int slot)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb2_ldb_port_qid_map *map;
+		struct dlb2_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain,
+					  struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (r0.field.count > 0)
+		return false;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = false;
+
+			dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_dir_vpp_v r1;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		unsigned int offs;
+		u32 virt_id;
+
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+	}
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_ldb_vpp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			unsigned int offs;
+			u32 virt_id;
+
+			if (hw->virt_mode == DLB2_VIRT_SRIOV)
+				virt_id = port->id.virt_id;
+			else
+				virt_id = port->id.phys_id;
+
+			offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+			DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+	union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+				    r0.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+				    r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+	union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+			    r0.val);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+			    r1.val);
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+		union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+		union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+				    r1.val);
+
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID_V(idx),
+				    r2.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID2QID(idx),
+				    r3.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID_V(idx),
+				    r1.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID2QID(idx),
+				    r2.val);
+		}
+	}
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+					       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_sn_chk_enbl r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.en = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			int i;
+
+			for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+				if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					    __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		port->enabled = false;
+
+		dlb2_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	union dlb2_sys_dir_pp_v r1;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+			    r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_ldb_pp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_ldb_queue *queue;
+	int i;
+	RTE_SET_USED(iter);
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty ldb queue %d\n",
+				    __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port, iter) {
+			if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb2_ldb_cq_token_count(hw, ldb_port)) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to empty ldb port %d\n",
+					    __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir queue %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb2_dir_cq_token_count(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir port %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						   struct dlb2_ldb_port *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP2PP(offs),
+			    DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP_V(offs),
+			    DLB2_SYS_VF_LDB_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    DLB2_SYS_LDB_PP_V_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_DSBL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_BASE_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID0(port->id.phys_id),
+		    DLB2_LSP_CQ2QID0_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID1(port->id.phys_id),
+		    DLB2_LSP_CQ2QID1_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+		    DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			__dlb2_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+				       struct dlb2_dir_pq_pair *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_DSBL_RST);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_FMT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP2PP(offs),
+			    DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP_V(offs),
+			    DLB2_SYS_VF_DIR_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		__dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+			    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_ITS(queue_id),
+			    DLB2_SYS_LDB_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN(queue_id),
+			    DLB2_CHP_ORD_QID_SN_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+			    DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_V(queue_id),
+			    DLB2_SYS_LDB_QID_V_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_CFG_V(queue_id),
+			    DLB2_SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB2_CSR_WR(hw,
+				    offs[queue->sn_group],
+				    DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX2(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX2_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_ATM_QID2CQIDIX(queue_id, i),
+				    DLB2_ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb2_domain_reset_dir_port_registers(hw, domain);
+
+	dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_dir_pq_pair *tmp_dir_port;
+	struct dlb2_ldb_queue *tmp_ldb_queue;
+	struct dlb2_ldb_port *tmp_ldb_port;
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_list_head *list;
+	int ret, i;
+	RTE_SET_USED(tmp_dir_port);
+	RTE_SET_USED(tmp_ldb_queue);
+	RTE_SET_USED(tmp_ldb_port);
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list = &domain->used_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb2_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		dlb2_list_del(&domain->used_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list = &domain->avail_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		ldb_queue->owned = false;
+
+		dlb2_list_del(&domain->avail_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		list = &domain->used_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB2_QUEUE_UNMAPPED;
+
+			dlb2_list_del(&domain->used_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list = &domain->avail_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			ldb_port->owned = false;
+
+			dlb2_list_del(&domain->avail_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list = &domain->used_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		dlb2_list_del(&domain->used_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list = &domain->avail_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				    domain->hist_list_entry_base,
+				    domain->total_hist_list_entries);
+	if (ret) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			    __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	dlb2_list_del(&rsrcs->used_domains, &domain->func_list);
+	dlb2_list_add(&rsrcs->avail_domains, &domain->func_list);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain,
+					    struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_ldb_port *port;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (!dlb2_list_empty(&domain->used_ldb_ports[i]))
+			break;
+	}
+
+	if (i == DLB2_NUM_COS_DOMAINS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: No configured LDB ports\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb2_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+	 * prior to this in dlb2_domain_drain_mapped_queues().
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings != 0 ||
+		    dlb2_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ *	hardware resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain  == NULL || !domain->configured)
+		return -EINVAL;
+
+	/* Disable VPPs */
+	if (vdev_req) {
+		dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+		dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+	}
+
+	/* Disable CQ interrupts */
+	dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+	dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+	dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb2_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_mapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	dlb2_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb2_domain_disable_dir_cqs(hw, domain);
+
+	/* Disable PPs */
+	dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+	ret = dlb2_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Reset the QID and port state. */
+	dlb2_domain_reset_registers(hw, domain);
+
+	/* Hardware reset complete. Reset the domain's software state */
+	ret = dlb2_domain_reset_software_state(hw, domain);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index bd8590d..061ee72 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -613,3 +613,18 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
 
 	return 0;
 }
+
+int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+			    struct dlb2_create_sched_domain_args *args,
+			    struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_sched_domain(hw, args, resp, NOT_VF_REQ,
+					   PF_ID_ZERO);
+}
+
+int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 7d64309..9310f72 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -108,15 +108,59 @@ dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
 	return 0;
 }
 
+static int
+dlb2_pf_sched_domain_create(struct dlb2_hw_dev *handle,
+			    struct dlb2_create_sched_domain_args *arg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (dlb2_dev->domain_reset_failed) {
+		response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = dlb2_pf_create_sched_domain(&dlb2_dev->hw, arg, &response);
+	if (ret)
+		goto done;
+
+done:
+
+	arg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static void
+dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret;
+
+	dlb2_dev = (struct dlb2_dev *)dlb2->qm_instance.pf_dev;
+	ret = dlb2_pf_reset_domain(&dlb2_dev->hw, dlb2->qm_instance.domain_id);
+	if (ret)
+		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
 	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
 	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_domain_reset = dlb2_pf_domain_reset;
 	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
 	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 08/22] event/dlb2: add infos get and configure

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Saturday, October 17, 2020 1:21 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH v2 08/22] event/dlb2: add infos get and configure
> 
> Add support for configuring the DLB2 hardware.
> In particular, this patch configures the DLB2
> hardware's scheduling domain, such that it is provisioned with
> the requested number of ports and queues, provided sufficient
> resources are available. Individual queues and ports are
> configured later in port setup and eventdev start.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v2 09/22] event/dlb2: add queue and port default conf

[Timothy McDaniel]

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 79b23da..1e28640 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -662,12 +662,42 @@ dlb2_eventdev_configure(const struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
+				    uint8_t port_id,
+				    struct rte_event_port_conf *port_conf)
+{
+	RTE_SET_USED(port_id);
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	port_conf->new_event_threshold = dlb2->new_event_limit;
+	port_conf->dequeue_depth = 32;
+	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
+	port_conf->event_port_cfg = 0;
+}
+
+static void
+dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
+				     uint8_t queue_id,
+				     struct rte_event_queue_conf *queue_conf)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(queue_id);
+
+	queue_conf->nb_atomic_flows = 1024;
+	queue_conf->nb_atomic_order_sequences = 64;
+	queue_conf->event_queue_cfg = 0;
+	queue_conf->priority = 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 09/22] event/dlb2: add queue and port default conf

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Saturday, October 17, 2020 1:21 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH v2 09/22] event/dlb2: add queue and port default conf
> 
> Add support for getting the queue and port default configuration.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v2 10/22] event/dlb2: add queue setup

[Timothy McDaniel]

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 314 +++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |  12 +
 drivers/event/dlb2/dlb2_iface.h            |  12 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 464 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  82 +++++
 6 files changed, 894 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 1e28640..bec8fa8 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -689,6 +689,319 @@ dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
 	queue_conf->priority = 0;
 }
 
+static int32_t
+dlb2_get_sn_allocation(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_allocation_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_set_sn_allocation(struct dlb2_eventdev *dlb2, int group, int num)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_set_sn_allocation_args cfg;
+	int ret;
+
+	cfg.num = num;
+	cfg.group = group;
+
+	ret = dlb2_iface_set_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: set_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int32_t
+dlb2_get_sn_occupancy(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_occupancy_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_occupancy(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_occupancy ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+/* Query the current sequence number allocations and, if they conflict with the
+ * requested LDB queue configuration, attempt to re-allocate sequence numbers.
+ * This is best-effort; if it fails, the PMD will attempt to configure the
+ * load-balanced queue and return an error.
+ */
+static void
+dlb2_program_sn_allocation(struct dlb2_eventdev *dlb2,
+			   const struct rte_event_queue_conf *queue_conf)
+{
+	int grp_occupancy[DLB2_NUM_SN_GROUPS];
+	int grp_alloc[DLB2_NUM_SN_GROUPS];
+	int i, sequence_numbers;
+
+	sequence_numbers = (int)queue_conf->nb_atomic_order_sequences;
+
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		int total_slots;
+
+		grp_alloc[i] = dlb2_get_sn_allocation(dlb2, i);
+		if (grp_alloc[i] < 0)
+			return;
+
+		total_slots = DLB2_MAX_LDB_SN_ALLOC / grp_alloc[i];
+
+		grp_occupancy[i] = dlb2_get_sn_occupancy(dlb2, i);
+		if (grp_occupancy[i] < 0)
+			return;
+
+		/* DLB has at least one available slot for the requested
+		 * sequence numbers, so no further configuration required.
+		 */
+		if (grp_alloc[i] == sequence_numbers &&
+		    grp_occupancy[i] < total_slots)
+			return;
+	}
+
+	/* None of the sequence number groups are configured for the requested
+	 * sequence numbers, so we have to reconfigure one of them. This is
+	 * only possible if a group is not in use.
+	 */
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		if (grp_occupancy[i] == 0)
+			break;
+	}
+
+	if (i == DLB2_NUM_SN_GROUPS) {
+		printf("[%s()] No groups with %d sequence_numbers are available or have free slots\n",
+		       __func__, sequence_numbers);
+		return;
+	}
+
+	/* Attempt to configure slot i with the requested number of sequence
+	 * numbers. Ignore the return value -- if this fails, the error will be
+	 * caught during subsequent queue configuration.
+	 */
+	dlb2_set_sn_allocation(dlb2, i, sequence_numbers);
+}
+
+static int32_t
+dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 const struct rte_event_queue_conf *evq_conf)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_queue *queue = &ev_queue->qm_queue;
+	struct dlb2_create_ldb_queue_args cfg;
+	int32_t ret;
+	uint32_t qm_qid;
+	int sched_type = -1;
+
+	if (evq_conf == NULL)
+		return -EINVAL;
+
+	if (evq_conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_ALL_TYPES) {
+		if (evq_conf->nb_atomic_order_sequences != 0)
+			sched_type = RTE_SCHED_TYPE_ORDERED;
+		else
+			sched_type = RTE_SCHED_TYPE_PARALLEL;
+	} else {
+		sched_type = evq_conf->schedule_type;
+	}
+
+	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
+	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
+	cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences;
+
+	if (sched_type != RTE_SCHED_TYPE_ORDERED) {
+		cfg.num_sequence_numbers = 0;
+		cfg.num_qid_inflights = 2048;
+	}
+
+	/* App should set this to the number of hardware flows they want, not
+	 * the overall number of flows they're going to use. E.g. if app is
+	 * using 64 flows and sets compression to 64, best-case they'll get
+	 * 64 unique hashed flows in hardware.
+	 */
+	switch (evq_conf->nb_atomic_flows) {
+	/* Valid DLB2 compression levels */
+	case 64:
+	case 128:
+	case 256:
+	case 512:
+	case (1 * 1024): /* 1K */
+	case (2 * 1024): /* 2K */
+	case (4 * 1024): /* 4K */
+	case (64 * 1024): /* 64K */
+		cfg.lock_id_comp_level = evq_conf->nb_atomic_flows;
+		break;
+	default:
+		/* Invalid compression level */
+		cfg.lock_id_comp_level = 0; /* no compression */
+	}
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else {
+		cfg.depth_threshold = ev_queue->depth_threshold;
+	}
+
+	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create LB event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	qm_qid = cfg.response.id;
+
+	/* Save off queue config for debug, resource lookups, and reconfig */
+	queue->num_qid_inflights = cfg.num_qid_inflights;
+	queue->num_atm_inflights = cfg.num_atomic_inflights;
+
+	queue->sched_type = sched_type;
+	queue->config_state = DLB2_CONFIGURED;
+
+	DLB2_LOG_DBG("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n",
+		     qm_qid,
+		     cfg.num_atomic_inflights,
+		     cfg.num_sequence_numbers,
+		     cfg.num_qid_inflights);
+
+	return qm_qid;
+}
+
+static int
+dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int32_t qm_qid;
+
+	if (queue_conf->nb_atomic_order_sequences)
+		dlb2_program_sn_allocation(dlb2, queue_conf);
+
+	qm_qid = dlb2_hw_create_ldb_queue(dlb2, ev_queue, queue_conf);
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the load-balanced queue\n");
+
+		return qm_qid;
+	}
+
+	dlb2->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int dlb2_num_dir_queues_setup(struct dlb2_eventdev *dlb2)
+{
+	int i, num = 0;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].setup_done &&
+		    dlb2->ev_queues[i].qm_queue.is_directed)
+			num++;
+	}
+
+	return num;
+}
+
+static void
+dlb2_queue_link_teardown(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue)
+{
+	struct dlb2_eventdev_port *ev_port;
+	int i, j;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (!ev_port->link[j].valid ||
+			    ev_port->link[j].queue_id != ev_queue->id)
+				continue;
+
+			ev_port->link[j].valid = false;
+			ev_port->num_links--;
+		}
+	}
+
+	ev_queue->num_links = 0;
+}
+
+static int
+dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
+			  uint8_t ev_qid,
+			  const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_queue *ev_queue;
+	int ret;
+
+	if (queue_conf == NULL)
+		return -EINVAL;
+
+	if (ev_qid >= dlb2->num_queues)
+		return -EINVAL;
+
+	ev_queue = &dlb2->ev_queues[ev_qid];
+
+	ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg &
+		RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+	ev_queue->id = ev_qid;
+	ev_queue->conf = *queue_conf;
+
+	if (!ev_queue->qm_queue.is_directed) {
+		ret = dlb2_eventdev_ldb_queue_setup(dev, ev_queue, queue_conf);
+	} else {
+		/* The directed queue isn't setup until link time, at which
+		 * point we know its directed port ID. Directed queue setup
+		 * will only fail if this queue is already setup or there are
+		 * no directed queues left to configure.
+		 */
+		ret = 0;
+
+		ev_queue->qm_queue.config_state = DLB2_NOT_CONFIGURED;
+
+		if (ev_queue->setup_done ||
+		    dlb2_num_dir_queues_setup(dlb2) == dlb2->num_dir_queues)
+			ret = -EINVAL;
+	}
+
+	/* Tear down pre-existing port->queue links */
+	if (!ret && dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_queue_link_teardown(dlb2, ev_queue);
+
+	if (!ret)
+		ev_queue->setup_done = true;
+
+	return ret;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -697,6 +1010,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a829b9b..4c07574 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -31,3 +31,15 @@ int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				struct dlb2_create_sched_domain_args *args);
 
 void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_ldb_queue_args *cfg);
+
+int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_sn_allocation_args *args);
+
+int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_set_sn_allocation_args *args);
+
+int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_occupancy_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 576c1c3..c1ef7c2 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -31,4 +31,16 @@ extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 
 extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
 
+extern int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_queue_args *cfg);
+
+extern int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_allocation_args *args);
+
+extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_set_sn_allocation_args *args);
+
+extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				  struct dlb2_get_sn_occupancy_args *args);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index fe41e35..2d11f69 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3506,3 +3506,467 @@ unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
 
 	return num;
 }
+
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_ldb_queue *queue,
+				     struct dlb2_create_ldb_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+	union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+	union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+	union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+	union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+	union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+	union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+	union dlb2_sys_ldb_qid_its r7 = { {0} };
+	union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+	union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+	union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+	union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+	union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+	union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+	struct dlb2_sn_group *sn_group;
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r3.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	r4.field.limit = args->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+	r5.field.limit = queue->aqed_limit;
+
+	if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+		r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+		    r5.val);
+
+	switch (args->lock_id_comp_level) {
+	case 64:
+		r6.field.compress_code = 1;
+		break;
+	case 128:
+		r6.field.compress_code = 2;
+		break;
+	case 256:
+		r6.field.compress_code = 3;
+		break;
+	case 512:
+		r6.field.compress_code = 4;
+		break;
+	case 1024:
+		r6.field.compress_code = 5;
+		break;
+	case 2048:
+		r6.field.compress_code = 6;
+		break;
+	case 4096:
+		r6.field.compress_code = 7;
+		break;
+	case 0:
+	case 65536:
+		r6.field.compress_code = 0;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+		    r6.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r7.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+		    r7.val);
+
+	r8.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+		    r8.val);
+
+	r9.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+		    r9.val);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	r10.field.qid_fid_limit = 512;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+		    r10.val);
+
+	/* Configure SNs */
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	r11.field.mode = sn_group->mode;
+	r11.field.slot = queue->sn_slot;
+	r11.field.grp  = sn_group->id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+	r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+	r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		r0.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+		r1.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+		r2.field.vqid = queue->id.virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+			    r2.val);
+	}
+
+	r13.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+				  struct dlb2_ldb_queue *queue,
+				  struct dlb2_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb2_sn_group_full(group)) {
+			slot = dlb2_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no sequence number slots available\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+				struct dlb2_hw_domain *domain,
+				struct dlb2_ldb_queue *queue,
+				struct dlb2_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_ldb_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (dlb2_list_empty(&domain->avail_ldb_queues)) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb2_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_ldb_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		    args->num_sequence_numbers);
+	DLB2_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		    args->num_qid_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		    args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+
+	dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			    __func__, __LINE__);
+		return ret;
+	}
+
+	dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_queues, &queue->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_queues, &queue->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+						unsigned int group_id,
+						unsigned long val)
+{
+	DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+	DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+	DLB2_HW_DBG(hw, "\tValue:    %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val)
+{
+	u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+	union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+	struct dlb2_sn_group *group;
+	int mode;
+
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	group = &hw->rsrcs.sn_groups[group_id];
+
+	/*
+	 * Once the first load-balanced queue using an SN group is configured,
+	 * the group cannot be changed.
+	 */
+	if (group->slot_use_bitmap != 0)
+		return -EPERM;
+
+	for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+		if (val == valid_allocations[mode])
+			break;
+
+	if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+		return -EINVAL;
+
+	group->mode = mode;
+	group->sequence_numbers_per_queue = val;
+
+	r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+	r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+	DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+	dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 061ee72..3a938b4 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -628,3 +628,13 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
 {
 	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_ldb_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 9310f72..a6cd178 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -150,6 +150,84 @@ dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
 		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
 }
 
+static int
+dlb2_pf_ldb_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_ldb_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_ldb_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw_dev *handle,
+			 struct dlb2_get_sn_occupancy_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_number_occupancy(&dlb2_dev->hw,
+						       args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_numbers(&dlb2_dev->hw, args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_set_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_set_group_sequence_numbers(&dlb2_dev->hw, args->group,
+					      args->num);
+
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -161,6 +239,10 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
+	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
+	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
+	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 10/22] event/dlb2: add queue setup

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Saturday, October 17, 2020 1:21 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH v2 10/22] event/dlb2: add queue setup
> 
> Load balanced (ldb) queues are setup here.
> Directed queues are not set up until link time, at which
> point we know the directed port ID. Directed queue setup
> will only fail if this queue is already setup or there are
> no directed queues left to configure.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v2 11/22] event/dlb2: add port setup

[Timothy McDaniel]

Configure the load balanded (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 501 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   9 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 922 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  37 +-
 drivers/event/dlb2/pf/dlb2_main.h          |   2 -
 drivers/event/dlb2/pf/dlb2_pf.c            | 181 ++++++
 7 files changed, 1649 insertions(+), 11 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index bec8fa8..e956a7a 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -103,6 +103,9 @@ dlb2_free_qe_mem(struct dlb2_port *qm_port)
 
 	rte_free(qm_port->consume_qe);
 	qm_port->consume_qe = NULL;
+
+	rte_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base);
+	rte_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].pp_addr);
 }
 
 /* override defaults with value(s) provided on command line */
@@ -1002,6 +1005,503 @@ dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
 	return ret;
 }
 
+static int
+dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_cq_pop_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_cq_pop_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL)	{
+		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->consume_qe = qe;
+
+	qe->qe_valid = 0;
+	qe->qe_frag = 0;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
+	 * and so on.
+	 */
+	qe->tokens = 0;	/* set at run time */
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_enqueue_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_enqueue_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL) {
+		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->int_arm_qe = qe;
+
+	/* V2 - INT ARM is CQ_TOKEN + FRAG */
+	qe->qe_valid = 0;
+	qe->qe_frag = 1;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
+{
+	int ret, sz;
+
+	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct dlb2_enqueue_qe);
+
+	qm_port->qe4 = rte_zmalloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
+
+	if (qm_port->qe4 == NULL) {
+		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	ret = dlb2_init_consume_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	return 0;
+
+error_exit:
+
+	dlb2_free_qe_mem(qm_port);
+
+	return ret;
+}
+
+static int
+dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_ldb_port_args cfg = {0};
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_CQ_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	cfg.cq_history_list_size = DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	if (handle->cos_id == DLB2_COS_DEFAULT)
+		cfg.cos_id = 0;
+	else
+		cfg.cos_id = handle->cos_id;
+
+	cfg.cos_strict = 0;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* If there are no directed ports, the kernel driver will ignore this
+	 * port's directed credit settings. Don't use enqueue_depth if it would
+	 * require more directed credits than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2; /* back ptr */
+	/*
+	 * Allocate and init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_ldb_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(qm_port->cfg.ldb));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     qm_port->ldb_credits,
+		     qm_port->dir_credits);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
+
+	return ret;
+}
+
+static void
+dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port)
+{
+	struct dlb2_eventdev_queue *ev_queue;
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (!ev_port->link[i].valid)
+			continue;
+
+		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
+
+		ev_port->link[i].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+}
+
+static int
+dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_port_args cfg = {0};
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (dlb2 == NULL || handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* Directed queues are configured at link time. */
+	cfg.queue_id = -1;
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* Don't use enqueue_depth if it would require more directed credits
+	 * than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2;  /* back ptr */
+
+	/*
+	 * Init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE to be aligned
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_dir_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = cfg.cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(qm_port->cfg.dir));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     dir_credit_high_watermark,
+		     ldb_credit_high_watermark);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_setup(struct rte_eventdev *dev,
+			 uint8_t ev_port_id,
+			 const struct rte_event_port_conf *port_conf)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_eventdev_port *ev_port;
+	int ret;
+
+	if (dev == NULL || port_conf == NULL) {
+		DLB2_LOG_ERR("Null parameter\n");
+		return -EINVAL;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
+		return -EINVAL;
+
+	if (port_conf->dequeue_depth >
+		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
+	    port_conf->enqueue_depth >
+		evdev_dlb2_default_info.max_event_port_enqueue_depth)
+		return -EINVAL;
+
+	ev_port = &dlb2->ev_ports[ev_port_id];
+	/* configured? */
+	if (ev_port->setup_done) {
+		DLB2_LOG_ERR("evport %d is already configured\n", ev_port_id);
+		return -EINVAL;
+	}
+
+	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
+		RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (!ev_port->qm_port.is_directed) {
+		ret = dlb2_hw_create_ldb_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the lB port ve portId=%d\n",
+				     ev_port_id);
+
+			return ret;
+		}
+	} else {
+		ret = dlb2_hw_create_dir_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the DIR port\n");
+			return ret;
+		}
+	}
+
+	/* Save off port config for reconfig */
+	ev_port->conf = *port_conf;
+
+	ev_port->id = ev_port_id;
+	ev_port->enq_configured = true;
+	ev_port->setup_done = true;
+	ev_port->inflight_max = port_conf->new_event_threshold;
+	ev_port->implicit_release = !(port_conf->event_port_cfg &
+		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
+	ev_port->outstanding_releases = 0;
+	ev_port->inflight_credits = 0;
+	ev_port->credit_update_quanta = RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
+	ev_port->dlb2 = dlb2; /* reverse link */
+
+	/* Tear down pre-existing port->queue links */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
+
+	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1012,6 +1512,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
+		.port_setup       = dlb2_eventdev_port_setup,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 4c07574..1690d63 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -43,3 +43,12 @@ int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				   struct dlb2_get_sn_occupancy_args *args);
+
+int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_dir_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index c1ef7c2..a090a54 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -43,4 +43,12 @@ extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				  struct dlb2_get_sn_occupancy_args *args);
 
+extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_ldb_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_dir_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 2d11f69..1a0f113 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3970,3 +3970,925 @@ int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_ldb_port *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_pp2vas r0 = { {0} };
+	union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+		union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_ldb_port *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_ldb_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+	union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+	union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+	union dlb2_chp_hist_list_lim r5 = { {0} };
+	union dlb2_chp_hist_list_base r6 = { {0} };
+	union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+	union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+	union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+	union dlb2_sys_ldb_cq_at r10 = { {0} };
+	union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+	union dlb2_chp_ldb_cq2vas r12 = { {0} };
+	union dlb2_lsp_cq2priov r13 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r14.field.token_count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    r14.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	r5.field.limit = port->hist_list_entry_limit - 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+	r6.field.base = port->hist_list_entry_base;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	r7.field.limit = args->cq_history_list_size;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+	r8.field.push_ptr = r6.field.base;
+	r8.field.generation = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    r8.val);
+
+	r9.field.pop_ptr = r6.field.base;
+	r9.field.generation = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+	/* Disable the port's QID mappings */
+	r13.field.v = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+	return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_ldb_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb2_ldb_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+	if (ret < 0)
+		return ret;
+
+	dlb2_ldb_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_ldb_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		    args->cq_history_list_size);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+	DLB2_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		    args->cos_strict);
+}
+
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_ldb_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+		resp->status = DLB2_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		if (dlb2_list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	} else {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			if (!dlb2_list_empty(&domain->avail_ldb_ports[i]))
+				break;
+		}
+
+		if (i == DLB2_NUM_COS_DOMAINS) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ *	its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	int ret, cos_id, i;
+
+	dlb2_log_create_ldb_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->cos_strict) {
+		cos_id = args->cos_id;
+
+		port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+					  typeof(*port));
+	} else {
+		int idx;
+
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+			port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+						  typeof(*port));
+			if (port)
+				break;
+		}
+
+		cos_id = idx;
+	}
+
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (port->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_ldb_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_ports[cos_id], &port->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_ports[cos_id], &port->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_dir_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+}
+
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+			    bool vdev_req,
+			    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+
+	return NULL;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_dir_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the queue is already configured, validate
+	 * the queue ID, its domain, and whether the queue is configured.
+	 */
+	if (args->queue_id != -1) {
+		struct dlb2_dir_pq_pair *queue;
+
+		queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+						    vdev_req,
+						    domain);
+
+		if (queue == NULL || queue->domain_id.phys_id !=
+				domain->id.phys_id ||
+				!queue->queue_configured) {
+			resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the port's queue is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->queue_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_dir_pq_pair *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_dir_pp2vas r0 = { {0} };
+	union dlb2_sys_dir_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+		union dlb2_sys_dir_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_dir_pq_pair *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_dir_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+	union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+	union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+	union dlb2_sys_dir_cq_fmt r9 = { {0} };
+	union dlb2_sys_dir_cq_at r10 = { {0} };
+	union dlb2_sys_dir_cq_pasid r11 = { {0} };
+	union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r13.field.count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    r13.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.disable_wb_opt = 0;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	r9.field.keep_pf_ppid = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+	return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_dir_pq_pair *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_dir_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb2_dir_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+
+	if (ret < 0)
+		return ret;
+
+	dlb2_dir_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ *	and queue. The port/queue pair have the same ID and name.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *port;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (args->queue_id != -1)
+		port = dlb2_get_domain_used_dir_pq(args->queue_id,
+						   vdev_req,
+						   domain);
+	else
+		port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					  typeof(*port));
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_dir_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs, &port->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs, &port->domain_list);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 3a938b4..ec6650e 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -128,20 +128,11 @@ static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
 static int
 dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
 {
-	int i;
-
 	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIR64B))
 		dlb2_dev->enqueue_four = dlb2_movdir64b;
 	else
 		dlb2_dev->enqueue_four = dlb2_movntdq;
 
-	/* Initialize software state */
-	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++)
-		dlb2_list_init_head(&dlb2_dev->ldb_port_pages[i].list);
-
-	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++)
-		dlb2_list_init_head(&dlb2_dev->dir_port_pages[i].list);
-
 	rte_spinlock_init(&dlb2_dev->resource_mutex);
 	rte_spinlock_init(&dlb2_dev->measurement_lock);
 
@@ -638,3 +629,31 @@ dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_ldb_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
+
+int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_dir_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index ec96f11..674aad0 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -33,8 +33,6 @@ struct dlb2_dev {
 	struct dlb2_hw hw;
 	/* struct list_head list; */
 	struct device *dlb2_device;
-	struct dlb2_port_memory ldb_port_pages[DLB2_MAX_NUM_LDB_PORTS];
-	struct dlb2_port_memory dir_port_pages[DLB2_MAX_NUM_DIR_PORTS];
 	/* The enqueue_four function enqueues four HCWs (one cache-line worth)
 	 * to the DLB2, using whichever mechanism is supported by the platform
 	 * on which this driver is running.
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a6cd178..b5fc030 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -228,6 +228,185 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static void *
+dlb2_alloc_coherent_aligned(const struct rte_memzone **mz, uintptr_t *phys,
+			    size_t size, int align)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t core_id = rte_lcore_id();
+	unsigned int socket_id;
+
+	snprintf(mz_name, sizeof(mz_name) - 1, "event_dlb2_pf_%lx",
+		 (unsigned long)rte_get_timer_cycles());
+	if (core_id == (unsigned int)LCORE_ID_ANY)
+		core_id = rte_get_master_lcore();
+	socket_id = rte_lcore_to_socket_id(core_id);
+	*mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
+					 RTE_MEMZONE_IOVA_CONTIG, align);
+	if (*mz == NULL) {
+		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu bytes - %s\n",
+			     size, rte_strerror(rte_errno));
+		*phys = 0;
+		return NULL;
+	}
+	*phys = (*mz)->iova;
+	return (*mz)->addr;
+}
+
+static int
+dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_ldb_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret, cq_alloc_depth;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = false;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	cq_alloc_depth = RTE_MAX(cfg->cq_depth, DLB2_MIN_HARDWARE_CQ_DEPTH);
+	alloc_sz = cq_alloc_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].cq_base = (void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_free(port_base);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+	return ret;
+}
+
+static int
+dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_dir_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = true;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	alloc_sz = cfg->cq_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
+		(void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_free(port_base);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -240,6 +419,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 11/22] event/dlb2: add port setup

[Eads, Gage]

[...]

> +static int
> +dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
> +			struct dlb2_create_ldb_port_args *cfg,
> +			enum dlb2_cq_poll_modes poll_mode)
> +{
> +	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> +	struct dlb2_cmd_response response = {0};
> +	struct dlb2_port_memory port_memory;
> +	int ret, cq_alloc_depth;
> +	uint8_t *port_base;
> +	const struct rte_memzone *mz;
> +	int alloc_sz, qe_sz;
> +	phys_addr_t cq_base;
> +	phys_addr_t pp_base;
> +	int is_dir = false;
> +
> +	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
> +
> +	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
> +		qe_sz = sizeof(struct dlb2_dequeue_qe);
> +	else
> +		qe_sz = RTE_CACHE_LINE_SIZE;
> +
> +	/* Calculate the port memory required, and round up to the nearest
> +	 * cache line.
> +	 */
> +	cq_alloc_depth = RTE_MAX(cfg->cq_depth,
> DLB2_MIN_HARDWARE_CQ_DEPTH);
> +	alloc_sz = cq_alloc_depth * qe_sz;
> +	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
> +
> +	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
> +						PAGE_SIZE);
> +	if (port_base == NULL)
> +		return -ENOMEM;
> +
> +	/* Lock the page in memory */
> +	ret = rte_mem_lock_page(port_base);
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device
> i/o\n");
> +		goto create_port_err;
> +	}
> +
> +

Nit: extra newline

> +	memset(port_base, 0, alloc_sz);
> +
> +	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
> +				      handle->domain_id,
> +				      cfg,
> +				      cq_base,
> +				      &response);
> +	if (ret)
> +		goto create_port_err;
> +
> +	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
> +	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
> +		(void *)(pp_base + (PAGE_SIZE * response.id));
> +
> +	dlb2_port[response.id][DLB2_LDB_PORT].cq_base = (void *)(port_base);
> +	memset(&port_memory, 0, sizeof(port_memory));
> +
> +	dlb2_port[response.id][DLB2_LDB_PORT].mz = mz;
> +
> +	dlb2_list_init_head(&port_memory.list);
> +
> +	cfg->response = response;
> +
> +	return 0;
> +
> +create_port_err:
> +
> +	rte_free(port_base);

This should be "rte_memzone_free(mz);"

> +
> +	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
> +		  __func__, ret);
> +	return ret;
> +}
> +
> +static int
> +dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
> +			struct dlb2_create_dir_port_args *cfg,
> +			enum dlb2_cq_poll_modes poll_mode)
> +{
> +	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> +	struct dlb2_cmd_response response = {0};
> +	struct dlb2_port_memory port_memory;
> +	int ret;
> +	uint8_t *port_base;
> +	const struct rte_memzone *mz;
> +	int alloc_sz, qe_sz;
> +	phys_addr_t cq_base;
> +	phys_addr_t pp_base;
> +	int is_dir = true;
> +
> +	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
> +
> +	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
> +		qe_sz = sizeof(struct dlb2_dequeue_qe);
> +	else
> +		qe_sz = RTE_CACHE_LINE_SIZE;
> +
> +	/* Calculate the port memory required, and round up to the nearest
> +	 * cache line.
> +	 */
> +	alloc_sz = cfg->cq_depth * qe_sz;
> +	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
> +
> +	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
> +						PAGE_SIZE);
> +	if (port_base == NULL)
> +		return -ENOMEM;
> +
> +	/* Lock the page in memory */
> +	ret = rte_mem_lock_page(port_base);
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device
> i/o\n");
> +		goto create_port_err;
> +	}
> +
> +	memset(port_base, 0, alloc_sz);
> +
> +	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
> +				      handle->domain_id,
> +				      cfg,
> +				      cq_base,
> +				      &response);
> +	if (ret)
> +		goto create_port_err;
> +
> +	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
> +	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
> +		(void *)(pp_base + (PAGE_SIZE * response.id));
> +
> +	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
> +		(void *)(port_base);
> +	memset(&port_memory, 0, sizeof(port_memory));
> +
> +	dlb2_port[response.id][DLB2_DIR_PORT].mz = mz;
> +
> +	dlb2_list_init_head(&port_memory.list);
> +
> +	cfg->response = response;
> +
> +	return 0;
> +
> +create_port_err:
> +
> +	rte_free(port_base);

Ditto

Thanks,
Gage

[dpdk-dev] [PATCH v2 12/22] event/dlb2: add port link

[Timothy McDaniel]

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 308 +++++++++++++-
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 633 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
 6 files changed, 1009 insertions(+), 4 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index e956a7a..b448f59 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -828,9 +828,8 @@ dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
 			sched_type = RTE_SCHED_TYPE_ORDERED;
 		else
 			sched_type = RTE_SCHED_TYPE_PARALLEL;
-	} else {
+	} else
 		sched_type = evq_conf->schedule_type;
-	}
 
 	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
 	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
@@ -866,9 +865,8 @@ dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
 	if (ev_queue->depth_threshold == 0) {
 		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
 		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
-	} else {
+	} else
 		cfg.depth_threshold = ev_queue->depth_threshold;
-	}
 
 	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
 	if (ret < 0) {
@@ -1502,6 +1500,307 @@ dlb2_eventdev_port_setup(struct rte_eventdev *dev,
 	return 0;
 }
 
+static int16_t
+dlb2_hw_map_ldb_qid_to_port(struct dlb2_hw_dev *handle,
+			    uint32_t qm_port_id,
+			    uint16_t qm_qid,
+			    uint8_t priority)
+{
+	struct dlb2_map_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	/* Build message */
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+	cfg.priority = EV_TO_DLB2_PRIO(priority);
+
+	ret = dlb2_iface_map_qid(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: map qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		DLB2_LOG_ERR("dlb2: grp=%d, qm_port=%d, qm_qid=%d prio=%d\n",
+			     handle->domain_id, cfg.port_id,
+			     cfg.qid,
+			     cfg.priority);
+	} else {
+		DLB2_LOG_DBG("dlb2: mapped queue %d to qm_port %d\n",
+			     qm_qid, qm_port_id);
+	}
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port,
+			  struct dlb2_eventdev_queue *ev_queue,
+			  uint8_t priority)
+{
+	int first_avail = -1;
+	int ret, i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid) {
+			if (ev_port->link[i].queue_id == ev_queue->id &&
+			    ev_port->link[i].priority == priority) {
+				if (ev_port->link[i].mapped)
+					return 0; /* already mapped */
+				first_avail = i;
+			}
+		} else if (first_avail == -1)
+			first_avail = i;
+	}
+	if (first_avail == -1) {
+		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
+			     ev_port->qm_port.id);
+		return -EINVAL;
+	}
+
+	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
+					  ev_port->qm_port.id,
+					  ev_queue->qm_queue.id,
+					  priority);
+
+	if (!ret)
+		ev_port->link[first_avail].mapped = true;
+
+	return ret;
+}
+
+static int32_t
+dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 int32_t qm_port_id)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_queue_args cfg;
+	int32_t ret;
+
+	/* The directed port is always configured before its queue */
+	cfg.port_id = qm_port_id;
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_dir_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	int32_t qm_qid;
+
+	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port->qm_port.id);
+
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the DIR queue\n");
+		return qm_qid;
+	}
+
+	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int
+dlb2_do_port_link(struct rte_eventdev *dev,
+		  struct dlb2_eventdev_queue *ev_queue,
+		  struct dlb2_eventdev_port *ev_port,
+		  uint8_t prio)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int err;
+
+	/* Don't link until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	if (ev_queue->qm_queue.is_directed)
+		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue, ev_port);
+	else
+		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue, prio);
+
+	if (err) {
+		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
+			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
+			     ev_queue->id, ev_port->id);
+
+		rte_errno = err;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
+			uint8_t queue_id,
+			bool link_exists,
+			int index)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	bool port_is_dir, queue_is_dir;
+
+	if (queue_id > dlb2->num_queues) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	ev_queue = &dlb2->ev_queues[queue_id];
+
+	if (!ev_queue->setup_done &&
+	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	port_is_dir = ev_port->qm_port.is_directed;
+	queue_is_dir = ev_queue->qm_queue.is_directed;
+
+	if (port_is_dir != queue_is_dir) {
+		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
+			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
+			     port_is_dir ? "DIR" : "LDB", ev_port->id);
+
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if there is space for the requested link */
+	if (!link_exists && index == -1) {
+		DLB2_LOG_ERR("no space for new link\n");
+		rte_errno = -ENOSPC;
+		return -1;
+	}
+
+	/* Check if the directed port is already linked */
+	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if the directed queue is already linked */
+	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
+			     ev_queue->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
+			const uint8_t queues[], const uint8_t priorities[],
+			uint16_t nb_links)
+
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i, j;
+
+	RTE_SET_USED(dev);
+
+	if (ev_port == NULL) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!ev_port->setup_done &&
+	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
+	 * queues pointer.
+	 */
+	if (nb_links == 0) {
+		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
+		     nb_links,
+		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
+		     ev_port->id);
+
+	for (i = 0; i < nb_links; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		uint8_t queue_id, prio;
+		bool found = false;
+		int index = -1;
+
+		queue_id = queues[i];
+		prio = priorities[i];
+
+		/* Check if the link already exists. */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].valid) {
+				if (ev_port->link[j].queue_id == queue_id) {
+					found = true;
+					index = j;
+					break;
+				}
+			} else if (index == -1) {
+				index = j;
+			}
+
+		/* could not link */
+		if (index == -1)
+			break;
+
+		/* Check if already linked at the requested priority */
+		if (found && ev_port->link[j].priority == prio)
+			continue;
+
+		if (dlb2_validate_port_link(ev_port, queue_id, found, index))
+			break; /* return index of offending queue */
+
+		ev_queue = &dlb2->ev_queues[queue_id];
+
+		if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+			break; /* return index of offending queue */
+
+		ev_queue->num_links++;
+
+		ev_port->link[index].queue_id = queue_id;
+		ev_port->link[index].priority = prio;
+		ev_port->link[index].valid = true;
+		/* Entry already exists?  If so, then must be prio change */
+		if (!found)
+			ev_port->num_links++;
+	}
+	return i;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1513,6 +1812,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_link        = dlb2_eventdev_port_link,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 1690d63..5f45b97 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -52,3 +52,9 @@ int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 				  struct dlb2_create_dir_port_args *cfg,
 				  enum dlb2_cq_poll_modes poll_mode);
 
+int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_dir_queue_args *cfg);
+
+int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+			  struct dlb2_map_qid_args *cfg);
+
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index a090a54..90f7b8a 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -51,4 +51,10 @@ extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 struct dlb2_create_dir_port_args *cfg,
 					 enum dlb2_cq_poll_modes poll_mode);
 
+extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+					struct dlb2_create_dir_queue_args *cfg);
+
+extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+				 struct dlb2_map_qid_args *cfg);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 1a0f113..8eb0afb 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -4892,3 +4892,636 @@ int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_dir_pq_pair *queue,
+				     struct dlb2_create_dir_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_dir_vasqid_v r0 = { {0} };
+	union dlb2_sys_dir_qid_its r1 = { {0} };
+	union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+	union dlb2_sys_dir_qid_v r5 = { {0} };
+
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r0.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r1.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+		    r1.val);
+
+	r2.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+		    r2.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		r3.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+		r4.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+	}
+
+	r5.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+	queue->queue_configured = true;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_dir_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_dir_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = dlb2_get_domain_used_dir_pq(args->port_id,
+						   vdev_req,
+						   domain);
+
+		if (port == NULL || port->domain_id.phys_id !=
+				domain->id.phys_id || !port->port_configured) {
+			resp->status = DLB2_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the queue's port is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->port_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->port_id != -1)
+		queue = dlb2_get_domain_used_dir_pq(args->port_id,
+						    vdev_req,
+						    domain);
+	else
+		queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					   typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &queue->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs,
+			      &queue->domain_list);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot,
+					      struct dlb2_map_qid_args *args)
+{
+	union dlb2_lsp_cq2priov r0;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+	r0.field.v |= 1 << slot;
+	r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+	dlb2_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+					      struct dlb2_ldb_queue *queue,
+					      struct dlb2_cmd_response *resp)
+{
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB2_QUEUE_UNMAPPED;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+			  bool vdev_req,
+			  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter)
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+			      bool vdev_req,
+			      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+
+		DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+	}
+
+	return NULL;
+}
+
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+				    u32 domain_id,
+				    struct dlb2_map_qid_args *args,
+				    struct dlb2_cmd_response *resp,
+				    bool vdev_req,
+				    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	struct dlb2_ldb_queue *queue;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB2_QID_PRIORITIES) {
+		resp->status = DLB2_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_map_qid_args *args,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	DLB2_HW_DBG(hw, "\tPriority:  %d\n",
+		    args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	int ret, i, id;
+	u8 prio;
+
+	dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_map_qid_args(hw,
+				       domain_id,
+				       args,
+				       resp,
+				       vdev_req,
+				       vdev_id);
+	if (ret)
+		return ret;
+
+	prio = args->priority;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	if (port->num_pending_removals)
+		dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].pending_priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+		if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb2_qid_map_state st;
+
+			if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+				DLB2_HW_ERR(hw,
+					    "[%s():%d] Internal error: port slot tracking failed\n",
+					    __func__, __LINE__);
+				return -EFAULT;
+			}
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb2_port_slot_state_transition(hw, port, queue,
+							      i, st);
+			if (ret)
+				return ret;
+
+			DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index ec6650e..d5f3a82 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -657,3 +657,13 @@ dlb2_pf_create_dir_port(struct dlb2_hw *hw,
 				       NOT_VF_REQ,
 				       PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_dir_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index b5fc030..ad1b0c6 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -407,6 +407,54 @@ dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_dir_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_dir_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_dir_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
+		struct dlb2_map_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_map_qid(&dlb2_dev->hw,
+			      handle->domain_id,
+			      cfg,
+			      &response,
+			      false,
+			      0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -420,7 +468,9 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
 	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
+	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 12/22] event/dlb2: add port link

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Saturday, October 17, 2020 1:21 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH v2 12/22] event/dlb2: add port link
> 
> Add port link entry point. Directed queues are identified and created
> at this stage. Their setup deferred until link-time, at which
> point we know the directed port ID. Directed queue setup
> will only fail if this queue is already setup or there are
> no directed queues left to configure.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  drivers/event/dlb2/dlb2.c                  | 308 +++++++++++++-
>  drivers/event/dlb2/dlb2_iface.c            |   6 +
>  drivers/event/dlb2/dlb2_iface.h            |   6 +
>  drivers/event/dlb2/pf/base/dlb2_resource.c | 633
> +++++++++++++++++++++++++++++
>  drivers/event/dlb2/pf/dlb2_main.c          |  10 +
>  drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
>  6 files changed, 1009 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> index e956a7a..b448f59 100644
> --- a/drivers/event/dlb2/dlb2.c
> +++ b/drivers/event/dlb2/dlb2.c
> @@ -828,9 +828,8 @@ dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
>  			sched_type = RTE_SCHED_TYPE_ORDERED;
>  		else
>  			sched_type = RTE_SCHED_TYPE_PARALLEL;
> -	} else {
> +	} else
>  		sched_type = evq_conf->schedule_type;
> -	}
> 
>  	cfg.num_atomic_inflights =
> DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
>  	cfg.num_sequence_numbers = evq_conf-
> >nb_atomic_order_sequences;
> @@ -866,9 +865,8 @@ dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
>  	if (ev_queue->depth_threshold == 0) {
>  		cfg.depth_threshold =
> RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
>  		ev_queue->depth_threshold =
> RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
> -	} else {
> +	} else
>  		cfg.depth_threshold = ev_queue->depth_threshold;
> -	}

These two changes should be in patch 10 ("event/dlb2: add queue setup"). Besides that:

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v2 13/22] event/dlb2: add port unlink and port unlinks in progress

[Timothy McDaniel]

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 163 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   5 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +-
 drivers/event/dlb2/pf/base/dlb2_resource.c | 283 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  51 ++++++
 5 files changed, 509 insertions(+), 1 deletion(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index b448f59..11fc88b 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1801,6 +1801,166 @@ dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
 	return i;
 }
 
+static int16_t
+dlb2_hw_unmap_ldb_qid_from_port(struct dlb2_hw_dev *handle,
+				uint32_t qm_port_id,
+				uint16_t qm_qid)
+{
+	struct dlb2_unmap_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+
+	ret = dlb2_iface_unmap_qid(handle, &cfg);
+	if (ret < 0)
+		DLB2_LOG_ERR("dlb2: unmap qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_detach_ldb(struct dlb2_eventdev *dlb2,
+			    struct dlb2_eventdev_port *ev_port,
+			    struct dlb2_eventdev_queue *ev_queue)
+{
+	int ret, i;
+
+	/* Don't unlink until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid &&
+		    ev_port->link[i].queue_id == ev_queue->id)
+			break; /* found */
+	}
+
+	/* This is expected with eventdev API!
+	 * It blindly attemmpts to unmap all queues.
+	 */
+	if (i == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_LOG_DBG("dlb2: ignoring LB QID %d not mapped for qm_port %d.\n",
+			     ev_queue->qm_queue.id,
+			     ev_port->qm_port.id);
+		return 0;
+	}
+
+	ret = dlb2_hw_unmap_ldb_qid_from_port(&dlb2->qm_instance,
+					      ev_port->qm_port.id,
+					      ev_queue->qm_queue.id);
+	if (!ret)
+		ev_port->link[i].mapped = false;
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
+			  uint8_t queues[], uint16_t nb_unlinks)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (queues == NULL || nb_unlinks == 0) {
+		DLB2_LOG_DBG("dlb2: queues is NULL or nb_unlinks is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	/* FIXME: How to handle unlink on a directed port? */
+	if (ev_port->qm_port.is_directed) {
+		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
+			     ev_port->id);
+		rte_errno = 0;
+		return nb_unlinks; /* as if success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	for (i = 0; i < nb_unlinks; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		int ret, j;
+
+		if (queues[i] >= dlb2->num_queues) {
+			DLB2_LOG_ERR("dlb2: invalid queue id %d\n", queues[i]);
+			rte_errno = -EINVAL;
+			return i; /* return index of offending queue */
+		}
+
+		ev_queue = &dlb2->ev_queues[queues[i]];
+
+		/* Does a link exist? */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].queue_id == queues[i] &&
+			    ev_port->link[j].valid)
+				break;
+
+		if (j == DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+			continue;
+
+		ret = dlb2_event_queue_detach_ldb(dlb2, ev_port, ev_queue);
+		if (ret) {
+			DLB2_LOG_ERR("unlink err=%d for port %d queue %d\n",
+				     ret, ev_port->id, queues[i]);
+			rte_errno = -ENOENT;
+			return i; /* return index of offending queue */
+		}
+
+		ev_port->link[j].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+
+	return nb_unlinks;
+}
+
+static int
+dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
+				       void *event_port)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	struct dlb2_pending_port_unmaps_args cfg;
+	int ret;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	cfg.port_id = ev_port->qm_port.id;
+	dlb2 = ev_port->dlb2;
+	handle = &dlb2->qm_instance;
+	ret = dlb2_iface_pending_port_unmaps(handle, &cfg);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: num_unlinks_in_progress ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1813,6 +1973,9 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
 		.port_link        = dlb2_eventdev_port_link,
+		.port_unlink      = dlb2_eventdev_port_unlink,
+		.port_unlinks_in_progress =
+				    dlb2_eventdev_port_unlinks_in_progress,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 5f45b97..bd241f3 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -58,3 +58,8 @@ int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 			  struct dlb2_map_qid_args *cfg);
 
+int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+			    struct dlb2_unmap_qid_args *cfg);
+
+int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 90f7b8a..182a5bf 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -52,9 +52,15 @@ extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 enum dlb2_cq_poll_modes poll_mode);
 
 extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
-					struct dlb2_create_dir_queue_args *cfg);
+				  struct dlb2_create_dir_queue_args *cfg);
 
 extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 				 struct dlb2_map_qid_args *cfg);
 
+extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+				   struct dlb2_unmap_qid_args *cfg);
+
+extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 8eb0afb..2c38edc 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5525,3 +5525,286 @@ int dlb2_hw_map_qid(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_unmap_qid_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+		DLB2_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			    hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+				      u32 domain_id,
+				      struct dlb2_unmap_qid_args *args,
+				      struct dlb2_cmd_response *resp,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	struct dlb2_ldb_port *port;
+	int slot;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			    __func__, args->qid);
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		return 0;
+
+	resp->status = DLB2_ST_INVALID_QID;
+	return -EINVAL;
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	bool unmap_complete;
+	int i, ret, id;
+
+	dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_unmap_qid_args(hw,
+					 domain_id,
+					 args,
+					 resp,
+					 vdev_req,
+					 vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+		st = DLB2_QUEUE_UNMAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		st = DLB2_QUEUE_UNMAP_IN_PROG;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB2_QUEUE_MAPPED;
+	if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: no available CQ slots\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asynchronous procedure. It requires
+	 * stopping the DLB2 from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb2_ldb_port_cq_disable(hw, port);
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete && !os_worker_active(hw))
+		os_schedule_work(hw);
+
+unmap_qid_done:
+	resp->status = 0;
+
+	return 0;
+}
+
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+				  struct dlb2_pending_port_unmaps_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+
+	dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index ad1b0c6..b58b4ba 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -455,6 +455,55 @@ dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw_dev *handle,
+		  struct dlb2_unmap_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_unmap_qid(&dlb2_dev->hw,
+				handle->domain_id,
+				cfg,
+				&response,
+				false,
+				0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
+			    struct dlb2_pending_port_unmaps_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_pending_port_unmaps(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -471,6 +520,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
+	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v2 14/22] event/dlb2: add eventdev start

[Timothy McDaniel]

Add support for the eventdev start entry point.
We delay initializing some resources until
eventdev start, since the number of linked queues can be
used to determine if we are dealing with a ldb or dir resource.
If this is a device restart, then the previous configuration
will be reapplied.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 133 +++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   3 +
 drivers/event/dlb2/dlb2_iface.h            |   3 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 123 ++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +++
 drivers/event/dlb2/pf/dlb2_pf.c            |  25 ++++++
 6 files changed, 297 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 11fc88b..3337181 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1961,6 +1961,138 @@ dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
 	return cfg.response.id;
 }
 
+static int
+dlb2_eventdev_reapply_configuration(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret, i;
+
+	/* If an event queue or port was previously configured, but hasn't been
+	 * reconfigured, reapply its original configuration.
+	 */
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+
+		ev_queue = &dlb2->ev_queues[i];
+
+		if (ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_queue_setup(dev, i, &ev_queue->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure queue %d", i);
+			return ret;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+
+		if (ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_port_setup(dev, i, &ev_port->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure ev_port %d",
+				     i);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_apply_port_links(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int i;
+
+	/* Perform requested port->queue links */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+		int j;
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			struct dlb2_eventdev_queue *ev_queue;
+			uint8_t prio, queue_id;
+
+			if (!ev_port->link[j].valid)
+				continue;
+
+			prio = ev_port->link[j].priority;
+			queue_id = ev_port->link[j].queue_id;
+
+			if (dlb2_validate_port_link(ev_port, queue_id, true, j))
+				return -EINVAL;
+
+			ev_queue = &dlb2->ev_queues[queue_id];
+
+			if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_start(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_start_domain_args cfg;
+	int ret, i;
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_ERR("bad state %d for dev_start\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return -EINVAL;
+	}
+	dlb2->run_state = DLB2_RUN_STATE_STARTING;
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	/* If the device was configured more than once, some event ports and/or
+	 * queues may need to be reconfigured.
+	 */
+	ret = dlb2_eventdev_reapply_configuration(dev);
+	if (ret)
+		return ret;
+
+	/* The DLB PMD delays port links until the device is started. */
+	ret = dlb2_eventdev_apply_port_links(dev);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		if (!dlb2->ev_ports[i].setup_done) {
+			DLB2_LOG_ERR("dlb2: port %d not setup", i);
+			return -ESTALE;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0) {
+			DLB2_LOG_ERR("dlb2: queue %d is not linked", i);
+			return -ENOLINK;
+		}
+	}
+
+	ret = dlb2_iface_sched_domain_start(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: sched_domain_start ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STARTED;
+	DLB2_LOG_DBG("dlb2: sched_domain_start completed OK\n");
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1968,6 +2100,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.dev_start        = dlb2_eventdev_start,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index bd241f3..a86191d 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -63,3 +63,6 @@ int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				     struct dlb2_start_domain_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 182a5bf..d5bb6be 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -63,4 +63,7 @@ extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
 
+extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				struct dlb2_start_domain_args *cfg);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 2c38edc..c441d93 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5808,3 +5808,126 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 struct dlb2_cmd_response *resp,
+					 bool vdev_req,
+					 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_start_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 start domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_hw_start_domain() - Lock the domain configuration
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @arg: User-provided arguments (unused, here for ioctl callback template).
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int
+dlb2_hw_start_domain(struct dlb2_hw *hw,
+		     u32 domain_id,
+		     __attribute((unused)) struct dlb2_start_domain_args *arg,
+		     struct dlb2_cmd_response *resp,
+		     bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_queue;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+	RTE_SET_USED(arg);
+	RTE_SET_USED(iter);
+
+	dlb2_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb2_verify_start_domain_args(hw,
+					    domain_id,
+					    resp,
+					    vdev_req,
+					    vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB2 will drop all
+	 * incoming traffic to those queues.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, ldb_queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r0.val);
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+	}
+
+	dlb2_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index d5f3a82..563c944 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -667,3 +667,13 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_start_domain(struct dlb2_hw *hw,
+		     u32 id,
+		     struct dlb2_start_domain_args *args,
+		     struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_start_domain(hw, id, args, resp, NOT_VF_REQ,
+				    PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index b58b4ba..c9dc200 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -504,6 +504,30 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
 
 	return ret;
 }
+
+static int
+dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
+			   struct dlb2_start_domain_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_start_domain(&dlb2_dev->hw,
+				   handle->domain_id,
+				   cfg,
+				   &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -521,6 +545,7 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 14/22] event/dlb2: add eventdev start

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Saturday, October 17, 2020 1:21 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH v2 14/22] event/dlb2: add eventdev start
> 
> Add support for the eventdev start entry point.
> We delay initializing some resources until
> eventdev start, since the number of linked queues can be
> used to determine if we are dealing with a ldb or dir resource.
> If this is a device restart, then the previous configuration
> will be reapplied.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v2 15/22] event/dlb2: add enqueue and its burst variants

[Timothy McDaniel]

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 578 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 578 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 3337181..6cef9cb 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2093,6 +2093,578 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
 	return 0;
 }
 
+static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
+	{
+		/* Load-balanced cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
+	},
+	{
+		/* Directed cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
+	},
+};
+
+static inline uint32_t
+dlb2_port_credits_get(struct dlb2_port *qm_port,
+		      enum dlb2_hw_queue_types type)
+{
+	uint32_t credits = *qm_port->credit_pool[type];
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	if (unlikely(credits < batch_size))
+		batch_size = credits;
+
+	if (likely(credits &&
+		   __atomic_compare_exchange_n(
+			qm_port->credit_pool[type],
+			&credits, credits - batch_size, false,
+			__ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
+		return batch_size;
+	else
+		return 0;
+}
+
+static inline void
+dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port)
+{
+	uint16_t quanta = ev_port->credit_update_quanta;
+
+	if (ev_port->inflight_credits >= quanta * 2) {
+		/* Replenish credits, saving one quanta for enqueues */
+		uint16_t val = ev_port->inflight_credits - quanta;
+
+		__atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits -= val;
+	}
+}
+
+static inline int
+dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
+						__ATOMIC_SEQ_CST);
+	const int num = 1;
+
+	if (unlikely(ev_port->inflight_max < sw_inflights)) {
+		DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
+		rte_errno = -ENOSPC;
+		return 1;
+	}
+
+	if (ev_port->inflight_credits < num) {
+		/* check if event enqueue brings ev_port over max threshold */
+		uint32_t credit_update_quanta = ev_port->credit_update_quanta;
+
+		if (sw_inflights + credit_update_quanta >
+				dlb2->new_event_limit) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_new_event_limit,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+
+		__atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
+				   __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits += (credit_update_quanta);
+
+		if (ev_port->inflight_credits < num) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_inflight_credits,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_ldb_credits == 0)) {
+		qm_port->cached_ldb_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_LDB_QUEUE);
+		if (unlikely(qm_port->cached_ldb_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
+			1);
+			DLB2_LOG_DBG("ldb credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_dir_credits == 0)) {
+		qm_port->cached_dir_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_DIR_QUEUE);
+		if (unlikely(qm_port->cached_dir_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
+			1);
+			DLB2_LOG_DBG("dir credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static __rte_always_inline void
+dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
+	      struct process_local_port_data *port_data)
+{
+	dlb2_movdir64b(qe4, port_data->pp_addr);
+}
+
+static inline int
+dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_cq_pop_qe *qe;
+
+	RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
+
+	qe = qm_port->consume_qe;
+
+	qe->tokens = num - 1;
+
+	/* No store fence needed since no pointer is being sent, and CQ token
+	 * pops can be safely reordered with other HCWs.
+	 */
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	dlb2_movntdq_single(qe, port_data->pp_addr);
+
+	DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
+
+	qm_port->owed_tokens = 0;
+
+	return 0;
+}
+
+static inline void
+dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
+		   bool do_sfence,
+		   struct process_local_port_data *port_data)
+{
+	/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
+	 * application writes complete before enqueueing the QE.
+	 */
+	if (do_sfence)
+		rte_wmb();
+
+	dlb2_pp_write(qm_port->qe4, port_data);
+}
+
+static inline void
+dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
+{
+	struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
+	int num = qm_port->owed_tokens;
+
+	qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
+	qe[idx].tokens = num - 1;
+
+	qm_port->owed_tokens = 0;
+}
+
+static inline void
+dlb2_event_build_hcws(struct dlb2_port *qm_port,
+		      const struct rte_event ev[],
+		      int num,
+		      uint8_t *sched_type,
+		      uint8_t *queue_id)
+{
+	struct dlb2_enqueue_qe *qe;
+	uint16_t sched_word[4];
+	__m128i sse_qe[2];
+	int i;
+
+	qe = qm_port->qe4;
+
+	sse_qe[0] = _mm_setzero_si128();
+	sse_qe[1] = _mm_setzero_si128();
+
+	switch (num) {
+	case 4:
+		/* Construct the metadata portion of two HCWs in one 128b SSE
+		 * register. HCW metadata is constructed in the SSE registers
+		 * like so:
+		 * sse_qe[0][63:0]:   qe[0]'s metadata
+		 * sse_qe[0][127:64]: qe[1]'s metadata
+		 * sse_qe[1][63:0]:   qe[2]'s metadata
+		 * sse_qe[1][127:64]: qe[3]'s metadata
+		 */
+
+		/* Convert the event operation into a command byte and store it
+		 * in the metadata:
+		 * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
+		 * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
+		 * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
+		 * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
+		 */
+#define DLB2_QE_CMD_BYTE 7
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[0].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[1].op],
+				DLB2_QE_CMD_BYTE + 8);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[2].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[3].op],
+				DLB2_QE_CMD_BYTE + 8);
+
+		/* Store priority, scheduling type, and queue ID in the sched
+		 * word array because these values are re-used when the
+		 * destination is a directed queue.
+		 */
+		sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
+				sched_type[0] << 8 |
+				queue_id[0];
+		sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
+				sched_type[1] << 8 |
+				queue_id[1];
+		sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
+				sched_type[2] << 8 |
+				queue_id[2];
+		sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
+				sched_type[3] << 8 |
+				queue_id[3];
+
+		/* Store the event priority, scheduling type, and queue ID in
+		 * the metadata:
+		 * sse_qe[0][31:16] = sched_word[0]
+		 * sse_qe[0][95:80] = sched_word[1]
+		 * sse_qe[1][31:16] = sched_word[2]
+		 * sse_qe[1][95:80] = sched_word[3]
+		 */
+#define DLB2_QE_QID_SCHED_WORD 1
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[0],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[1],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[2],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[3],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+
+		/* If the destination is a load-balanced queue, store the lock
+		 * ID. If it is a directed queue, DLB places this field in
+		 * bytes 10-11 of the received QE, so we format it accordingly:
+		 * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
+		 * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
+		 * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
+		 * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
+		 */
+#define DLB2_QE_LOCK_ID_WORD 2
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[0] == DLB2_SCHED_DIRECTED) ?
+					sched_word[0] : ev[0].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[1] == DLB2_SCHED_DIRECTED) ?
+					sched_word[1] : ev[1].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[2] == DLB2_SCHED_DIRECTED) ?
+					sched_word[2] : ev[2].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[3] == DLB2_SCHED_DIRECTED) ?
+					sched_word[3] : ev[3].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+
+		/* Store the event type and sub event type in the metadata:
+		 * sse_qe[0][15:0]  = flow_id[0]
+		 * sse_qe[0][79:64] = flow_id[1]
+		 * sse_qe[1][15:0]  = flow_id[2]
+		 * sse_qe[1][79:64] = flow_id[3]
+		 */
+#define DLB2_QE_EV_TYPE_WORD 0
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[0].sub_event_type << 8 |
+						ev[0].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[1].sub_event_type << 8 |
+						ev[1].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[2].sub_event_type << 8 |
+						ev[2].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[3].sub_event_type << 8 |
+						ev[3].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+
+		/* Store the metadata to memory (use the double-precision
+		 * _mm_storeh_pd because there is no integer function for
+		 * storing the upper 64b):
+		 * qe[0] metadata = sse_qe[0][63:0]
+		 * qe[1] metadata = sse_qe[0][127:64]
+		 * qe[2] metadata = sse_qe[1][63:0]
+		 * qe[3] metadata = sse_qe[1][127:64]
+		 */
+		_mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
+		_mm_storeh_pd((double *)&qe[1].u.opaque_data,
+			      (__m128d)sse_qe[0]);
+		_mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
+		_mm_storeh_pd((double *)&qe[3].u.opaque_data,
+			      (__m128d)sse_qe[1]);
+
+		qe[0].data = ev[0].u64;
+		qe[1].data = ev[1].u64;
+		qe[2].data = ev[2].u64;
+		qe[3].data = ev[3].u64;
+
+		break;
+	case 3:
+	case 2:
+	case 1:
+		/* At least one QE will be valid, so only zero out three */
+		qe[1].cmd_byte = 0;
+		qe[2].cmd_byte = 0;
+		qe[3].cmd_byte = 0;
+
+		for (i = 0; i < num; i++) {
+			qe[i].cmd_byte =
+				cmd_byte_map[qm_port->is_directed][ev[i].op];
+			qe[i].sched_type = sched_type[i];
+			qe[i].data = ev[i].u64;
+			qe[i].qid = queue_id[i];
+			qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
+			qe[i].lock_id = ev[i].flow_id;
+			if (sched_type[i] == DLB2_SCHED_DIRECTED) {
+				struct dlb2_msg_info *info =
+					(struct dlb2_msg_info *)&qe[i].lock_id;
+
+				info->qid = queue_id[i];
+				info->sched_type = DLB2_SCHED_DIRECTED;
+				info->priority = qe[i].priority;
+			}
+			qe[i].u.event_type.major = ev[i].event_type;
+			qe[i].u.event_type.sub = ev[i].sub_event_type;
+		}
+		break;
+	}
+}
+
+static inline int
+dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
+			struct dlb2_port *qm_port,
+			const struct rte_event ev[],
+			uint8_t *sched_type,
+			uint8_t *queue_id)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	uint16_t *cached_credits = NULL;
+	struct dlb2_queue *qm_queue;
+
+	ev_queue = &dlb2->ev_queues[ev->queue_id];
+	qm_queue = &ev_queue->qm_queue;
+	*queue_id = qm_queue->id;
+
+	/* Ignore sched_type and hardware credits on release events */
+	if (ev->op == RTE_EVENT_OP_RELEASE)
+		goto op_check;
+
+	if (!qm_queue->is_directed) {
+		/* Load balanced destination queue */
+
+		if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_ldb_credits;
+
+		switch (ev->sched_type) {
+		case RTE_SCHED_TYPE_ORDERED:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
+			if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
+				DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
+					     *queue_id);
+				rte_errno = -EINVAL;
+				return 1;
+			}
+			*sched_type = DLB2_SCHED_ORDERED;
+			break;
+		case RTE_SCHED_TYPE_ATOMIC:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
+			*sched_type = DLB2_SCHED_ATOMIC;
+			break;
+		case RTE_SCHED_TYPE_PARALLEL:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
+			if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
+				*sched_type = DLB2_SCHED_ORDERED;
+			else
+				*sched_type = DLB2_SCHED_UNORDERED;
+			break;
+		default:
+			DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
+			DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
+			rte_errno = -EINVAL;
+			return 1;
+		}
+	} else {
+		/* Directed destination queue */
+
+		if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_dir_credits;
+
+		DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
+
+		*sched_type = DLB2_SCHED_DIRECTED;
+	}
+
+op_check:
+	switch (ev->op) {
+	case RTE_EVENT_OP_NEW:
+		/* Check that a sw credit is available */
+		if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		ev_port->inflight_credits--;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_FORWARD:
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_RELEASE:
+		ev_port->inflight_credits++;
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+
+		/* Replenish s/w credits if enough are cached */
+		dlb2_replenish_sw_credits(dlb2, ev_port);
+		break;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+	if (ev->op != RTE_EVENT_OP_RELEASE) {
+		DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
+		DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
+	}
+#endif
+
+	return 0;
+}
+
+static inline uint16_t
+dlb2_event_enqueue_burst(void *event_port,
+			 const struct rte_event events[],
+			 uint16_t num)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct process_local_port_data *port_data;
+	int i, cnt;
+
+	RTE_ASSERT(ev_port->enq_configured);
+	RTE_ASSERT(events != NULL);
+
+	cnt = 0;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
+		int j = 0;
+
+		for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
+			const struct rte_event *ev = &events[i + j];
+
+			if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
+						    &sched_types[j],
+						    &queue_ids[j]))
+				break;
+		}
+
+		if (j == 0)
+			break;
+
+		dlb2_event_build_hcws(qm_port, &events[i], j,
+				      sched_types, queue_ids);
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+
+		if (j < DLB2_NUM_QES_PER_CACHE_LINE)
+			break;
+	}
+
+	return cnt;
+}
+
+static inline uint16_t
+dlb2_event_enqueue(void *event_port,
+		   const struct rte_event events[])
+{
+	return dlb2_event_enqueue_burst(event_port, events, 1);
+}
+
+static uint16_t
+dlb2_event_enqueue_new_burst(void *event_port,
+			     const struct rte_event events[],
+			     uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
+static uint16_t
+dlb2_event_enqueue_forward_burst(void *event_port,
+				 const struct rte_event events[],
+				 uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2116,7 +2688,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
 	};
 
+	/* Expose PMD's eventdev interface */
+
 	dev->dev_ops = &dlb2_eventdev_entry_ops;
+	dev->enqueue = dlb2_event_enqueue;
+	dev->enqueue_burst = dlb2_event_enqueue_burst;
+	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
+	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v2 16/22] event/dlb2: add dequeue and its burst variants

[Timothy McDaniel]

Add support for dequeue, dequeue_burst, ...

DLB2 does not currently support interrupts, but instead use
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB2 supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c | 761 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 761 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 6cef9cb..417c5d0 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2665,9 +2665,761 @@ dlb2_event_enqueue_forward_burst(void *event_port,
 	return dlb2_event_enqueue_burst(event_port, events, num);
 }
 
+static inline void
+dlb2_port_credits_inc(struct dlb2_port *qm_port, int num)
+{
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	/* increment port credits, and return to pool if exceeds threshold */
+	if (!qm_port->is_directed) {
+		qm_port->cached_ldb_credits += num;
+		if (qm_port->cached_ldb_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_LDB_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_ldb_credits -= batch_size;
+		}
+	} else {
+		qm_port->cached_dir_credits += num;
+		if (qm_port->cached_dir_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_DIR_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_dir_credits -= batch_size;
+		}
+	}
+}
+
+static inline bool
+dlb2_cq_is_empty(struct dlb2_port *qm_port)
+{
+	volatile struct dlb2_dequeue_qe *qe_ptr;
+	struct dlb2_dequeue_qe qe;
+
+	qe_ptr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	qe = qe_ptr[qm_port->cq_idx];
+
+	return (qe.cq_gen != qm_port->gen_bit);
+}
+
+static inline int
+dlb2_dequeue_wait(struct dlb2_eventdev *dlb2,
+		  struct dlb2_eventdev_port *ev_port,
+		  struct dlb2_port *qm_port,
+		  uint64_t timeout,
+		  uint64_t start_ticks)
+{
+	struct process_local_port_data *port_data;
+	uint64_t elapsed_ticks;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	elapsed_ticks = rte_get_timer_cycles() - start_ticks;
+
+	/* Wait/poll time expired */
+	if (elapsed_ticks >= timeout) {
+		return 1;
+	} else if (dlb2->umwait_allowed) {
+		volatile struct dlb2_dequeue_qe *cq_base;
+
+		cq_base = port_data->cq_base;
+
+		/* Block on cache line write to CQ. Note: it's
+		 * safe to access the per-process cq_base
+		 * address here, since the PMD has already
+		 * attempted at least one CQ dequeue.
+		 */
+		dlb2_umonitor(&cq_base[qm_port->cq_idx]);
+
+		/* Avoid race condition. Check if still empty */
+		if (dlb2_cq_is_empty(qm_port)) {
+			dlb2_umwait(RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE,
+				    timeout + start_ticks);
+			DLB2_INC_STAT(
+				ev_port->stats.traffic.rx_umonitor_umwait, 1);
+		}
+	} else {
+		uint64_t poll_interval = RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL;
+		uint64_t curr_ticks = rte_get_timer_cycles();
+		uint64_t init_ticks = curr_ticks;
+
+		while ((curr_ticks - start_ticks < timeout) &&
+		       (curr_ticks - init_ticks < poll_interval))
+			curr_ticks = rte_get_timer_cycles();
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
+			 struct dlb2_port *qm_port,
+			 struct rte_event *events,
+			 struct dlb2_dequeue_qe *qes,
+			 int cnt)
+{
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	int i, num, evq_id;
+
+	for (i = 0, num = 0; i < cnt; i++) {
+		struct dlb2_dequeue_qe *qe = &qes[i];
+		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
+			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+			[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+		};
+
+		/* Fill in event information.
+		 * Note that flow_id must be embedded in the data by
+		 * the app, such as the mbuf RSS hash field if the data
+		 * buffer is a mbuf.
+		 */
+		if (unlikely(qe->error)) {
+			DLB2_LOG_ERR("QE error bit ON\n");
+			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
+			dlb2_consume_qe_immediate(qm_port, 1);
+			continue; /* Ignore */
+		}
+
+		events[num].u64 = qe->data;
+		events[num].flow_id = qe->flow_id;
+		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
+		events[num].event_type = qe->u.event_type.major;
+		events[num].sub_event_type = qe->u.event_type.sub;
+		events[num].sched_type = sched_type_map[qe->sched_type];
+		events[num].impl_opaque = qe->qid_depth;
+
+		/* qid not preserved for directed queues */
+		if (qm_port->is_directed)
+			evq_id = ev_port->link[0].queue_id;
+		else
+			evq_id = qid_mappings[qe->qid];
+
+		events[num].queue_id = evq_id;
+		DLB2_INC_STAT(
+			ev_port->stats.queue[evq_id].qid_depth[qe->qid_depth],
+			1);
+		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type], 1);
+		num++;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num);
+
+	return num;
+}
+
+static inline int
+dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
+			      struct dlb2_port *qm_port,
+			      struct rte_event *events,
+			      struct dlb2_dequeue_qe *qes)
+{
+	int sched_type_map[] = {
+		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+	};
+	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	__m128i sse_evt[2];
+
+	/* In the unlikely case that any of the QE error bits are set, process
+	 * them one at a time.
+	 */
+	if (unlikely(qes[0].error || qes[1].error ||
+		     qes[2].error || qes[3].error))
+		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
+						 qes, num_events);
+
+	events[0].u64 = qes[0].data;
+	events[1].u64 = qes[1].data;
+	events[2].u64 = qes[2].data;
+	events[3].u64 = qes[3].data;
+
+	/* Construct the metadata portion of two struct rte_events
+	 * in one 128b SSE register. Event metadata is constructed in the SSE
+	 * registers like so:
+	 * sse_evt[0][63:0]:   event[0]'s metadata
+	 * sse_evt[0][127:64]: event[1]'s metadata
+	 * sse_evt[1][63:0]:   event[2]'s metadata
+	 * sse_evt[1][127:64]: event[3]'s metadata
+	 */
+	sse_evt[0] = _mm_setzero_si128();
+	sse_evt[1] = _mm_setzero_si128();
+
+	/* Convert the hardware queue ID to an event queue ID and store it in
+	 * the metadata:
+	 * sse_evt[0][47:40]   = qid_mappings[qes[0].qid]
+	 * sse_evt[0][111:104] = qid_mappings[qes[1].qid]
+	 * sse_evt[1][47:40]   = qid_mappings[qes[2].qid]
+	 * sse_evt[1][111:104] = qid_mappings[qes[3].qid]
+	 */
+#define RTE_EVENT_QUEUE_ID_BYTE 5
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[0].qid],
+				     RTE_EVENT_QUEUE_ID_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[1].qid],
+				     RTE_EVENT_QUEUE_ID_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[2].qid],
+				     RTE_EVENT_QUEUE_ID_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[3].qid],
+				     RTE_EVENT_QUEUE_ID_BYTE + 8);
+
+	/* Convert the hardware priority to an event priority and store it in
+	 * the metadata, while also returning the queue depth status
+	 * value captured by the hardware, storing it in impl_opaque, which can
+	 * be read by the application but not modified
+	 * sse_evt[0][55:48]   = DLB2_TO_EV_PRIO(qes[0].priority)
+	 * sse_evt[0][63:56]   = qes[0].qid_depth
+	 * sse_evt[0][119:112] = DLB2_TO_EV_PRIO(qes[1].priority)
+	 * sse_evt[0][127:120] = qes[1].qid_depth
+	 * sse_evt[1][55:48]   = DLB2_TO_EV_PRIO(qes[2].priority)
+	 * sse_evt[1][63:56]   = qes[2].qid_depth
+	 * sse_evt[1][119:112] = DLB2_TO_EV_PRIO(qes[3].priority)
+	 * sse_evt[1][127:120] = qes[3].qid_depth
+	 */
+#define RTE_EVENT_PRIO_IMPL_OPAQUE_WORD 3
+#define RTE_BYTE_SHIFT 8
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[0].priority) |
+			(qes[0].qid_depth << RTE_BYTE_SHIFT),
+			RTE_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[1].priority) |
+			(qes[1].qid_depth << RTE_BYTE_SHIFT),
+			RTE_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[2].priority) |
+			(qes[2].qid_depth << RTE_BYTE_SHIFT),
+			RTE_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[3].priority) |
+			(qes[3].qid_depth << RTE_BYTE_SHIFT),
+			RTE_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+
+	/* Write the event type, sub event type, and flow_id to the event
+	 * metadata.
+	 * sse_evt[0][31:0]   = qes[0].flow_id |
+	 *			qes[0].u.event_type.major << 28 |
+	 *			qes[0].u.event_type.sub << 20;
+	 * sse_evt[0][95:64]  = qes[1].flow_id |
+	 *			qes[1].u.event_type.major << 28 |
+	 *			qes[1].u.event_type.sub << 20;
+	 * sse_evt[1][31:0]   = qes[2].flow_id |
+	 *			qes[2].u.event_type.major << 28 |
+	 *			qes[2].u.event_type.sub << 20;
+	 * sse_evt[1][95:64]  = qes[3].flow_id |
+	 *			qes[3].u.event_type.major << 28 |
+	 *			qes[3].u.event_type.sub << 20;
+	 */
+#define RTE_EVENT_EV_TYPE_DW 0
+#define RTE_EVENT_EV_TYPE_SHIFT 28
+#define RTE_EVENT_SUB_EV_TYPE_SHIFT 20
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[0].flow_id |
+			qes[0].u.event_type.major << RTE_EVENT_EV_TYPE_SHIFT |
+			qes[0].u.event_type.sub <<  RTE_EVENT_SUB_EV_TYPE_SHIFT,
+			RTE_EVENT_EV_TYPE_DW);
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[1].flow_id |
+			qes[1].u.event_type.major << RTE_EVENT_EV_TYPE_SHIFT |
+			qes[1].u.event_type.sub <<  RTE_EVENT_SUB_EV_TYPE_SHIFT,
+			RTE_EVENT_EV_TYPE_DW + 2);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[2].flow_id |
+			qes[2].u.event_type.major << RTE_EVENT_EV_TYPE_SHIFT |
+			qes[2].u.event_type.sub <<  RTE_EVENT_SUB_EV_TYPE_SHIFT,
+			RTE_EVENT_EV_TYPE_DW);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[3].flow_id |
+			qes[3].u.event_type.major << RTE_EVENT_EV_TYPE_SHIFT  |
+			qes[3].u.event_type.sub << RTE_EVENT_SUB_EV_TYPE_SHIFT,
+			RTE_EVENT_EV_TYPE_DW + 2);
+
+	/* Write the sched type to the event metadata. 'op' and 'rsvd' are not
+	 * set:
+	 * sse_evt[0][39:32]  = sched_type_map[qes[0].sched_type] << 6
+	 * sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6
+	 * sse_evt[1][39:32]  = sched_type_map[qes[2].sched_type] << 6
+	 * sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6
+	 */
+#define RTE_EVENT_SCHED_TYPE_BYTE 4
+#define RTE_EVENT_SCHED_TYPE_SHIFT 6
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[0].sched_type] << RTE_EVENT_SCHED_TYPE_SHIFT,
+		RTE_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[1].sched_type] << RTE_EVENT_SCHED_TYPE_SHIFT,
+		RTE_EVENT_SCHED_TYPE_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[2].sched_type] << RTE_EVENT_SCHED_TYPE_SHIFT,
+		RTE_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[3].sched_type] << RTE_EVENT_SCHED_TYPE_SHIFT,
+		RTE_EVENT_SCHED_TYPE_BYTE + 8);
+
+	/* Store the metadata to the event (use the double-precision
+	 * _mm_storeh_pd because there is no integer function for storing the
+	 * upper 64b):
+	 * events[0].event = sse_evt[0][63:0]
+	 * events[1].event = sse_evt[0][127:64]
+	 * events[2].event = sse_evt[1][63:0]
+	 * events[3].event = sse_evt[1][127:64]
+	 */
+	_mm_storel_epi64((__m128i *)&events[0].event, sse_evt[0]);
+	_mm_storeh_pd((double *)&events[1].event, (__m128d) sse_evt[0]);
+	_mm_storel_epi64((__m128i *)&events[2].event, sse_evt[1]);
+	_mm_storeh_pd((double *)&events[3].event, (__m128d) sse_evt[1]);
+
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[0].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[1].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[2].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[3].sched_type], 1);
+
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[0].queue_id].
+			qid_depth[qes[0].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[1].queue_id].
+			qid_depth[qes[1].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[2].queue_id].
+			qid_depth[qes[2].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[3].queue_id].
+			qid_depth[qes[3].qid_depth],
+		1);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num_events);
+
+	return num_events;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe_sparse(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe)
+{
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	uint8_t xor_mask[2] = {0x0F, 0x00};
+	const uint8_t and_mask = 0x0F;
+	__m128i *qes = (__m128i *)qe;
+	uint8_t gen_bits, gen_bit;
+	uintptr_t addr[4];
+	uint16_t idx;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+
+	idx = qm_port->cq_idx;
+
+	/* Load the next 4 QEs */
+	addr[0] = (uintptr_t)&cq_addr[idx];
+	addr[1] = (uintptr_t)&cq_addr[(idx +  4) & qm_port->cq_depth_mask];
+	addr[2] = (uintptr_t)&cq_addr[(idx +  8) & qm_port->cq_depth_mask];
+	addr[3] = (uintptr_t)&cq_addr[(idx + 12) & qm_port->cq_depth_mask];
+
+	/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */
+	rte_prefetch0(&cq_addr[(idx + 16) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 20) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 24) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 28) & qm_port->cq_depth_mask]);
+
+	/* Correct the xor_mask for wrap-around QEs */
+	gen_bit = qm_port->gen_bit;
+	xor_mask[gen_bit] ^= !!((idx +  4) > qm_port->cq_depth_mask) << 1;
+	xor_mask[gen_bit] ^= !!((idx +  8) > qm_port->cq_depth_mask) << 2;
+	xor_mask[gen_bit] ^= !!((idx + 12) > qm_port->cq_depth_mask) << 3;
+
+	/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is
+	 * valid, then QEs[0:N-1] are too.
+	 */
+	qes[3] = _mm_load_si128((__m128i *)(void *)addr[3]);
+	rte_compiler_barrier();
+	qes[2] = _mm_load_si128((__m128i *)(void *)addr[2]);
+	rte_compiler_barrier();
+	qes[1] = _mm_load_si128((__m128i *)(void *)addr[1]);
+	rte_compiler_barrier();
+	qes[0] = _mm_load_si128((__m128i *)(void *)addr[0]);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[gen_bit];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask;
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline void
+dlb2_inc_cq_idx(struct dlb2_port *qm_port, int cnt)
+{
+	uint16_t idx = qm_port->cq_idx_unmasked + cnt;
+
+	qm_port->cq_idx_unmasked = idx;
+	qm_port->cq_idx = idx & qm_port->cq_depth_mask;
+	qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) & 0x1;
+}
+
+static int
+dlb2_event_release(struct dlb2_eventdev *dlb2,
+		   uint8_t port_id,
+		   int n)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_eventdev_port *ev_port;
+	struct dlb2_port *qm_port;
+	int i, cnt;
+
+	if (port_id > dlb2->num_ports) {
+		DLB2_LOG_ERR("Invalid port id %d in dlb2-event_release\n",
+			     port_id);
+		rte_errno = -EINVAL;
+		return rte_errno;
+	}
+
+	ev_port = &dlb2->ev_ports[port_id];
+	qm_port = &ev_port->qm_port;
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	cnt = 0;
+
+	if (qm_port->is_directed) {
+		cnt = n;
+		goto sw_credit_update;
+	}
+
+	for (i = 0; i < n; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		int j;
+
+		/* Zero-out QEs */
+		qm_port->qe4[0].cmd_byte = 0;
+		qm_port->qe4[1].cmd_byte = 0;
+		qm_port->qe4[2].cmd_byte = 0;
+		qm_port->qe4[3].cmd_byte = 0;
+
+		for (j = 0; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++)
+			qm_port->qe4[j].cmd_byte = DLB2_COMP_CMD_BYTE;
+
+		qm_port->issued_releases += j;
+
+		if (j == 0)
+			break;
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+	}
+
+sw_credit_update:
+	/* each release returns one credit */
+	if (!ev_port->outstanding_releases) {
+		DLB2_LOG_ERR("Unrecoverable application error. Outstanding releases underflowed.\n");
+		rte_errno = -ENOTRECOVERABLE;
+		return rte_errno;
+	}
+
+	ev_port->outstanding_releases -= cnt;
+	ev_port->inflight_credits += cnt;
+
+	/* Replenish s/w credits if enough releases are performed */
+	dlb2_replenish_sw_credits(dlb2, ev_port);
+	return 0;
+}
+
+static inline int16_t
+dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
+		       struct dlb2_eventdev_port *ev_port,
+		       struct rte_event *events,
+		       uint16_t max_num,
+		       uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe_sparse(qm_port, qes);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail << 2);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							      qm_port,
+							      &events[num],
+							      &qes[0]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							 qm_port,
+							 &events[num],
+							 &qes[0],
+							 num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe,
+	     uint8_t *offset)
+{
+	uint8_t xor_mask[2][4] = { {0x0F, 0x0E, 0x0C, 0x08},
+				   {0x00, 0x01, 0x03, 0x07} };
+	uint8_t and_mask[4] = {0x0F, 0x0E, 0x0C, 0x08};
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	__m128i *qes = (__m128i *)qe;
+	uint64_t *cache_line_base;
+	uint8_t gen_bits;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	cq_addr = &cq_addr[qm_port->cq_idx];
+
+	cache_line_base = (void *)(((uintptr_t)cq_addr) & ~0x3F);
+	*offset = ((uintptr_t)cq_addr & 0x30) >> 4;
+
+	/* Load the next CQ cache line from memory. Pack these reads as tight
+	 * as possible to reduce the chance that DLB invalidates the line while
+	 * the CPU is reading it. Read the cache line backwards to ensure that
+	 * if QE[N] (N > 0) is valid, then QEs[0:N-1] are too.
+	 *
+	 * (Valid QEs start at &qe[offset])
+	 */
+	qes[3] = _mm_load_si128((__m128i *)&cache_line_base[6]);
+	qes[2] = _mm_load_si128((__m128i *)&cache_line_base[4]);
+	qes[1] = _mm_load_si128((__m128i *)&cache_line_base[2]);
+	qes[0] = _mm_load_si128((__m128i *)&cache_line_base[0]);
+
+	/* Evict the cache line ASAP */
+	dlb2_cldemote(cache_line_base);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[qm_port->gen_bit][*offset];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask[*offset];
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline int16_t
+dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
+		struct dlb2_eventdev_port *ev_port,
+		struct rte_event *events,
+		uint16_t max_num,
+		uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t offset;
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe(qm_port, qes, &offset);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							     qm_port,
+							     &events[num],
+							     &qes[offset]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							qm_port,
+							&events[num],
+							&qes[offset],
+							num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static uint16_t
+dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
+			 uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue(void *event_port, struct rte_event *ev, uint64_t wait)
+{
+	return dlb2_event_dequeue_burst(event_port, ev, 1, wait);
+}
+
+static uint16_t
+dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
+				uint16_t num, uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
+			  uint64_t wait)
+{
+	return dlb2_event_dequeue_burst_sparse(event_port, ev, 1, wait);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
+	struct dlb2_eventdev *dlb2;
+
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
@@ -2695,6 +3447,15 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	dev->enqueue_burst = dlb2_event_enqueue_burst;
 	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
 	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
+
+	dlb2 = dev->data->dev_private;
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE) {
+		dev->dequeue = dlb2_event_dequeue_sparse;
+		dev->dequeue_burst = dlb2_event_dequeue_burst_sparse;
+	} else {
+		dev->dequeue = dlb2_event_dequeue;
+		dev->dequeue_burst = dlb2_event_dequeue_burst;
+	}
 }
 
 int
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 16/22] event/dlb2: add dequeue and its burst variants

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Saturday, October 17, 2020 1:21 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH v2 16/22] event/dlb2: add dequeue and its burst variants
> 
> Add support for dequeue, dequeue_burst, ...
> 
> DLB2 does not currently support interrupts, but instead use
> umonitor/umwait if supported by the processor. This allows
> the software to monitor and wait on writes to a cache-line.
> 
> DLB2 supports normal and sparse cq mode. In normal mode the
> hardware will pack 4 QEs into each cache line. In sparse cq
> mode, the hardware will only populate one QE per cache line.
> Software must be aware of the cq mode, and take the appropriate
> actions, based on the mode.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v2 17/22] event/dlb2: add eventdev stop and close

[Timothy McDaniel]

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 257 +++++++++++++++++++++++++++--
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c |  91 ++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  52 ++++++
 5 files changed, 396 insertions(+), 16 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 417c5d0..483659e 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -74,21 +74,6 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
-/*
- * DUMMY - added so that xstats path will compile/link.
- * Will be replaced by real version in a subsequent
- * patch.
- */
-uint32_t
-dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
-		     struct dlb2_eventdev_queue *queue)
-{
-	RTE_SET_USED(dlb2);
-	RTE_SET_USED(queue);
-
-	return 0;
-}
-
 static void
 dlb2_free_qe_mem(struct dlb2_port *qm_port)
 {
@@ -1881,7 +1866,6 @@ dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
 		return 0; /* Ignore and return success */
 	}
 
-	/* FIXME: How to handle unlink on a directed port? */
 	if (ev_port->qm_port.is_directed) {
 		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
 			     ev_port->id);
@@ -3416,6 +3400,245 @@ dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
 }
 
 static void
+dlb2_flush_port(struct rte_eventdev *dev, int port_id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	eventdev_stop_flush_t flush;
+	struct rte_event ev;
+	uint8_t dev_id;
+	void *arg;
+	int i;
+
+	flush = dev->dev_ops->dev_stop_flush;
+	dev_id = dev->data->dev_id;
+	arg = dev->data->dev_stop_flush_arg;
+
+	while (rte_event_dequeue_burst(dev_id, port_id, &ev, 1, 0)) {
+		if (flush)
+			flush(dev_id, ev, arg);
+
+		if (dlb2->ev_ports[port_id].qm_port.is_directed)
+			continue;
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+	}
+
+	/* Enqueue any additional outstanding releases */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = dlb2->ev_ports[port_id].outstanding_releases; i > 0; i--)
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+}
+
+static uint32_t
+dlb2_get_ldb_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_ldb_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_ldb_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_ldb_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static uint32_t
+dlb2_get_dir_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_dir_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_dir_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_dir_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	if (queue->qm_queue.is_directed)
+		return dlb2_get_dir_queue_depth(dlb2, queue);
+	else
+		return dlb2_get_ldb_queue_depth(dlb2, queue);
+}
+
+static bool
+dlb2_queue_is_empty(struct dlb2_eventdev *dlb2,
+		    struct dlb2_eventdev_queue *queue)
+{
+	return dlb2_get_queue_depth(dlb2, queue) == 0;
+}
+
+static bool
+dlb2_linked_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0)
+			continue;
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static bool
+dlb2_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static void
+dlb2_drain(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_port *ev_port = NULL;
+	uint8_t dev_id;
+	int i;
+
+	dev_id = dev->data->dev_id;
+
+	while (!dlb2_linked_queues_empty(dlb2)) {
+		/* Flush all the ev_ports, which will drain all their connected
+		 * queues.
+		 */
+		for (i = 0; i < dlb2->num_ports; i++)
+			dlb2_flush_port(dev, i);
+	}
+
+	/* The queues are empty, but there may be events left in the ports. */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_flush_port(dev, i);
+
+	/* If the domain's queues are empty, we're done. */
+	if (dlb2_queues_empty(dlb2))
+		return;
+
+	/* Else, there must be at least one unlinked load-balanced queue.
+	 * Select a load-balanced port with which to drain the unlinked
+	 * queue(s).
+	 */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		if (!ev_port->qm_port.is_directed)
+			break;
+	}
+
+	if (i == dlb2->num_ports) {
+		DLB2_LOG_ERR("internal error: no LDB ev_ports\n");
+		return;
+	}
+
+	rte_errno = 0;
+	rte_event_port_unlink(dev_id, ev_port->id, NULL, 0);
+
+	if (rte_errno) {
+		DLB2_LOG_ERR("internal error: failed to unlink ev_port %d\n",
+			     ev_port->id);
+		return;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		uint8_t qid, prio;
+		int ret;
+
+		if (dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			continue;
+
+		qid = i;
+		prio = 0;
+
+		/* Link the ev_port to the queue */
+		ret = rte_event_port_link(dev_id, ev_port->id, &qid, &prio, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to link ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+
+		/* Flush the queue */
+		while (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			dlb2_flush_port(dev, ev_port->id);
+
+		/* Drain any extant events in the ev_port. */
+		dlb2_flush_port(dev, ev_port->id);
+
+		/* Unlink the ev_port from the queue */
+		ret = rte_event_port_unlink(dev_id, ev_port->id, &qid, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to unlink ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+	}
+}
+
+static void
+dlb2_eventdev_stop(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_DBG("Internal error: already stopped\n");
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	} else if (dlb2->run_state != DLB2_RUN_STATE_STARTED) {
+		DLB2_LOG_ERR("Internal error: bad state %d for dev_stop\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPING;
+
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	dlb2_drain(dev);
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPED;
+}
+
+static int
+dlb2_eventdev_close(struct rte_eventdev *dev)
+{
+	dlb2_hw_reset_sched_domain(dev, false);
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3425,6 +3648,8 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.dev_start        = dlb2_eventdev_start,
+		.dev_stop         = dlb2_eventdev_stop,
+		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a86191d..5471dd8 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -66,3 +66,9 @@ int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				     struct dlb2_start_domain_args *cfg);
+
+int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index d5bb6be..bcd9446 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -66,4 +66,10 @@ extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				struct dlb2_start_domain_args *cfg);
 
+extern int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+extern int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index c441d93..71b4604 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5931,3 +5931,94 @@ dlb2_hw_start_domain(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_dir_queue_depth(hw, queue);
+
+	return 0;
+}
+
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+
+	dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index c9dc200..99e6d8e 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -528,6 +528,56 @@ dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_ldb_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_ldb_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_dir_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret = 0;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_dir_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -545,6 +595,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth;
+	dlb2_iface_get_dir_queue_depth = dlb2_pf_get_dir_queue_depth;
 	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 17/22] event/dlb2: add eventdev stop and close

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Saturday, October 17, 2020 1:21 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH v2 17/22] event/dlb2: add eventdev stop and close
> 
> Add support for eventdev stop and close entry points.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v2 18/22] event/dlb2: add PMD's token pop public interface

[Timothy McDaniel]

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 doc/api/doxy-api-index.md                         |  1 +
 drivers/event/dlb2/dlb2.c                         | 53 ++++++++++++++++--
 drivers/event/dlb2/dlb2_priv.h                    |  3 +
 drivers/event/dlb2/meson.build                    |  5 +-
 drivers/event/dlb2/rte_pmd_dlb2.c                 | 39 +++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2.h                 | 68 +++++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2_event_version.map |  6 ++
 7 files changed, 168 insertions(+), 7 deletions(-)
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index b855a8f..2b2020c 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -26,6 +26,7 @@ The public API headers are grouped by topics:
   [event_eth_tx_adapter]   (@ref rte_event_eth_tx_adapter.h),
   [event_timer_adapter]    (@ref rte_event_timer_adapter.h),
   [event_crypto_adapter]   (@ref rte_event_crypto_adapter.h),
+  [dlb2]	       (@ref rte_pmd_dlb2.h)
   [rawdev]             (@ref rte_rawdev.h),
   [metrics]            (@ref rte_metrics.h),
   [bitrate]            (@ref rte_bitrate.h),
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 483659e..9a41418 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1200,7 +1200,7 @@ dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
 	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
 
 	qm_port->dequeue_depth = dequeue_depth;
-
+	qm_port->token_pop_thresh = dequeue_depth;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -1368,6 +1368,8 @@ dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
 
 	qm_port->dequeue_depth = dequeue_depth;
 
+	/* Directed ports are auto-pop, by default. */
+	qm_port->token_pop_mode = AUTO_POP;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -2615,6 +2617,14 @@ dlb2_event_enqueue_burst(void *event_port,
 		dlb2_event_build_hcws(qm_port, &events[i], j,
 				      sched_types, queue_ids);
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
@@ -2623,6 +2633,11 @@ dlb2_event_enqueue_burst(void *event_port,
 			break;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
 	return cnt;
 }
 
@@ -3105,11 +3120,25 @@ dlb2_event_release(struct dlb2_eventdev *dlb2,
 		if (j == 0)
 			break;
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
 sw_credit_update:
 	/* each release returns one credit */
 	if (!ev_port->outstanding_releases) {
@@ -3193,8 +3222,8 @@ dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3320,8 +3349,8 @@ dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3336,6 +3365,7 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 			 uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3351,6 +3381,9 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3370,6 +3403,7 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 				uint16_t num, uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3385,6 +3419,9 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3689,7 +3726,7 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    struct dlb2_devargs *dlb2_args)
 {
 	struct dlb2_eventdev *dlb2;
-	int err;
+	int err, i;
 
 	dlb2 = dev->data->dev_private;
 
@@ -3739,6 +3776,10 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Initialize each port's token pop mode */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
+		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 61567a6..b73cf3f 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -12,6 +12,7 @@
 #include <rte_config.h>
 #include "dlb2_user.h"
 #include "dlb2_log.h"
+#include "rte_pmd_dlb2.h"
 
 #ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
 #define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
@@ -290,6 +291,7 @@ struct dlb2_port {
 	bool gen_bit;
 	uint16_t dir_credits;
 	uint32_t dequeue_depth;
+	enum dlb2_token_pop_mode token_pop_mode;
 	union dlb2_port_config cfg;
 	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
 	uint16_t cached_ldb_credits;
@@ -298,6 +300,7 @@ struct dlb2_port {
 	bool int_armed;
 	uint16_t owed_tokens;
 	int16_t issued_releases;
+	int16_t token_pop_thresh;
 	int cq_depth;
 	uint16_t cq_idx;
 	uint16_t cq_idx_unmasked;
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 6bf8adf..539773c 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,3 +1,4 @@
+
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
@@ -6,7 +7,9 @@ sources = files('dlb2.c',
 		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
-		'pf/base/dlb2_resource.c'
+		'pf/base/dlb2_resource.c',
+		'rte_pmd_dlb2.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
+install_headers('rte_pmd_dlb2.h')
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
new file mode 100644
index 0000000..b09b585
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "rte_pmd_dlb2.h"
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode)
+{
+	struct dlb2_eventdev *dlb2;
+	struct rte_eventdev *dev;
+
+	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+	dev = &rte_eventdevs[dev_id];
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (mode >= NUM_TOKEN_POP_MODES)
+		return -EINVAL;
+
+	/* The event device must be configured, but not yet started */
+	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
+		return -EINVAL;
+
+	/* The token pop mode must be set before configuring the port */
+	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
+		return -EINVAL;
+
+	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
new file mode 100644
index 0000000..489693f
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.h
@@ -0,0 +1,68 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+/*!
+ *  @file      rte_pmd_dlb2.h
+ *
+ *  @brief     DLB PMD-specific functions
+ *
+ *  @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ */
+
+#ifndef _RTE_PMD_DLB2_H_
+#define _RTE_PMD_DLB2_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+/**
+ * Selects the token pop mode for a DLB2 port.
+ */
+enum dlb2_token_pop_mode {
+	/* Pop the CQ tokens immediately after dequeueing. */
+	AUTO_POP,
+	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
+	 * Supported on load-balanced ports only.
+	 */
+	DELAYED_POP,
+	/* Pop the CQ tokens during next dequeue operation. */
+	DEFERRED_POP,
+
+	/* NUM_TOKEN_POP_MODES must be last */
+	NUM_TOKEN_POP_MODES
+};
+
+/*!
+ * Configure the token pop mode for a DLB2 port. By default, all ports use
+ * AUTO_POP. This function must be called before calling rte_event_port_setup()
+ * for the port, but after calling rte_event_dev_configure().
+ *
+ * @param dev_id
+ *    The identifier of the event device.
+ * @param port_id
+ *    The identifier of the event port.
+ * @param mode
+ *    The token pop mode.
+ *
+ * @return
+ * - 0: Success
+ * - EINVAL: Invalid dev_id, port_id, or mode
+ * - EINVAL: The DLB2 is not configured, is already running, or the port is
+ *   already setup
+ */
+
+__rte_experimental
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_DLB2_H_ */
diff --git a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
index 299ae63..84b81a4 100644
--- a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
+++ b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
@@ -1,3 +1,9 @@
 DPDK_21.0 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmd_dlb2_set_token_pop_mode;
+};
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 18/22] event/dlb2: add PMD's token pop public interface

[Jerin Jacob]

On Sat, Oct 17, 2020 at 11:57 PM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> The PMD uses a public interface to allow applications to
> control the token pop mode. Supported token pop modes are
> as follows, and they impact core scheduling affinity for
> ldb ports.
>
> AUTO_POP: Pop the CQ tokens immediately after dequeueing.
> DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
>              are released. Supported on load-balanced ports
>              only.
> DEFERRED_POP: Pop the CQ tokens during next dequeue operation.
>
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  doc/api/doxy-api-index.md                         |  1 +
>  drivers/event/dlb2/dlb2.c                         | 53 ++++++++++++++++--
>  drivers/event/dlb2/dlb2_priv.h                    |  3 +
>  drivers/event/dlb2/meson.build                    |  5 +-
>  drivers/event/dlb2/rte_pmd_dlb2.c                 | 39 +++++++++++++
>  drivers/event/dlb2/rte_pmd_dlb2.h                 | 68 +++++++++++++++++++++++
>  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |  6 ++
>  7 files changed, 168 insertions(+), 7 deletions(-)
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
>
> diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
> index b855a8f..2b2020c 100644
> --- a/doc/api/doxy-api-index.md
> +++ b/doc/api/doxy-api-index.md
> @@ -26,6 +26,7 @@ The public API headers are grouped by topics:
>    [event_eth_tx_adapter]   (@ref rte_event_eth_tx_adapter.h),
>    [event_timer_adapter]    (@ref rte_event_timer_adapter.h),
>    [event_crypto_adapter]   (@ref rte_event_crypto_adapter.h),
> +  [dlb2]              (@ref rte_pmd_dlb2.h)

move this under the already existing "- **device specific**:" section.

Re: [dpdk-dev] [PATCH v2 18/22] event/dlb2: add PMD's token pop public interface

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Sunday, October 18, 2020 4:14 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: Mcnamara, John <john.mcnamara@intel.com>; Kovacevic, Marko
> <Marko.Kovacevic@intel.com>; Ray Kinsella <mdr@ashroe.eu>; Neil Horman
> <nhorman@tuxdriver.com>; dpdk-dev <dev@dpdk.org>; Carrillo, Erik G
> <Erik.G.Carrillo@intel.com>; Eads, Gage <gage.eads@intel.com>; Van Haaren,
> Harry <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH v2 18/22] event/dlb2: add PMD's token pop
> public interface
> 
> On Sat, Oct 17, 2020 at 11:57 PM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > The PMD uses a public interface to allow applications to
> > control the token pop mode. Supported token pop modes are
> > as follows, and they impact core scheduling affinity for
> > ldb ports.
> >
> > AUTO_POP: Pop the CQ tokens immediately after dequeueing.
> > DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
> >              are released. Supported on load-balanced ports
> >              only.
> > DEFERRED_POP: Pop the CQ tokens during next dequeue operation.
> >
> > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > ---
> >  doc/api/doxy-api-index.md                         |  1 +
> >  drivers/event/dlb2/dlb2.c                         | 53 ++++++++++++++++--
> >  drivers/event/dlb2/dlb2_priv.h                    |  3 +
> >  drivers/event/dlb2/meson.build                    |  5 +-
> >  drivers/event/dlb2/rte_pmd_dlb2.c                 | 39 +++++++++++++
> >  drivers/event/dlb2/rte_pmd_dlb2.h                 | 68 +++++++++++++++++++++++
> >  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |  6 ++
> >  7 files changed, 168 insertions(+), 7 deletions(-)
> >  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
> >  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
> >
> > diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
> > index b855a8f..2b2020c 100644
> > --- a/doc/api/doxy-api-index.md
> > +++ b/doc/api/doxy-api-index.md
> > @@ -26,6 +26,7 @@ The public API headers are grouped by topics:
> >    [event_eth_tx_adapter]   (@ref rte_event_eth_tx_adapter.h),
> >    [event_timer_adapter]    (@ref rte_event_timer_adapter.h),
> >    [event_crypto_adapter]   (@ref rte_event_crypto_adapter.h),
> > +  [dlb2]              (@ref rte_pmd_dlb2.h)
> 
> move this under the already existing "- **device specific**:" section.

okay, will do.

Re: [dpdk-dev] [PATCH v2 18/22] event/dlb2: add PMD's token pop public interface

[Kinsella, Ray]

On 17/10/2020 19:21, Timothy McDaniel wrote:
> The PMD uses a public interface to allow applications to
> control the token pop mode. Supported token pop modes are
> as follows, and they impact core scheduling affinity for
> ldb ports.
> 
> AUTO_POP: Pop the CQ tokens immediately after dequeueing.
> DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
>              are released. Supported on load-balanced ports
>              only.
> DEFERRED_POP: Pop the CQ tokens during next dequeue operation.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  doc/api/doxy-api-index.md                         |  1 +
>  drivers/event/dlb2/dlb2.c                         | 53 ++++++++++++++++--
>  drivers/event/dlb2/dlb2_priv.h                    |  3 +
>  drivers/event/dlb2/meson.build                    |  5 +-
>  drivers/event/dlb2/rte_pmd_dlb2.c                 | 39 +++++++++++++
>  drivers/event/dlb2/rte_pmd_dlb2.h                 | 68 +++++++++++++++++++++++
>  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |  6 ++
>  7 files changed, 168 insertions(+), 7 deletions(-)
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
> 
> diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
> index b855a8f..2b2020c 100644
> --- a/doc/api/doxy-api-index.md
> +++ b/doc/api/doxy-api-index.md
> @@ -26,6 +26,7 @@ The public API headers are grouped by topics:
>    [event_eth_tx_adapter]   (@ref rte_event_eth_tx_adapter.h),
>    [event_timer_adapter]    (@ref rte_event_timer_adapter.h),
>    [event_crypto_adapter]   (@ref rte_event_crypto_adapter.h),
> +  [dlb2]	       (@ref rte_pmd_dlb2.h)
>    [rawdev]             (@ref rte_rawdev.h),
>    [metrics]            (@ref rte_metrics.h),
>    [bitrate]            (@ref rte_bitrate.h),
> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> index 483659e..9a41418 100644
> --- a/drivers/event/dlb2/dlb2.c
> +++ b/drivers/event/dlb2/dlb2.c
> @@ -1200,7 +1200,7 @@ dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
>  	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
>  
>  	qm_port->dequeue_depth = dequeue_depth;
> -
> +	qm_port->token_pop_thresh = dequeue_depth;
>  	qm_port->owed_tokens = 0;
>  	qm_port->issued_releases = 0;
>  
> @@ -1368,6 +1368,8 @@ dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
>  
>  	qm_port->dequeue_depth = dequeue_depth;
>  
> +	/* Directed ports are auto-pop, by default. */
> +	qm_port->token_pop_mode = AUTO_POP;
>  	qm_port->owed_tokens = 0;
>  	qm_port->issued_releases = 0;
>  
> @@ -2615,6 +2617,14 @@ dlb2_event_enqueue_burst(void *event_port,
>  		dlb2_event_build_hcws(qm_port, &events[i], j,
>  				      sched_types, queue_ids);
>  
> +		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
> +		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
> +			dlb2_construct_token_pop_qe(qm_port, j);
> +
> +			/* Reset the releases counter for the next QE batch */
> +			qm_port->issued_releases -= qm_port->token_pop_thresh;
> +		}
> +
>  		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
>  
>  		cnt += j;
> @@ -2623,6 +2633,11 @@ dlb2_event_enqueue_burst(void *event_port,
>  			break;
>  	}
>  
> +	if (qm_port->token_pop_mode == DELAYED_POP &&
> +	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
> +		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
> +		qm_port->issued_releases -= qm_port->token_pop_thresh;
> +	}
>  	return cnt;
>  }
>  
> @@ -3105,11 +3120,25 @@ dlb2_event_release(struct dlb2_eventdev *dlb2,
>  		if (j == 0)
>  			break;
>  
> +		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
> +		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
> +			dlb2_construct_token_pop_qe(qm_port, j);
> +
> +			/* Reset the releases counter for the next QE batch */
> +			qm_port->issued_releases -= qm_port->token_pop_thresh;
> +		}
> +
>  		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
>  
>  		cnt += j;
>  	}
>  
> +	if (qm_port->token_pop_mode == DELAYED_POP &&
> +	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
> +		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
> +		qm_port->issued_releases -= qm_port->token_pop_thresh;
> +	}
> +
>  sw_credit_update:
>  	/* each release returns one credit */
>  	if (!ev_port->outstanding_releases) {
> @@ -3193,8 +3222,8 @@ dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
>  	qm_port->owed_tokens += num;
>  
>  	if (num) {
> -
> -		dlb2_consume_qe_immediate(qm_port, num);
> +		if (qm_port->token_pop_mode == AUTO_POP)
> +			dlb2_consume_qe_immediate(qm_port, num);
>  
>  		ev_port->outstanding_releases += num;
>  
> @@ -3320,8 +3349,8 @@ dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
>  	qm_port->owed_tokens += num;
>  
>  	if (num) {
> -
> -		dlb2_consume_qe_immediate(qm_port, num);
> +		if (qm_port->token_pop_mode == AUTO_POP)
> +			dlb2_consume_qe_immediate(qm_port, num);
>  
>  		ev_port->outstanding_releases += num;
>  
> @@ -3336,6 +3365,7 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
>  			 uint64_t wait)
>  {
>  	struct dlb2_eventdev_port *ev_port = event_port;
> +	struct dlb2_port *qm_port = &ev_port->qm_port;
>  	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
>  	uint16_t cnt;
>  
> @@ -3351,6 +3381,9 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
>  		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
>  	}
>  
> +	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
> +		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
> +
>  	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
>  
>  	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
> @@ -3370,6 +3403,7 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
>  				uint16_t num, uint64_t wait)
>  {
>  	struct dlb2_eventdev_port *ev_port = event_port;
> +	struct dlb2_port *qm_port = &ev_port->qm_port;
>  	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
>  	uint16_t cnt;
>  
> @@ -3385,6 +3419,9 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
>  		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
>  	}
>  
> +	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
> +		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
> +
>  	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
>  
>  	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
> @@ -3689,7 +3726,7 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
>  			    struct dlb2_devargs *dlb2_args)
>  {
>  	struct dlb2_eventdev *dlb2;
> -	int err;
> +	int err, i;
>  
>  	dlb2 = dev->data->dev_private;
>  
> @@ -3739,6 +3776,10 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
>  		return err;
>  	}
>  
> +	/* Initialize each port's token pop mode */
> +	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
> +		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
> +
>  	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
>  
>  	dlb2_iface_low_level_io_init();
> diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
> index 61567a6..b73cf3f 100644
> --- a/drivers/event/dlb2/dlb2_priv.h
> +++ b/drivers/event/dlb2/dlb2_priv.h
> @@ -12,6 +12,7 @@
>  #include <rte_config.h>
>  #include "dlb2_user.h"
>  #include "dlb2_log.h"
> +#include "rte_pmd_dlb2.h"
>  
>  #ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
>  #define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
> @@ -290,6 +291,7 @@ struct dlb2_port {
>  	bool gen_bit;
>  	uint16_t dir_credits;
>  	uint32_t dequeue_depth;
> +	enum dlb2_token_pop_mode token_pop_mode;
>  	union dlb2_port_config cfg;
>  	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
>  	uint16_t cached_ldb_credits;
> @@ -298,6 +300,7 @@ struct dlb2_port {
>  	bool int_armed;
>  	uint16_t owed_tokens;
>  	int16_t issued_releases;
> +	int16_t token_pop_thresh;
>  	int cq_depth;
>  	uint16_t cq_idx;
>  	uint16_t cq_idx_unmasked;
> diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
> index 6bf8adf..539773c 100644
> --- a/drivers/event/dlb2/meson.build
> +++ b/drivers/event/dlb2/meson.build
> @@ -1,3 +1,4 @@
> +
>  # SPDX-License-Identifier: BSD-3-Clause
>  # Copyright(c) 2019-2020 Intel Corporation
>  
> @@ -6,7 +7,9 @@ sources = files('dlb2.c',
>  		'dlb2_xstats.c',
>  		'pf/dlb2_main.c',
>  		'pf/dlb2_pf.c',
> -		'pf/base/dlb2_resource.c'
> +		'pf/base/dlb2_resource.c',
> +		'rte_pmd_dlb2.c'
>  )
>  
>  deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
> +install_headers('rte_pmd_dlb2.h')
> diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
> new file mode 100644
> index 0000000..b09b585
> --- /dev/null
> +++ b/drivers/event/dlb2/rte_pmd_dlb2.c
> @@ -0,0 +1,39 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2020 Intel Corporation
> + */
> +
> +#include <rte_eventdev.h>
> +#include <rte_eventdev_pmd.h>
> +
> +#include "rte_pmd_dlb2.h"
> +#include "dlb2_priv.h"
> +#include "dlb2_inline_fns.h"
> +
> +int
> +rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
> +				uint8_t port_id,
> +				enum dlb2_token_pop_mode mode)
> +{
> +	struct dlb2_eventdev *dlb2;
> +	struct rte_eventdev *dev;
> +
> +	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
> +	dev = &rte_eventdevs[dev_id];
> +
> +	dlb2 = dlb2_pmd_priv(dev);
> +
> +	if (mode >= NUM_TOKEN_POP_MODES)
> +		return -EINVAL;
> +
> +	/* The event device must be configured, but not yet started */
> +	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
> +		return -EINVAL;
> +
> +	/* The token pop mode must be set before configuring the port */
> +	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
> +		return -EINVAL;
> +
> +	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
> +
> +	return 0;
> +}
> diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
> new file mode 100644
> index 0000000..489693f
> --- /dev/null
> +++ b/drivers/event/dlb2/rte_pmd_dlb2.h
> @@ -0,0 +1,68 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2020 Intel Corporation
> + */
> +
> +/*!
> + *  @file      rte_pmd_dlb2.h
> + *
> + *  @brief     DLB PMD-specific functions
> + *
> + *  @b EXPERIMENTAL: this API may change, or be removed, without prior notice
> + */
> +
> +#ifndef _RTE_PMD_DLB2_H_
> +#define _RTE_PMD_DLB2_H_
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#include <stdint.h>
> +
> +/**
> + * Selects the token pop mode for a DLB2 port.
> + */
> +enum dlb2_token_pop_mode {
> +	/* Pop the CQ tokens immediately after dequeueing. */
> +	AUTO_POP,
> +	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
> +	 * Supported on load-balanced ports only.
> +	 */
> +	DELAYED_POP,
> +	/* Pop the CQ tokens during next dequeue operation. */
> +	DEFERRED_POP,
> +
> +	/* NUM_TOKEN_POP_MODES must be last */
> +	NUM_TOKEN_POP_MODES
> +};
> +


Missing the banner.

+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice

> +/*!
> + * Configure the token pop mode for a DLB2 port. By default, all ports use
> + * AUTO_POP. This function must be called before calling rte_event_port_setup()
> + * for the port, but after calling rte_event_dev_configure().
> + *
> + * @param dev_id
> + *    The identifier of the event device.
> + * @param port_id
> + *    The identifier of the event port.
> + * @param mode
> + *    The token pop mode.
> + *
> + * @return
> + * - 0: Success
> + * - EINVAL: Invalid dev_id, port_id, or mode
> + * - EINVAL: The DLB2 is not configured, is already running, or the port is
> + *   already setup
> + */
> +
> +__rte_experimental
> +int
> +rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
> +				uint8_t port_id,
> +				enum dlb2_token_pop_mode mode);
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif /* _RTE_PMD_DLB2_H_ */
> diff --git a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> index 299ae63..84b81a4 100644
> --- a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> +++ b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> @@ -1,3 +1,9 @@
>  DPDK_21.0 {
>  	local: *;
>  };
> +
> +EXPERIMENTAL {
> +	global:
> +
> +	rte_pmd_dlb2_set_token_pop_mode;
> +};
> 

Re: [dpdk-dev] [PATCH v2 18/22] event/dlb2: add PMD's token pop public interface

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Saturday, October 17, 2020 1:21 PM
> To: Mcnamara, John <john.mcnamara@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>; Ray Kinsella <mdr@ashroe.eu>; Neil Horman
> <nhorman@tuxdriver.com>
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH v2 18/22] event/dlb2: add PMD's token pop public interface
> 
> The PMD uses a public interface to allow applications to
> control the token pop mode. Supported token pop modes are
> as follows, and they impact core scheduling affinity for
> ldb ports.
> 
> AUTO_POP: Pop the CQ tokens immediately after dequeueing.
> DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
>              are released. Supported on load-balanced ports
>              only.
> DEFERRED_POP: Pop the CQ tokens during next dequeue operation.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

With Jerin and Ray's issues addressed:
Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v2 19/22] event/dlb2: add PMD self-tests

[Timothy McDaniel]

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 app/test/test_eventdev.c           |    7 +
 drivers/event/dlb2/dlb2.c          |    1 +
 drivers/event/dlb2/dlb2_selftest.c | 1570 ++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build     |    3 +-
 4 files changed, 1580 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c

diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
index 62019c1..bcfaa53 100644
--- a/app/test/test_eventdev.c
+++ b/app/test/test_eventdev.c
@@ -1030,6 +1030,12 @@ test_eventdev_selftest_dpaa2(void)
 	return test_eventdev_selftest_impl("event_dpaa2", "");
 }
 
+static int
+test_eventdev_selftest_dlb2(void)
+{
+	return test_eventdev_selftest_impl("dlb2_event", "");
+}
+
 REGISTER_TEST_COMMAND(eventdev_common_autotest, test_eventdev_common);
 REGISTER_TEST_COMMAND(eventdev_selftest_sw, test_eventdev_selftest_sw);
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
@@ -1037,3 +1043,4 @@ REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
 		test_eventdev_selftest_octeontx2);
 REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2, test_eventdev_selftest_dpaa2);
+REGISTER_TEST_COMMAND(eventdev_selftest_dlb2, test_eventdev_selftest_dlb2);
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 9a41418..06a59a5 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3700,6 +3700,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
 		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+		.dev_selftest     = test_dlb2_eventdev,
 	};
 
 	/* Expose PMD's eventdev interface */
diff --git a/drivers/event/dlb2/dlb2_selftest.c b/drivers/event/dlb2/dlb2_selftest.c
new file mode 100644
index 0000000..418a53b
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_selftest.c
@@ -0,0 +1,1570 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <unistd.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_eventdev.h>
+#include <rte_pause.h>
+
+#include "dlb2_priv.h"
+#include "rte_pmd_dlb2.h"
+
+#define MAX_PORTS 32
+#define MAX_QIDS 32
+#define DEFAULT_NUM_SEQ_NUMS 64
+
+static struct rte_mempool *eventdev_func_mempool;
+static int evdev;
+
+struct test {
+	struct rte_mempool *mbuf_pool;
+	int nb_qids;
+};
+
+/* initialization and config */
+static inline int
+init(struct test *t, int nb_queues, int nb_ports)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	memset(t, 0, sizeof(*t));
+
+	t->mbuf_pool = eventdev_func_mempool;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = nb_queues;
+	config.nb_event_ports = nb_ports;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0)
+		printf("%d: Error configuring device\n", __LINE__);
+
+	return ret;
+}
+
+static inline int
+create_ports(int num_ports)
+{
+	int i;
+
+	if (num_ports > MAX_PORTS)
+		return -1;
+
+	for (i = 0; i < num_ports; i++) {
+		struct rte_event_port_conf conf;
+
+		if (rte_event_port_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default port conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		if (rte_event_port_setup(evdev, i, &conf) < 0) {
+			printf("%d: Error setting up port %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+create_lb_qids(struct test *t, int num_qids, uint32_t flags)
+{
+	int i;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		struct rte_event_queue_conf conf;
+
+		if (rte_event_queue_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default queue conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		conf.schedule_type = flags;
+
+		if (conf.schedule_type == RTE_SCHED_TYPE_PARALLEL)
+			conf.nb_atomic_order_sequences = 0;
+		else
+			conf.nb_atomic_order_sequences = DEFAULT_NUM_SEQ_NUMS;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+static inline int
+create_atomic_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ATOMIC);
+}
+
+static inline int
+create_ordered_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ORDERED);
+}
+
+static inline int
+create_unordered_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_PARALLEL);
+}
+
+static inline int
+create_directed_qids(struct test *t, int num_qids, const uint8_t ports[])
+{
+	static struct rte_event_queue_conf conf;
+	int i;
+
+	conf.priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
+	conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		uint8_t queue_id;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, ports[i - t->nb_qids],
+					&queue_id, NULL, 1) != 1) {
+			printf("%d: error creating link for qid %d\n",
+			       __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+/* destruction */
+static inline void
+cleanup(void)
+{
+	int ret = 0;
+
+	rte_event_dev_stop(evdev);
+	ret = rte_event_dev_close(evdev);
+
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+};
+
+static inline int
+enqueue_timeout(uint8_t port_id, struct rte_event *ev, uint64_t tmo_us)
+{
+	const uint64_t start = rte_get_timer_cycles();
+	const uint64_t ticks = (tmo_us * rte_get_timer_hz()) / 1E6;
+
+	while ((rte_get_timer_cycles() - start) < ticks) {
+		if (rte_event_enqueue_burst(evdev, port_id, ev, 1) == 1)
+			return 0;
+
+		if (rte_errno != -ENOSPC) {
+			printf("enqueue_burst returned rte_errno %d\n",
+			       rte_errno);
+			return -1;
+		}
+	}
+	printf("%s time out\n", __func__);
+	return -1;
+}
+
+static void
+flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused)
+{
+	rte_pktmbuf_free(event.mbuf);
+}
+
+static int
+test_stop_flush(struct test *t) /* test to check we can properly flush events */
+{
+	struct rte_event ev;
+	uint32_t dequeue_depth;
+	unsigned int i, count;
+	uint8_t queue_id;
+
+	ev.op = RTE_EVENT_OP_NEW;
+
+	if (init(t, 2, 1) < 0 ||
+	    create_ports(1) < 0 ||
+	    create_atomic_qids(t, 2) < 0) {
+		printf("%d: Error initializing device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_port_link(evdev, 0, NULL, NULL, 0) != 2) {
+		printf("%d: Error linking queues to the port\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: Error with start call\n", __LINE__);
+		goto err;
+	}
+
+	/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked
+	 * queue.
+	 */
+	queue_id = 1;
+
+	if (rte_event_port_unlink(evdev, 0, &queue_id, 1) != 1) {
+		printf("%d: Error unlinking queue 1 from port\n", __LINE__);
+		goto err;
+	}
+
+	count = rte_mempool_avail_count(t->mbuf_pool);
+
+	if (rte_event_port_attr_get(evdev,
+				    0,
+				    RTE_EVENT_PORT_ATTR_DEQ_DEPTH,
+				    &dequeue_depth)) {
+		printf("%d: Error retrieveing dequeue depth\n", __LINE__);
+		goto err;
+	}
+
+	/* Send QEs to queue 0 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 0;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Send QEs to queue 1 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 1;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Now the DLB is scheduling events from the port to the IQ, and at
+	 * least one event should be remaining in each queue.
+	 */
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, flush, NULL)) {
+		printf("%d: Error installing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+
+	if (count != rte_mempool_avail_count(t->mbuf_pool)) {
+		printf("%d: Error executing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, NULL, NULL)) {
+		printf("%d: Error uninstalling the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	return 0;
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_single_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Create a directed port */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) == 0) {
+		printf("%d: port 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Create a load-balanced port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a directed queue */
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed queue */
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) == 0) {
+		printf("%d: queue 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a load-balanced queue */
+	queue_conf.event_queue_cfg = 0;
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to link directed and load-balanced resources */
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 0 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 1 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Link ports to queues */
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+#define NUM_LDB_PORTS 64
+#define NUM_LDB_QUEUES 32
+
+static int
+test_info_get(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	config.nb_event_ports = info.max_event_ports;
+	config.nb_event_queues = NUM_LDB_QUEUES + info.max_event_ports / 2;
+	config.nb_single_link_event_port_queues = info.max_event_ports / 2;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		goto err;
+	}
+
+	/* The DLB2 PMD only reports load-balanced ports and queues in its
+	 * info_get function. Confirm that these values don't include the
+	 * directed port or queue counts.
+	 */
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret) {
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+		return -1;
+	}
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+static int
+test_reconfiguration_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 2 LDB queues */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+			printf("%d: P0->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P1->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+			printf("%d: P1->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Configure device with 2 DIR ports and 2 DIR queues */
+	config.nb_single_link_event_port_queues = 2;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q1 */
+	queue_id = 1;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: P0->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Link P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: P1->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_queues = 5;
+	config.nb_event_ports = 5;
+	config.nb_single_link_event_port_queues = 1;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < config.nb_event_queues - 1; i++) {
+		port_conf.event_port_cfg = 0;
+		queue_conf.event_queue_cfg = 0;
+
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: P%d->Q%d link expected to succeed\n",
+			       __LINE__, i, i);
+			goto err;
+		}
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+		printf("%d: port %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+		printf("%d: queue %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	queue_id = i;
+
+	if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+		printf("%d: P%d->Q%d link expected to succeed\n",
+		       __LINE__, i, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_ports += 1;
+
+	/* Reconfigure device with 1 more load-balanced port */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Configure the new port */
+	if (rte_event_port_setup(evdev, config.nb_event_ports - 1,
+				 &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_load_balanced_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_directed_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 DIR port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (ev.queue_id != 0) {
+		printf("%d: invalid dequeued event queue ID (%d)\n",
+		       __LINE__, ev.queue_id);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_deferred_sched(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	const int num_events = 128;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 1 queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 1, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.dequeue_depth = 1;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.schedule_type = RTE_SCHED_TYPE_PARALLEL;
+	queue_conf.nb_atomic_order_sequences = 0;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 128 NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_PARALLEL;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue one event from port 0 */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue (and release) all other events from port 1. Deferred
+	 * scheduling ensures no other events are scheduled to port 0 without a
+	 * subsequent rte_event_dequeue_burst() call.
+	 */
+	for (i = 0; i < num_events - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 1, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		if (rte_event_enqueue_burst(evdev, 1, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_delayed_pop(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	int ret, i, num_events;
+	struct rte_event ev;
+	uint8_t queue_id;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DELAYED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	num_events = 2 * port_conf.dequeue_depth;
+
+	/* Enqueue 2 * dequeue_depth NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue dequeue_depth events but only release dequeue_depth - 2.
+	 * Delayed pop won't perform the pop and no more events will be
+	 * scheduled.
+	 */
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = 0; i < port_conf.dequeue_depth - 2; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	ret = rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout);
+	if (ret != 0) {
+		printf("%d: event dequeue expected to fail (ret = %d)\n",
+		       __LINE__, ret);
+		goto err;
+	}
+
+	/* Release one more event. This will trigger the token pop, and
+	 * dequeue_depth - 1 more events will be scheduled to the device.
+	 */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: RELEASE enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 0) {
+		printf("%d: event dequeue expected to fail\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+do_selftest(void)
+{
+	struct test t;
+	int ret;
+
+	/* Only create mbuf pool once, reuse for each test run */
+	if (!eventdev_func_mempool) {
+		eventdev_func_mempool =
+			rte_pktmbuf_pool_create("EVENTDEV_DLB2_ST_POOL",
+						(1 << 12), /* 4k buffers */
+						32 /*MBUF_CACHE_SIZE*/,
+						0,
+						512, /* use very small mbufs */
+						rte_socket_id());
+		if (!eventdev_func_mempool) {
+			printf("ERROR creating mempool\n");
+			goto test_fail;
+		}
+	}
+	t.mbuf_pool = eventdev_func_mempool;
+
+	printf("*** Running Stop Flush test...\n");
+	ret = test_stop_flush(&t);
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Single Link test...\n");
+	ret = test_single_link();
+	if (ret != 0) {
+		printf("ERROR - Single Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Info Get test...\n");
+	ret = test_info_get();
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Reconfiguration Link test...\n");
+	ret = test_reconfiguration_link();
+	if (ret != 0) {
+		printf("ERROR - Reconfiguration Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Load-Balanced Traffic test...\n");
+	ret = test_load_balanced_traffic();
+	if (ret != 0) {
+		printf("ERROR - Load-Balanced Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Directed Traffic test...\n");
+	ret = test_directed_traffic();
+	if (ret != 0) {
+		printf("ERROR - Directed Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Deferred Scheduling test...\n");
+	ret = test_deferred_sched();
+	if (ret != 0) {
+		printf("ERROR - Deferred Scheduling test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Delayed Pop test...\n");
+	ret = test_delayed_pop();
+	if (ret != 0) {
+		printf("ERROR - Delayed Pop test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	return 0;
+
+test_fail:
+	return -1;
+}
+
+int
+test_dlb2_eventdev(void)
+{
+	const char *dlb2_eventdev_name = "dlb2_event";
+	uint8_t num_evdevs = rte_event_dev_count();
+	int i, ret = 0;
+	int found = 0, skipped = 0, passed = 0, failed = 0;
+	struct rte_event_dev_info info;
+
+	for (i = 0; found + skipped < num_evdevs && i < RTE_EVENT_MAX_DEVS;
+	     i++) {
+		ret = rte_event_dev_info_get(i, &info);
+		if (ret < 0)
+			continue;
+
+		/* skip non-dlb2 event devices */
+		if (strncmp(info.driver_name, dlb2_eventdev_name,
+			    sizeof(*info.driver_name)) != 0) {
+			skipped++;
+			continue;
+		}
+
+		evdev = rte_event_dev_get_dev_id(info.driver_name);
+		if (evdev < 0) {
+			printf("Could not get dev_id for eventdev with name %s, i=%d\n",
+			       info.driver_name, i);
+			skipped++;
+			continue;
+		}
+		found++;
+		printf("Running selftest on eventdev %s\n", info.driver_name);
+		ret = do_selftest();
+		if (ret == 0) {
+			passed++;
+			printf("Selftest passed for eventdev %s\n",
+			       info.driver_name);
+		} else {
+			failed++;
+			printf("Selftest failed for eventdev %s, err=%d\n",
+			       info.driver_name, ret);
+		}
+	}
+
+	printf("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n",
+	       found, skipped, passed, failed);
+	return ret;
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 539773c..fe0065c 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -8,7 +8,8 @@ sources = files('dlb2.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c',
-		'rte_pmd_dlb2.c'
+		'rte_pmd_dlb2.c',
+		'dlb2_selftest.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v2 20/22] event/dlb2: add queue and port release

[Timothy McDaniel]

DLB does not support reconfiguring individual queues
or ports on the fly. The entire device must be reconfigured.
Previously allocated port QE ond memzone memory
is freed in this patch.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 06a59a5..968923e 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -89,8 +89,8 @@ dlb2_free_qe_mem(struct dlb2_port *qm_port)
 	rte_free(qm_port->consume_qe);
 	qm_port->consume_qe = NULL;
 
-	rte_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base);
-	rte_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].pp_addr);
+	rte_memzone_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz);
+	dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz = NULL;
 }
 
 /* override defaults with value(s) provided on command line */
@@ -3676,6 +3676,28 @@ dlb2_eventdev_close(struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(id);
+
+	/* This function intentionally left blank. */
+}
+
+static void
+dlb2_eventdev_port_release(void *port)
+{
+	struct dlb2_eventdev_port *ev_port = port;
+	struct dlb2_port *qm_port;
+
+	if (ev_port) {
+		qm_port = &ev_port->qm_port;
+		if (qm_port->config_state == DLB2_CONFIGURED)
+			dlb2_free_qe_mem(qm_port);
+	}
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3689,8 +3711,10 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
+		.queue_release    = dlb2_eventdev_queue_release,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_release     = dlb2_eventdev_port_release,
 		.port_link        = dlb2_eventdev_port_link,
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 20/22] event/dlb2: add queue and port release

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Saturday, October 17, 2020 1:21 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH v2 20/22] event/dlb2: add queue and port release
> 
> DLB does not support reconfiguring individual queues
> or ports on the fly. The entire device must be reconfigured.
> Previously allocated port QE ond memzone memory
> is freed in this patch.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  drivers/event/dlb2/dlb2.c | 28 ++++++++++++++++++++++++++--
>  1 file changed, 26 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> index 06a59a5..968923e 100644
> --- a/drivers/event/dlb2/dlb2.c
> +++ b/drivers/event/dlb2/dlb2.c
> @@ -89,8 +89,8 @@ dlb2_free_qe_mem(struct dlb2_port *qm_port)
>  	rte_free(qm_port->consume_qe);
>  	qm_port->consume_qe = NULL;
> 
> -	rte_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base);
> -	rte_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].pp_addr);
> +	rte_memzone_free(dlb2_port[qm_port-
> >id][PORT_TYPE(qm_port)].mz);
> +	dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz = NULL;
>  }

This code should be a part of patch 8, when free_qe_mem is introduced. With that change:
Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH v2 21/22] event/dlb2: add timeout ticks entry point

[Timothy McDaniel]

Adds the timeout ticks conversion function.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 968923e..48879a1 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3697,6 +3697,18 @@ dlb2_eventdev_port_release(void *port)
 	}
 }
 
+static int
+dlb2_eventdev_timeout_ticks(struct rte_eventdev *dev, uint64_t ns,
+			    uint64_t *timeout_ticks)
+{
+	RTE_SET_USED(dev);
+	uint64_t cycles_per_ns = rte_get_timer_hz() / 1E9;
+
+	*timeout_ticks = ns * cycles_per_ns;
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -3719,6 +3731,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
 				    dlb2_eventdev_port_unlinks_in_progress,
+		.timeout_ticks    = dlb2_eventdev_timeout_ticks,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v2 22/22] doc: add new DLB2 eventdev driver to relnotes

[Timothy McDaniel]

Added announcement of availabililty for the new driver
for Intel Dynamic Load Balancer 2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 MAINTAINERS                            | 5 +++++
 doc/guides/rel_notes/release_20_11.rst | 5 +++++
 2 files changed, 10 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 3b16d7a..f732f28 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1188,6 +1188,11 @@ M: Peter Mccarthy <peter.mccarthy@intel.com>
 F: drivers/event/opdl/
 F: doc/guides/eventdevs/opdl.rst
 
+Intel DLB2 Eventdev PMD
+M: Timothy McDaniel <timothy.mcdaniel@intel.com>
+F: drivers/event/dlb2/
+F: doc/guides/eventdevs/dlb2.rst
+
 
 Rawdev Drivers
 --------------
diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
index df227a1..f80df21 100644
--- a/doc/guides/rel_notes/release_20_11.rst
+++ b/doc/guides/rel_notes/release_20_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
+
+  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
+  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
+
 
 Removed Items
 -------------
-- 
2.6.4

Re: [dpdk-dev] [PATCH v2 22/22] doc: add new DLB2 eventdev driver to relnotes

[Jerin Jacob]

On Sat, Oct 17, 2020 at 11:58 PM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> Added announcement of availabililty for the new driver
> for Intel Dynamic Load Balancer 2.0 hardware.
>
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  MAINTAINERS                            | 5 +++++
>  doc/guides/rel_notes/release_20_11.rst | 5 +++++
>  2 files changed, 10 insertions(+)
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 3b16d7a..f732f28 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1188,6 +1188,11 @@ M: Peter Mccarthy <peter.mccarthy@intel.com>
>  F: drivers/event/opdl/
>  F: doc/guides/eventdevs/opdl.rst
>
> +Intel DLB2 Eventdev PMD
> +M: Timothy McDaniel <timothy.mcdaniel@intel.com>
> +F: drivers/event/dlb2/
> +F: doc/guides/eventdevs/dlb2.rst


Move this section to the first patch.


>
>  Rawdev Drivers
>  --------------
> diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
> index df227a1..f80df21 100644
> --- a/doc/guides/rel_notes/release_20_11.rst
> +++ b/doc/guides/rel_notes/release_20_11.rst
> @@ -55,6 +55,11 @@ New Features
>       Also, make sure to start the actual text at the margin.
>       =======================================================
>
> +* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
> +
> +  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
> +  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.


This specific patch is not required.  Move this section to the last
patch in this series.

Also, fix the following checkpatch issues in this series.

WARNING:REPEATED_WORD: Possible repeated word: 'from'
#155: FILE: drivers/event/dlb2/dlb2_user.h:132:
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *     from any class-of-service with available ports.

WARNING:REPEATED_WORD: Possible repeated word: 'are'
#969: FILE: drivers/event/dlb2/pf/base/dlb2_hw_types.h:321:
+ * -- A domain's used resources list. These are are domain-owned configured

### event/dlb2: add queue and port release

WARNING:TYPO_SPELLING: 'ond' may be misspelled - perhaps 'one'?
#8:
Previously allocated port QE ond memzone memory



> +
>
>  Removed Items
>  -------------
> --
> 2.6.4
>

Re: [dpdk-dev] [PATCH v2 22/22] doc: add new DLB2 eventdev driver to relnotes

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Sunday, October 18, 2020 4:23 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>; Mcnamara, John
> <john.mcnamara@intel.com>; Kovacevic, Marko
> <Marko.Kovacevic@intel.com>; dpdk-dev <dev@dpdk.org>; Carrillo, Erik G
> <Erik.G.Carrillo@intel.com>; Eads, Gage <gage.eads@intel.com>; Van Haaren,
> Harry <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH v2 22/22] doc: add new DLB2 eventdev driver to
> relnotes
> 
> On Sat, Oct 17, 2020 at 11:58 PM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > Added announcement of availabililty for the new driver
> > for Intel Dynamic Load Balancer 2.0 hardware.
> >
> > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > ---
> >  MAINTAINERS                            | 5 +++++
> >  doc/guides/rel_notes/release_20_11.rst | 5 +++++
> >  2 files changed, 10 insertions(+)
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 3b16d7a..f732f28 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -1188,6 +1188,11 @@ M: Peter Mccarthy <peter.mccarthy@intel.com>
> >  F: drivers/event/opdl/
> >  F: doc/guides/eventdevs/opdl.rst
> >
> > +Intel DLB2 Eventdev PMD
> > +M: Timothy McDaniel <timothy.mcdaniel@intel.com>
> > +F: drivers/event/dlb2/
> > +F: doc/guides/eventdevs/dlb2.rst
> 
> 
> Move this section to the first patch.
> 

Will do


> 
> >
> >  Rawdev Drivers
> >  --------------
> > diff --git a/doc/guides/rel_notes/release_20_11.rst
> b/doc/guides/rel_notes/release_20_11.rst
> > index df227a1..f80df21 100644
> > --- a/doc/guides/rel_notes/release_20_11.rst
> > +++ b/doc/guides/rel_notes/release_20_11.rst
> > @@ -55,6 +55,11 @@ New Features
> >       Also, make sure to start the actual text at the margin.
> >       =======================================================
> >
> > +* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
> > +
> > +  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
> > +  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
> 
> 
> This specific patch is not required.  Move this section to the last
> patch in this series.
> 
> Also, fix the following checkpatch issues in this series.
> 

sure, will do

> WARNING:REPEATED_WORD: Possible repeated word: 'from'
> #155: FILE: drivers/event/dlb2/dlb2_user.h:132:
> + * - num_ldb_ports: Number of load-balanced ports that can be allocated from
> + *     from any class-of-service with available ports.
> 
> WARNING:REPEATED_WORD: Possible repeated word: 'are'
> #969: FILE: drivers/event/dlb2/pf/base/dlb2_hw_types.h:321:
> + * -- A domain's used resources list. These are are domain-owned configured
> 
> ### event/dlb2: add queue and port release
> 
> WARNING:TYPO_SPELLING: 'ond' may be misspelled - perhaps 'one'?
> #8:
> Previously allocated port QE ond memzone memory
> 
> 
> 
> > +
> >
> >  Removed Items
> >  -------------
> > --
> > 2.6.4
> >

Re: [dpdk-dev] [PATCH v2 00/22] Add DLB2 PMD

[Thomas Monjalon]

17/10/2020 20:20, Timothy McDaniel:
> Note that the DLB2 hardware is a successor of the DLB hardware, and
> as such is structured similarly, both in terms of code layout and
> implementation.

That's quite funny that you are unable to have a single driver
for both similar hardware. I guess that's the Intel way.

[dpdk-dev] [PATCH v3 00/23] Add DLB2 PMD

[Timothy McDaniel]

The following patch series adds support for a new eventdev PMD. The DLB2
PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
hardware.
The DLB2 is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB2 hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB2 ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a
directed queue. Additionally, even though LDB ports may link multiple
queues,
the number of queues that may be linked is limited by hardware.

While reviewing the code, please be aware that this PMD has full
control over the DLB2 hardware. Intel will be extending the DLB2 PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.
Note that the DLB2 hardware is a successor of the DLB hardware, and
as such is structured similarly, both in terms of code layout and
implementation.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Changes since V2:
================
- fixed meson conditional build. Moved test into driver’s meson.build
  file instead of event/meson.build
- documentation is populated as associated code is introduced
- add log_register in add dynamic logging patch
- rename RTE_xxx symbol(s) as DLB2_xxx
- replaced function ptr enqueue_four with direct call to movdir64b
- remove unused port_pages
- broke up probe patch into 3 smaller patches for easier review
- changed param order of movdir64b/movntdq to match intrinsics
- added self to MAINTAINERS files
- squashed announcement of availability into last patch in series
- correct spelling errors and delete repeated words
- DPDK_21.0 -> DPDK 21 in map file
- add experimental banner to public structs and APIs

Changes since V1:
=================
- implement changes requested in code reviews by Gage Eads and Mike
  Chen
- fix a memzone leak
- convert to use eal rte-cpuflags patch from Liang Ma

Known Issues:
- the documentation contained in dlb2.rst is not complete, and
  should be updated to include command line parameters (class of service,
	etc ...).

Depends-on: patch-79539 ("eal: add new x86 cpuid support for WAITPKG")

Timothy McDaniel (23):
  event/dlb2: add documentation and meson build infrastructure
  event/dlb2: add dynamic logging
  event/dlb2: add private data structures and constants
  event/dlb2: add definitions shared with LKM or shared code
  event/dlb2: add inline functions
  event/dlb2: add eventdev probe
  event/dlb2: add flexible interface
  event/dlb2: add probe-time hardware init
  event/dlb2: add xstats
  event/dlb2: add infos get and configure
  event/dlb2: add queue and port default conf
  event/dlb2: add queue setup
  event/dlb2: add port setup
  event/dlb2: add port link
  event/dlb2: add port unlink and port unlinks in progress
  event/dlb2: add eventdev start
  event/dlb2: add enqueue and its burst variants
  event/dlb2: add dequeue and its burst variants
  event/dlb2: add eventdev stop and close
  event/dlb2: add PMD's token pop public interface
  event/dlb2: add PMD self-tests
  event/dlb2: add queue and port release
  event/dlb2: add timeout ticks entry point

 MAINTAINERS                                       |    5 +
 app/test/test_eventdev.c                          |    7 +
 config/rte_config.h                               |    7 +
 doc/api/doxy-api-index.md                         |    1 +
 doc/guides/eventdevs/dlb2.rst                     |  330 ++
 doc/guides/eventdevs/index.rst                    |    1 +
 doc/guides/rel_notes/release_20_11.rst            |    5 +
 drivers/event/dlb2/dlb2.c                         | 3951 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c                   |   74 +
 drivers/event/dlb2/dlb2_iface.h                   |   74 +
 drivers/event/dlb2/dlb2_inline_fns.h              |   61 +
 drivers/event/dlb2/dlb2_log.h                     |   25 +
 drivers/event/dlb2/dlb2_priv.h                    |  578 ++
 drivers/event/dlb2/dlb2_selftest.c                | 1570 ++++++
 drivers/event/dlb2/dlb2_user.h                    |  679 +++
 drivers/event/dlb2/dlb2_xstats.c                  | 1240 +++++
 drivers/event/dlb2/meson.build                    |   22 +
 drivers/event/dlb2/pf/base/dlb2_hw_types.h        |  367 ++
 drivers/event/dlb2/pf/base/dlb2_mbox.h            |  596 ++
 drivers/event/dlb2/pf/base/dlb2_osdep.h           |  230 +
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h    |  440 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h      |  131 +
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h     |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h            | 2527 +++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c        | 6027 +++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h        | 1913 +++++++
 drivers/event/dlb2/pf/dlb2_main.c                 |  673 +++
 drivers/event/dlb2/pf/dlb2_main.h                 |   97 +
 drivers/event/dlb2/pf/dlb2_pf.c                   |  728 +++
 drivers/event/dlb2/rte_pmd_dlb2.c                 |   39 +
 drivers/event/dlb2/rte_pmd_dlb2.h                 |   72 +
 drivers/event/dlb2/rte_pmd_dlb2_event_version.map |    9 +
 drivers/event/meson.build                         |    3 +-
 33 files changed, 22512 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
 create mode 100644 drivers/event/dlb2/dlb2_log.h
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c
 create mode 100644 drivers/event/dlb2/dlb2_user.h
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map

-- 
2.6.4

[dpdk-dev] [PATCH v3 01/23] event/dlb2: add documentation and meson build infrastructure

[Timothy McDaniel]

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux X86 platforms at this time.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 MAINTAINERS                                       |  5 +++
 config/rte_config.h                               |  7 ++++
 doc/guides/eventdevs/dlb2.rst                     | 40 +++++++++++++++++++++++
 doc/guides/eventdevs/index.rst                    |  1 +
 drivers/event/dlb2/meson.build                    | 13 ++++++++
 drivers/event/dlb2/rte_pmd_dlb2_event_version.map |  3 ++
 drivers/event/meson.build                         |  3 +-
 7 files changed, 71 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 3b16d7a..f732f28 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1188,6 +1188,11 @@ M: Peter Mccarthy <peter.mccarthy@intel.com>
 F: drivers/event/opdl/
 F: doc/guides/eventdevs/opdl.rst
 
+Intel DLB2 Eventdev PMD
+M: Timothy McDaniel <timothy.mcdaniel@intel.com>
+F: drivers/event/dlb2/
+F: doc/guides/eventdevs/dlb2.rst
+
 
 Rawdev Drivers
 --------------
diff --git a/config/rte_config.h b/config/rte_config.h
index 0bae630..fd1b3c3 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -131,4 +131,11 @@
 /* QEDE PMD defines */
 #define RTE_LIBRTE_QEDE_FW ""
 
+/* DLB2 defines */
+#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
+#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
+#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
+#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
+
 #endif /* _RTE_CONFIG_H_ */
diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
new file mode 100644
index 0000000..e3091c9
--- /dev/null
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -0,0 +1,40 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Driver for the Intel® Dynamic Load Balancer (DLB2)
+==================================================
+
+The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
+
+Prerequisites
+-------------
+
+Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
+the basic DPDK environment.
+
+Configuration
+-------------
+
+The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
+device access. To use this operation mode, the PCIe PF device must be bound
+to a DPDK-compatible VFIO driver, such as vfio-pci.
+
+Eventdev API Notes
+------------------
+
+The DLB2 provides the functions of a DPDK event device; specifically, it
+supports atomic, ordered, and parallel scheduling events from queues to ports.
+However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
+features are abstracted by the PMD such as directed ports.
+
+In general the dlb PMD is designed for ease-of-use and does not require a
+detailed understanding of the hardware, but these details are important when
+writing high-performance code. This section describes the places where the
+eventdev API and DLB2 misalign.
+
+Flow ID
+~~~~~~~
+
+The flow ID field is preserved in the event when it is scheduled in the
+DLB2.
+
diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
index bb66a5e..738788d 100644
--- a/doc/guides/eventdevs/index.rst
+++ b/doc/guides/eventdevs/index.rst
@@ -11,6 +11,7 @@ application through the eventdev API.
     :maxdepth: 2
     :numbered:
 
+    dlb2
     dpaa
     dpaa2
     dsw
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
new file mode 100644
index 0000000..5324043
--- /dev/null
+++ b/drivers/event/dlb2/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2020 Intel Corporation
+
+if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
+        build = false
+        reason = 'only supported on ARCH_X86_64 Linux'
+        subdir_done()
+endif
+
+sources = files(
+)
+
+deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
new file mode 100644
index 0000000..4a76d1d
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index ebe76a7..1a54de4 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -5,7 +5,8 @@ if is_windows
 	subdir_done()
 endif
 
-drivers = ['dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw', 'dsw']
+drivers = ['dlb2', 'dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw',
+	   'dsw']
 if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
-- 
2.6.4

Re: [dpdk-dev] [PATCH v3 01/23] event/dlb2: add documentation and meson build infrastructure

[Jerin Jacob]

On Fri, Oct 23, 2020 at 11:59 PM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> Adds the meson build infrastructure, which includes
> compile-time constants in rte_config.h. DLB2 is
> only supported on Linux X86 platforms at this time.
>
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> Reviewed-by: Gage Eads <gage.eads@intel.com>

There is the following build error[1] with this patch when testing
with the latest dpdk-next-eventdev branch.

Timothy, Please check ./devtools/test-meson-builds.sh for each patch
before sending the next version of DLB and DLB2 to avoid delay in
pulling this series.


[1]
Message: Disabling mlx5 [drivers/vdpa/mlx5]: missing internal
dependency "common_mlx5"
drivers/meson.build:179: WARNING: Custom target input
'/export/dpdk-next-eventdev/drivers/event/dlb2/version.map' can't be
converted to File object(s).
This will become a hard error in the future.
drivers/meson.build:184: WARNING: Custom target input
'/export/dpdk-next-eventdev/drivers/event/dlb2/version.map' can't be
converted to File object(s).
This will become a hard error in the future.

drivers/meson.build:207:3: ERROR: File
/export/dpdk-next-eventdev/drivers/event/dlb2/version.map does not
exist.


> ---
>  MAINTAINERS                                       |  5 +++
>  config/rte_config.h                               |  7 ++++
>  doc/guides/eventdevs/dlb2.rst                     | 40 +++++++++++++++++++++++
>  doc/guides/eventdevs/index.rst                    |  1 +
>  drivers/event/dlb2/meson.build                    | 13 ++++++++
>  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |  3 ++
>  drivers/event/meson.build                         |  3 +-
>  7 files changed, 71 insertions(+), 1 deletion(-)
>  create mode 100644 doc/guides/eventdevs/dlb2.rst
>  create mode 100644 drivers/event/dlb2/meson.build
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 3b16d7a..f732f28 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1188,6 +1188,11 @@ M: Peter Mccarthy <peter.mccarthy@intel.com>
>  F: drivers/event/opdl/
>  F: doc/guides/eventdevs/opdl.rst
>
> +Intel DLB2 Eventdev PMD
> +M: Timothy McDaniel <timothy.mcdaniel@intel.com>
> +F: drivers/event/dlb2/
> +F: doc/guides/eventdevs/dlb2.rst
> +
>
>  Rawdev Drivers
>  --------------
> diff --git a/config/rte_config.h b/config/rte_config.h
> index 0bae630..fd1b3c3 100644
> --- a/config/rte_config.h
> +++ b/config/rte_config.h
> @@ -131,4 +131,11 @@
>  /* QEDE PMD defines */
>  #define RTE_LIBRTE_QEDE_FW ""
>
> +/* DLB2 defines */
> +#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
> +#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
> +#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
> +#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
> +#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
> +
>  #endif /* _RTE_CONFIG_H_ */
> diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
> new file mode 100644
> index 0000000..e3091c9
> --- /dev/null
> +++ b/doc/guides/eventdevs/dlb2.rst
> @@ -0,0 +1,40 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2020 Intel Corporation.
> +
> +Driver for the Intel® Dynamic Load Balancer (DLB2)
> +==================================================
> +
> +The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
> +
> +Prerequisites
> +-------------
> +
> +Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
> +the basic DPDK environment.
> +
> +Configuration
> +-------------
> +
> +The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
> +device access. To use this operation mode, the PCIe PF device must be bound
> +to a DPDK-compatible VFIO driver, such as vfio-pci.
> +
> +Eventdev API Notes
> +------------------
> +
> +The DLB2 provides the functions of a DPDK event device; specifically, it
> +supports atomic, ordered, and parallel scheduling events from queues to ports.
> +However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
> +features are abstracted by the PMD such as directed ports.
> +
> +In general the dlb PMD is designed for ease-of-use and does not require a
> +detailed understanding of the hardware, but these details are important when
> +writing high-performance code. This section describes the places where the
> +eventdev API and DLB2 misalign.
> +
> +Flow ID
> +~~~~~~~
> +
> +The flow ID field is preserved in the event when it is scheduled in the
> +DLB2.
> +
> diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
> index bb66a5e..738788d 100644
> --- a/doc/guides/eventdevs/index.rst
> +++ b/doc/guides/eventdevs/index.rst
> @@ -11,6 +11,7 @@ application through the eventdev API.
>      :maxdepth: 2
>      :numbered:
>
> +    dlb2
>      dpaa
>      dpaa2
>      dsw
> diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
> new file mode 100644
> index 0000000..5324043
> --- /dev/null
> +++ b/drivers/event/dlb2/meson.build
> @@ -0,0 +1,13 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2019-2020 Intel Corporation
> +
> +if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
> +        build = false
> +        reason = 'only supported on ARCH_X86_64 Linux'
> +        subdir_done()
> +endif
> +
> +sources = files(
> +)
> +
> +deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
> diff --git a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> new file mode 100644
> index 0000000..4a76d1d
> --- /dev/null
> +++ b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
> @@ -0,0 +1,3 @@
> +DPDK_21 {
> +       local: *;
> +};
> diff --git a/drivers/event/meson.build b/drivers/event/meson.build
> index ebe76a7..1a54de4 100644
> --- a/drivers/event/meson.build
> +++ b/drivers/event/meson.build
> @@ -5,7 +5,8 @@ if is_windows
>         subdir_done()
>  endif
>
> -drivers = ['dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw', 'dsw']
> +drivers = ['dlb2', 'dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw',
> +          'dsw']
>  if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
>         dpdk_conf.has('RTE_ARCH_ARM64'))
>         drivers += 'octeontx'
> --
> 2.6.4
>

[dpdk-dev] [PATCH v3 02/23] event/dlb2: add dynamic logging

[Timothy McDaniel]

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c      |  7 +++++++
 drivers/event/dlb2/dlb2_log.h  | 25 +++++++++++++++++++++++++
 drivers/event/dlb2/meson.build |  3 +--
 3 files changed, 33 insertions(+), 2 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_log.h

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
new file mode 100644
index 0000000..2082dd6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2.c
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <rte_log.h>
+
+RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
new file mode 100644
index 0000000..dc1481e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_log.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_EVDEV_LOG_H_
+#define _DLB2_EVDEV_LOG_H_
+
+extern int eventdev_dlb2_log_level;
+
+/* Dynamic logging */
+#define DLB2_LOG_IMPL(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
+		__func__, ##args)
+
+#define DLB2_LOG_INFO(fmt, args...) \
+	DLB2_LOG_IMPL(INFO, fmt, ## args)
+
+#define DLB2_LOG_ERR(fmt, args...) \
+	DLB2_LOG_IMPL(ERR, fmt, ## args)
+
+/* remove debug logs at compile time unless actually debugging */
+#define DLB2_LOG_DBG(fmt, args...) \
+	RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
+
+#endif /* _DLB2_EVDEV_LOG_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 5324043..435c1ee 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,7 +7,6 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files(
-)
+sources = files('dlb2.c')
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v3 03/23] event/dlb2: add private data structures and constants

[Timothy McDaniel]

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_priv.h | 575 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 575 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_priv.h

diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
new file mode 100644
index 0000000..61567a6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -0,0 +1,575 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_PRIV_H_
+#define _DLB2_PRIV_H_
+
+#include <emmintrin.h>
+#include <stdbool.h>
+
+#include <rte_eventdev.h>
+#include <rte_config.h>
+#include "dlb2_user.h"
+#include "dlb2_log.h"
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
+#else
+#define DLB2_INC_STAT(_stat, _incr_val)
+#endif
+
+#define EVDEV_DLB2_NAME_PMD dlb2_event
+
+/*  command line arg strings */
+#define NUMA_NODE_ARG "numa_node"
+#define DLB2_MAX_NUM_EVENTS "max_num_events"
+#define DLB2_NUM_DIR_CREDITS "num_dir_credits"
+#define DEV_ID_ARG "dev_id"
+#define DLB2_DEFER_SCHED_ARG "defer_sched"
+#define DLB2_QID_DEPTH_THRESH_ARG "qid_depth_thresh"
+#define DLB2_COS_ARG "cos"
+
+/* Begin HW related defines and structs */
+
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_VFS 16
+#define DLB2_MAX_NUM_LDB_QUEUES 32
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS 64
+#define DLB2_MAX_NUM_DIR_QUEUES 64
+#define DLB2_MAX_NUM_FLOWS (64 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDITS (8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS (2 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_DIR_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQOS_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_MAX_DEVICE_PATH 32
+#define DLB2_MIN_DEQUEUE_TIMEOUT_NS 1
+/* Note: "- 1" here to support the timeout range check in eventdev_autotest */
+#define DLB2_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1)
+#define DLB2_SW_CREDIT_BATCH_SZ 32
+#define DLB2_NUM_SN_GROUPS 2
+#define DLB2_MAX_LDB_SN_ALLOC 1024
+#define DLB2_MAX_QUEUE_DEPTH_THRESHOLD 8191
+
+/* 2048 total hist list entries and 64 total ldb ports, which
+ * makes for 2048/64 == 32 hist list entries per port. However, CQ
+ * depth must be a power of 2 and must also be >= HIST LIST entries.
+ * As a result we just limit the maximum dequeue depth to 32.
+ */
+#define DLB2_MIN_CQ_DEPTH 1
+#define DLB2_MAX_CQ_DEPTH 32
+#define DLB2_MIN_HARDWARE_CQ_DEPTH 8
+#define DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \
+	DLB2_MAX_CQ_DEPTH
+
+/*
+ * Static per queue/port provisioning values
+ */
+#define DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 64
+
+#define CQ_BASE(is_dir) ((is_dir) ? DLB2_DIR_CQ_BASE : DLB2_LDB_CQ_BASE)
+#define CQ_SIZE(is_dir) ((is_dir) ? DLB2_DIR_CQ_MAX_SIZE : \
+				    DLB2_LDB_CQ_MAX_SIZE)
+#define PP_BASE(is_dir) ((is_dir) ? DLB2_DIR_PP_BASE : DLB2_LDB_PP_BASE)
+
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+
+#define DLB2_NUM_QES_PER_CACHE_LINE 4
+
+#define DLB2_MAX_ENQUEUE_DEPTH 64
+#define DLB2_MIN_ENQUEUE_DEPTH 4
+
+#define DLB2_NAME_SIZE 64
+
+#define DLB2_1K 1024
+#define DLB2_2K (2 * DLB2_1K)
+#define DLB2_4K (4 * DLB2_1K)
+#define DLB2_16K (16 * DLB2_1K)
+#define DLB2_32K (32 * DLB2_1K)
+#define DLB2_1MB (DLB2_1K * DLB2_1K)
+#define DLB2_16MB (16 * DLB2_1MB)
+
+/* Use the upper 3 bits of the event priority to select the DLB2 priority */
+#define EV_TO_DLB2_PRIO(x) ((x) >> 5)
+#define DLB2_TO_EV_PRIO(x) ((x) << 5)
+
+enum dlb2_hw_port_types {
+	DLB2_LDB_PORT,
+	DLB2_DIR_PORT,
+	DLB2_NUM_PORT_TYPES /* Must be last */
+};
+
+enum dlb2_hw_queue_types {
+	DLB2_LDB_QUEUE,
+	DLB2_DIR_QUEUE,
+	DLB2_NUM_QUEUE_TYPES /* Must be last */
+};
+
+#define PORT_TYPE(p) ((p)->is_directed ? DLB2_DIR_PORT : DLB2_LDB_PORT)
+
+/* Do not change - must match hardware! */
+enum dlb2_hw_sched_type {
+	DLB2_SCHED_ATOMIC = 0,
+	DLB2_SCHED_UNORDERED,
+	DLB2_SCHED_ORDERED,
+	DLB2_SCHED_DIRECTED,
+	/* DLB2_NUM_HW_SCHED_TYPES must be last */
+	DLB2_NUM_HW_SCHED_TYPES
+};
+
+struct dlb2_hw_rsrcs {
+	int32_t nb_events_limit;
+	uint32_t num_queues;		/* Total queues (lb + dir) */
+	uint32_t num_ldb_queues;	/* Number of available ldb queues */
+	uint32_t num_ldb_ports;         /* Number of load balanced ports */
+	uint32_t num_dir_ports;         /* Number of directed ports */
+	uint32_t num_ldb_credits;       /* Number of load balanced credits */
+	uint32_t num_dir_credits;       /* Number of directed credits */
+	uint32_t reorder_window_size;   /* Size of reorder window */
+};
+
+struct dlb2_hw_resource_info {
+	/**> Max resources that can be provided */
+	struct dlb2_hw_rsrcs hw_rsrc_max;
+	int num_sched_domains;
+	uint32_t socket_id;
+};
+
+enum dlb2_enqueue_type {
+	/**>
+	 * New : Used to inject a new packet into the QM.
+	 */
+	DLB2_ENQ_NEW,
+	/**>
+	 * Forward : Enqueues a packet, and
+	 *  - if atomic: release any lock it holds in the QM
+	 *  - if ordered: release the packet for egress re-ordering
+	 */
+	DLB2_ENQ_FWD,
+	/**>
+	 * Enqueue Drop : Release an inflight packet. Must be called with
+	 * event == NULL. Used to drop a packet.
+	 *
+	 * Note that all packets dequeued from a load-balanced port must be
+	 * released, either with DLB2_ENQ_DROP or DLB2_ENQ_FWD.
+	 */
+	DLB2_ENQ_DROP,
+
+	/* marker for array sizing etc. */
+	_DLB2_NB_ENQ_TYPES
+};
+
+/* hw-specific format - do not change */
+
+struct dlb2_event_type {
+	uint8_t major:4;
+	uint8_t unused:4;
+	uint8_t sub;
+};
+
+union dlb2_opaque_data {
+	uint16_t opaque_data;
+	struct dlb2_event_type event_type;
+};
+
+struct dlb2_msg_info {
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+};
+
+#define DLB2_NEW_CMD_BYTE 0x08
+#define DLB2_FWD_CMD_BYTE 0x0A
+#define DLB2_COMP_CMD_BYTE 0x02
+#define DLB2_POP_CMD_BYTE 0x01
+#define DLB2_NOOP_CMD_BYTE 0x00
+
+/* hw-specific format - do not change */
+struct dlb2_enqueue_qe {
+	uint64_t data;
+	/* Word 3 */
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	/* Word 4 */
+	uint16_t lock_id;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_cq_pop_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t tokens:10;
+	uint16_t rsvd2:6;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_dequeue_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t flow_id:16; /* was pp_id in v1 */
+	uint8_t debug;
+	uint8_t cq_gen:1;
+	uint8_t qid_depth:2; /* 2 bits in v2 */
+	uint8_t rsvd1:2;
+	uint8_t error:1;
+	uint8_t rsvd2:2;
+};
+
+union dlb2_port_config {
+	struct dlb2_create_ldb_port_args ldb;
+	struct dlb2_create_dir_port_args dir;
+};
+
+enum dlb2_port_state {
+	PORT_CLOSED,
+	PORT_STARTED,
+	PORT_STOPPED
+};
+
+enum dlb2_configuration_state {
+	/* The resource has not been configured */
+	DLB2_NOT_CONFIGURED,
+	/* The resource was configured, but the device was stopped */
+	DLB2_PREV_CONFIGURED,
+	/* The resource is currently configured */
+	DLB2_CONFIGURED
+};
+
+struct dlb2_port {
+	uint32_t id;
+	bool is_directed;
+	bool gen_bit;
+	uint16_t dir_credits;
+	uint32_t dequeue_depth;
+	union dlb2_port_config cfg;
+	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
+	uint16_t cached_ldb_credits;
+	uint16_t ldb_credits;
+	uint16_t cached_dir_credits;
+	bool int_armed;
+	uint16_t owed_tokens;
+	int16_t issued_releases;
+	int cq_depth;
+	uint16_t cq_idx;
+	uint16_t cq_idx_unmasked;
+	uint16_t cq_depth_mask;
+	uint16_t gen_bit_shift;
+	enum dlb2_port_state state;
+	enum dlb2_configuration_state config_state;
+	int num_mapped_qids;
+	uint8_t *qid_mappings;
+	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
+	struct dlb2_enqueue_qe *int_arm_qe;
+	struct dlb2_cq_pop_qe *consume_qe;
+	struct dlb2_eventdev *dlb2; /* back ptr */
+	struct dlb2_eventdev_port *ev_port; /* back ptr */
+};
+
+/* Per-process per-port mmio and memory pointers */
+struct process_local_port_data {
+	uint64_t *pp_addr;
+	struct dlb2_dequeue_qe *cq_base;
+	const struct rte_memzone *mz;
+	bool mmaped;
+};
+
+struct dlb2_eventdev;
+
+struct dlb2_config {
+	int configured;
+	int reserved;
+	uint32_t num_ldb_credits;
+	uint32_t num_dir_credits;
+	struct dlb2_create_sched_domain_args resources;
+};
+
+enum dlb2_cos {
+	DLB2_COS_DEFAULT = -1,
+	DLB2_COS_0 = 0,
+	DLB2_COS_1,
+	DLB2_COS_2,
+	DLB2_COS_3
+};
+
+struct dlb2_hw_dev {
+	struct dlb2_config cfg;
+	struct dlb2_hw_resource_info info;
+	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
+	uint32_t domain_id;
+	enum dlb2_cos cos_id;
+	rte_spinlock_t resource_lock; /* for MP support */
+} __rte_cache_aligned;
+
+/* End HW related defines and structs */
+
+/* Begin DLB2 PMD Eventdev related defines and structs */
+
+#define DLB2_MAX_NUM_QUEUES \
+	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
+
+#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS + DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
+
+/** Structure to hold the queue to port link establishment attributes */
+
+struct dlb2_event_queue_link {
+	uint8_t queue_id;
+	uint8_t priority;
+	bool mapped;
+	bool valid;
+};
+
+struct dlb2_traffic_stats {
+	uint64_t rx_ok;
+	uint64_t rx_drop;
+	uint64_t rx_interrupt_wait;
+	uint64_t rx_umonitor_umwait;
+	uint64_t tx_ok;
+	uint64_t total_polls;
+	uint64_t zero_polls;
+	uint64_t tx_nospc_ldb_hw_credits;
+	uint64_t tx_nospc_dir_hw_credits;
+	uint64_t tx_nospc_inflight_max;
+	uint64_t tx_nospc_new_event_limit;
+	uint64_t tx_nospc_inflight_credits;
+};
+
+/* DLB2 HW sets the 2bit qid_depth in rx QEs based on the programmable depth
+ * threshold. The global default value in config/common_base (or rte_config.h)
+ * can be overridden on a per-qid basis using a vdev command line parameter.
+ * 3: depth > threshold
+ * 2: threshold >= depth > 3/4 threshold
+ * 1: 3/4 threshold >= depth > 1/2 threshold
+ * 0: depth <= 1/2 threshold.
+ */
+#define DLB2_QID_DEPTH_LE50 0
+#define DLB2_QID_DEPTH_GT50_LE75 1
+#define DLB2_QID_DEPTH_GT75_LE100 2
+#define DLB2_QID_DEPTH_GT100 3
+#define DLB2_NUM_QID_DEPTH_STAT_VALS 4 /* 2 bits */
+
+struct dlb2_queue_stats {
+	uint64_t enq_ok;
+	uint64_t qid_depth[DLB2_NUM_QID_DEPTH_STAT_VALS];
+};
+
+struct dlb2_port_stats {
+	struct dlb2_traffic_stats traffic;
+	uint64_t tx_op_cnt[4]; /* indexed by rte_event.op */
+	uint64_t tx_implicit_rel;
+	uint64_t tx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t tx_invalid;
+	uint64_t rx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t rx_sched_invalid;
+	struct dlb2_queue_stats queue[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_eventdev_port {
+	struct dlb2_port qm_port; /* hw specific data structure */
+	struct rte_event_port_conf conf; /* user-supplied configuration */
+	uint16_t inflight_credits; /* num credits this port has right now */
+	uint16_t credit_update_quanta;
+	struct dlb2_eventdev *dlb2; /* backlink optimization */
+	struct dlb2_port_stats stats __rte_cache_aligned;
+	struct dlb2_event_queue_link link[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	int num_links;
+	uint32_t id; /* port id */
+	/* num releases yet to be completed on this port.
+	 * Only applies to load-balanced ports.
+	 */
+	uint16_t outstanding_releases;
+	uint16_t inflight_max; /* app requested max inflights for this port */
+	/* setup_done is set when the event port is setup */
+	bool setup_done;
+	/* enq_configured is set when the qm port is created */
+	bool enq_configured;
+	uint8_t implicit_release; /* release events before dequeueing */
+}  __rte_cache_aligned;
+
+struct dlb2_queue {
+	uint32_t num_qid_inflights; /* User config */
+	uint32_t num_atm_inflights; /* User config */
+	enum dlb2_configuration_state config_state;
+	int sched_type; /* LB queue only */
+	uint32_t id;
+	bool is_directed;
+};
+
+struct dlb2_eventdev_queue {
+	struct dlb2_queue qm_queue;
+	struct rte_event_queue_conf conf; /* User config */
+	int depth_threshold; /* use default if 0 */
+	uint32_t id;
+	bool setup_done;
+	uint8_t num_links;
+};
+
+enum dlb2_run_state {
+	DLB2_RUN_STATE_STOPPED = 0,
+	DLB2_RUN_STATE_STOPPING,
+	DLB2_RUN_STATE_STARTING,
+	DLB2_RUN_STATE_STARTED
+};
+
+struct dlb2_eventdev {
+	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
+	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each queue */
+	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
+	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each port */
+	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
+	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
+	struct dlb2_get_num_resources_args hw_rsrc_query_results;
+	uint32_t xstats_count_mode_queue;
+	struct dlb2_hw_dev qm_instance; /* strictly hw related */
+	uint64_t global_dequeue_wait_ticks;
+	struct dlb2_xstats_entry *xstats;
+	struct rte_eventdev *event_dev; /* backlink to dev */
+	uint32_t xstats_count_mode_dev;
+	uint32_t xstats_count_mode_port;
+	uint32_t xstats_count;
+	uint32_t inflights; /* use __atomic builtins */
+	uint32_t new_event_limit;
+	int max_num_events_override;
+	int num_dir_credits_override;
+	volatile enum dlb2_run_state run_state;
+	uint16_t num_dir_queues; /* total num of evdev dir queues requested */
+	uint16_t num_dir_credits;
+	uint16_t num_ldb_credits;
+	uint16_t num_queues; /* total queues */
+	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested */
+	uint16_t num_ports; /* total num of evdev ports requested */
+	uint16_t num_ldb_ports; /* total num of ldb ports requested */
+	uint16_t num_dir_ports; /* total num of dir ports requested */
+	bool umwait_allowed;
+	bool global_dequeue_wait; /* Not using per dequeue wait if true */
+	bool defer_sched;
+	enum dlb2_cq_poll_modes poll_mode;
+	uint8_t revision;
+	bool configured;
+	uint16_t max_ldb_credits;
+	uint16_t max_dir_credits;
+
+	/* force hw credit pool counters into exclusive cache lines */
+
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t ldb_credit_pool __rte_cache_aligned;
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t dir_credit_pool __rte_cache_aligned;
+};
+
+/* used for collecting and passing around the dev args */
+struct dlb2_qid_depth_thresholds {
+	int val[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_devargs {
+	int socket_id;
+	int max_num_events;
+	int num_dir_credits_override;
+	int dev_id;
+	int defer_sched;
+	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
+	enum dlb2_cos cos_id;
+};
+
+/* End Eventdev related defines and structs */
+
+/* Forwards for non-inlined functions */
+
+void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
+
+int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
+
+void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
+
+int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n);
+
+int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstat_names,
+		unsigned int *ids, unsigned int size);
+
+uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+					  const char *name, unsigned int *id);
+
+int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode,
+		int16_t queue_port_id,
+		const uint32_t ids[],
+		uint32_t nb_ids);
+
+int test_dlb2_eventdev(void);
+
+int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+				const char *name,
+				struct dlb2_devargs *dlb2_args);
+
+int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+				  const char *name);
+
+uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *queue);
+
+int dlb2_parse_params(const char *params,
+		      const char *name,
+		      struct dlb2_devargs *dlb2_args);
+
+/* Extern globals */
+extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
+
+#endif	/* _DLB2_PRIV_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v3 04/23] event/dlb2: add definitions shared with LKM or shared code

[Timothy McDaniel]

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_user.h | 679 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 679 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_user.h

diff --git a/drivers/event/dlb2/dlb2_user.h b/drivers/event/dlb2/dlb2_user.h
new file mode 100644
index 0000000..f4bda78
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_user.h
@@ -0,0 +1,679 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#define DLB2_MAX_NAME_LEN 64
+
+#include <linux/types.h>
+
+enum dlb2_error {
+	DLB2_ST_SUCCESS = 0,
+	DLB2_ST_NAME_EXISTS,
+	DLB2_ST_DOMAIN_UNAVAILABLE,
+	DLB2_ST_LDB_PORTS_UNAVAILABLE,
+	DLB2_ST_DIR_PORTS_UNAVAILABLE,
+	DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB2_ST_INVALID_DOMAIN_ID,
+	DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB2_ST_INVALID_LDB_QUEUE_ID,
+	DLB2_ST_INVALID_CQ_DEPTH,
+	DLB2_ST_INVALID_CQ_VIRT_ADDR,
+	DLB2_ST_INVALID_PORT_ID,
+	DLB2_ST_INVALID_QID,
+	DLB2_ST_INVALID_PRIORITY,
+	DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+	DLB2_ST_INVALID_DIR_QUEUE_ID,
+	DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB2_ST_DOMAIN_NOT_CONFIGURED,
+	DLB2_ST_INTERNAL_ERROR,
+	DLB2_ST_DOMAIN_IN_USE,
+	DLB2_ST_DOMAIN_NOT_FOUND,
+	DLB2_ST_QUEUE_NOT_FOUND,
+	DLB2_ST_DOMAIN_STARTED,
+	DLB2_ST_DOMAIN_NOT_STARTED,
+	DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB2_ST_DOMAIN_RESET_FAILED,
+	DLB2_ST_MBOX_ERROR,
+	DLB2_ST_INVALID_HIST_LIST_DEPTH,
+	DLB2_ST_NO_MEMORY,
+	DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB2_ST_INVALID_COS_ID,
+};
+
+static const char dlb2_error_strings[][128] = {
+	"DLB2_ST_SUCCESS",
+	"DLB2_ST_NAME_EXISTS",
+	"DLB2_ST_DOMAIN_UNAVAILABLE",
+	"DLB2_ST_LDB_PORTS_UNAVAILABLE",
+	"DLB2_ST_DIR_PORTS_UNAVAILABLE",
+	"DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+	"DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+	"DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+	"DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+	"DLB2_ST_INVALID_DOMAIN_ID",
+	"DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+	"DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+	"DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+	"DLB2_ST_INVALID_LDB_QUEUE_ID",
+	"DLB2_ST_INVALID_CQ_DEPTH",
+	"DLB2_ST_INVALID_CQ_VIRT_ADDR",
+	"DLB2_ST_INVALID_PORT_ID",
+	"DLB2_ST_INVALID_QID",
+	"DLB2_ST_INVALID_PRIORITY",
+	"DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+	"DLB2_ST_INVALID_DIR_QUEUE_ID",
+	"DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+	"DLB2_ST_DOMAIN_NOT_CONFIGURED",
+	"DLB2_ST_INTERNAL_ERROR",
+	"DLB2_ST_DOMAIN_IN_USE",
+	"DLB2_ST_DOMAIN_NOT_FOUND",
+	"DLB2_ST_QUEUE_NOT_FOUND",
+	"DLB2_ST_DOMAIN_STARTED",
+	"DLB2_ST_DOMAIN_NOT_STARTED",
+	"DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+	"DLB2_ST_DOMAIN_RESET_FAILED",
+	"DLB2_ST_MBOX_ERROR",
+	"DLB2_ST_INVALID_HIST_LIST_DEPTH",
+	"DLB2_ST_NO_MEMORY",
+	"DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+	"DLB2_ST_INVALID_COS_ID",
+};
+
+struct dlb2_cmd_response {
+	__u32 status; /* Interpret using enum dlb2_error */
+	__u32 id;
+};
+
+/*******************/
+/* 'dlb2' commands */
+/*******************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+	DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	This ioctl interface is the same in all driver versions and is always
+ *	the first ioctl.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+	/* Output parameters */
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ *	load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ *		percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ *	load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ *   ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+enum dlb2_cq_poll_modes {
+	DLB2_CQ_POLL_MODE_STD,
+	DLB2_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB2_CQ_POLL_MODE must be last */
+	NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/********************************/
+/* 'scheduling domain' commands */
+/********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_start_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
+#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
+
+
+#endif /* __DLB2_USER_H */
-- 
2.6.4

[dpdk-dev] [PATCH v3 05/23] event/dlb2: add inline functions

[Timothy McDaniel]

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_inline_fns.h | 61 ++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h

diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
new file mode 100644
index 0000000..ccbc4f4
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_inline_fns.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_INLINE_FNS_H_
+#define _DLB2_INLINE_FNS_H_
+
+/* Inline functions required in more than one source file. */
+
+static inline struct dlb2_eventdev *
+dlb2_pmd_priv(const struct rte_eventdev *eventdev)
+{
+	return eventdev->data->dev_private;
+}
+
+static inline void
+dlb2_umonitor(volatile void *addr)
+{
+	asm volatile(".byte 0xf3, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (addr));
+}
+
+static inline void
+dlb2_umwait(int state, uint64_t timeout)
+{
+	uint32_t eax = timeout & UINT32_MAX;
+	uint32_t edx = timeout >> 32;
+
+	asm volatile(".byte 0xf2, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (state),  "a" (eax), "d" (edx));
+}
+
+static inline void
+dlb2_movntdq_single(void *pp_addr, void *qe4)
+{
+	long long *_qe  = (long long *)qe4;
+	__v2di src_data0 = (__v2di){_qe[0], _qe[1]};
+
+	__builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
+}
+
+static inline void
+dlb2_cldemote(void *addr)
+{
+	/* Load addr into RSI, then demote the cache line of the address
+	 * contained in that register.
+	 */
+	asm volatile(".byte 0x0f, 0x1c, 0x06" :: "S" (addr));
+}
+
+static inline void
+dlb2_movdir64b(void *pp_addr, void *qe4)
+{
+	asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
+		     :
+		     : "a" (pp_addr), "d" (qe4));
+}
+
+#endif /* _DLB2_INLINE_FNS_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v3 06/23] event/dlb2: add eventdev probe

[Timothy McDaniel]

Add the eventdev portion of probe, and parse command line
options, but do not initialize hardware.

Changes since v2 patch-set probe:

Primary and secondary probe-time init has been removed, and
will be introduced in subsequent patches contained in
the v3 patch-set.

Hardware init has been moved to a subsequent patch in order to
minimize the patch size. The files dlb2/pf/dlb2_pf.c and
dlb2/pf/dlb_main.c include stubs of hardware init functions,
and these will be replaced with the real versions when the
hardware layer is committed.

Initialization of the flexible interface layer has been moved to
a subsequent patch in order to minimize patch size.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                      |  343 ++++
 drivers/event/dlb2/meson.build                 |    5 +-
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +++
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  621 ++++++
 drivers/event/dlb2/pf/dlb2_main.h              |   98 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  196 ++
 13 files changed, 7497 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2082dd6..1b11979 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2,6 +2,349 @@
  * Copyright(c) 2016-2020 Intel Corporation
  */
 
+#include <assert.h>
+#include <errno.h>
+#include <nmmintrin.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_cycles.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_io.h>
+#include <rte_kvargs.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_ring.h>
+#include <rte_string_fns.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+/*
+ * Resources exposed to eventdev. Some values overridden at runtime using
+ * values returned by the DLB kernel driver.
+ */
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
+#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
+#endif
+
+struct process_local_port_data
+dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
+
+#define DLB2_BASE_10 10
+
+static int
+dlb2_string_to_int(int *result, const char *str)
+{
+	long ret;
+	char *endptr;
+
+	if (str == NULL || result == NULL)
+		return -EINVAL;
+
+	errno = 0;
+	ret = strtol(str, &endptr, DLB2_BASE_10);
+	if (errno)
+		return -errno;
+
+	/* long int and int may be different width for some architectures */
+	if (ret < INT_MIN || ret > INT_MAX || endptr == str)
+		return -EINVAL;
+
+	*result = ret;
+	return 0;
+}
+
+static int
+set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
+{
+	int *socket_id = opaque;
+	int ret;
+
+	ret = dlb2_string_to_int(socket_id, value);
+	if (ret < 0)
+		return ret;
+
+	if (*socket_id > RTE_MAX_NUMA_NODES)
+		return -EINVAL;
+	return 0;
+}
+
+static int
+set_max_num_events(const char *key __rte_unused,
+		   const char *value,
+		   void *opaque)
+{
+	int *max_num_events = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(max_num_events, value);
+	if (ret < 0)
+		return ret;
+
+	if (*max_num_events < 0 || *max_num_events >
+			DLB2_MAX_NUM_LDB_CREDITS) {
+		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
+			     DLB2_MAX_NUM_LDB_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_num_dir_credits(const char *key __rte_unused,
+		    const char *value,
+		    void *opaque)
+{
+	int *num_dir_credits = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(num_dir_credits, value);
+	if (ret < 0)
+		return ret;
+
+	if (*num_dir_credits < 0 ||
+	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
+		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
+			     DLB2_MAX_NUM_DIR_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_dev_id(const char *key __rte_unused,
+	   const char *value,
+	   void *opaque)
+{
+	int *dev_id = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(dev_id, value);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+set_cos(const char *key __rte_unused,
+	const char *value,
+	void *opaque)
+{
+	enum dlb2_cos *cos_id = opaque;
+	int x = 0;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(&x, value);
+	if (ret < 0)
+		return ret;
+
+	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
+		DLB2_LOG_ERR(
+			"COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
+			x);
+		return -EINVAL;
+	}
+
+	*cos_id = x;
+
+	return 0;
+}
+
+
+static int
+set_qid_depth_thresh(const char *key __rte_unused,
+		     const char *value,
+		     void *opaque)
+{
+	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
+	int first, last, thresh, i;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	/* command line override may take one of the following 3 forms:
+	 * qid_depth_thresh=all:<threshold_value> ... all queues
+	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
+	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
+	 */
+	if (sscanf(value, "all:%d", &thresh) == 1) {
+		first = 0;
+		last = DLB2_MAX_NUM_QUEUES - 1;
+	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
+		/* we have everything we need */
+	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
+		last = first;
+	} else {
+		DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val, qid-qid:val, or qid:val\n");
+		return -EINVAL;
+	}
+
+	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
+		return -EINVAL;
+	}
+
+	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
+			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
+		return -EINVAL;
+	}
+
+	for (i = first; i <= last; i++)
+		qid_thresh->val[i] = thresh; /* indexed by qid */
+
+	return 0;
+}
+
+int
+dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+			    const char *name,
+			    struct dlb2_devargs *dlb2_args)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+	RTE_SET_USED(dlb2_args);
+
+	return 0;
+}
+
+int
+dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+			      const char *name)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+int
+dlb2_parse_params(const char *params,
+		  const char *name,
+		  struct dlb2_devargs *dlb2_args)
+{
+	int ret = 0;
+	static const char * const args[] = { NUMA_NODE_ARG,
+					     DLB2_MAX_NUM_EVENTS,
+					     DLB2_NUM_DIR_CREDITS,
+					     DEV_ID_ARG,
+					     DLB2_QID_DEPTH_THRESH_ARG,
+					     DLB2_COS_ARG,
+					     NULL };
+
+	if (params != NULL && params[0] != '\0') {
+		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
+
+		if (kvlist == NULL) {
+			RTE_LOG(INFO, PMD,
+				"Ignoring unsupported parameters when creating device '%s'\n",
+				name);
+		} else {
+			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
+						     set_numa_node,
+						     &dlb2_args->socket_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing numa node parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
+						 set_max_num_events,
+						 &dlb2_args->max_num_events);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist,
+					DLB2_NUM_DIR_CREDITS,
+					set_num_dir_credits,
+					&dlb2_args->num_dir_credits_override);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
+						 set_dev_id,
+						 &dlb2_args->dev_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(
+					kvlist,
+					DLB2_QID_DEPTH_THRESH_ARG,
+					set_qid_depth_thresh,
+					&dlb2_args->qid_depth_thresholds);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
+						 set_cos,
+						 &dlb2_args->cos_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing cos parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
 
+			rte_kvargs_free(kvlist);
+		}
+	}
+	return ret;
+}
 RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 435c1ee..1eab9b9 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,6 +7,9 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files('dlb2.c')
+sources = files('dlb2.c',
+		'pf/dlb2_main.c',
+		'pf/dlb2_pf.c'
+)
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
new file mode 100644
index 0000000..1d99f1e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_osdep_list.h"
+#include "dlb2_osdep_types.h"
+
+#define DLB2_MAX_NUM_VDEVS			16
+#define DLB2_MAX_NUM_DOMAINS			32
+#define DLB2_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS			64
+#define DLB2_MAX_NUM_DIR_PORTS			64
+#define DLB2_MAX_NUM_LDB_CREDITS		(8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS		(2 * 1024)
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB2_MAX_NUM_AQED_ENTRIES		2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB2_QID_PRIORITIES			8
+#define DLB2_NUM_ARB_WEIGHTS			8
+#define DLB2_MAX_WEIGHT				255
+#define DLB2_NUM_COS_DOMAINS			4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#ifdef FPGA
+#define DLB2_HZ					2000000
+#else
+#define DLB2_HZ					800000000
+#endif
+
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
+
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+	DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+	DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID		3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID		4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW		5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ	1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
+#define DLB2_VF_BASE_CQ_VECTOR_ID	     0
+#define DLB2_VF_LAST_CQ_VECTOR_ID	     30
+#define DLB2_VF_MBOX_VECTOR_ID		     31
+#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
+
+#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
+					     DLB2_MAX_NUM_DIR_PORTS + 1)
+
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE   0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
+				DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE   0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
+				DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE       0x2100000
+#define DLB2_LDB_PP_STRIDE     0x1000
+#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
+				DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE       0x2000000
+#define DLB2_DIR_PP_STRIDE     0x1000
+#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
+				DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
+struct dlb2_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb2_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb2_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
+struct dlb2_ldb_queue {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB2_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB2_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB2_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB2_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+	enum dlb2_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb2_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+	u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+	u32 bound[6] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & (1 << i))) {
+			group->slot_use_bitmap |= 1 << i;
+			return i;
+		}
+	}
+
+	return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+	int i, cnt = 0;
+
+	for (i = 0; i < 32; i++)
+		cnt += !!(group->slot_use_bitmap & (1 << i));
+
+	return cnt;
+}
+
+struct dlb2_hw_domain {
+	struct dlb2_function_resources *parent_func;
+	struct dlb2_list_entry func_list;
+	struct dlb2_list_head used_ldb_queues;
+	struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head used_dir_pq_pairs;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb2_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb2_bitmap;
+
+struct dlb2_function_resources {
+	struct dlb2_list_head avail_domains;
+	struct dlb2_list_head used_domains;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	struct dlb2_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+	struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+	struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+	struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
+struct dlb2_mbox {
+	u32 *mbox;
+	u32 *isr_in_progress;
+};
+
+struct dlb2_sw_mbox {
+	struct dlb2_mbox vdev_to_pf;
+	struct dlb2_mbox pf_to_vdev;
+	void (*pf_to_vdev_inject)(void *arg);
+	void *pf_to_vdev_inject_arg;
+};
+
+struct dlb2_hw {
+	/* BAR 0 address */
+	void  *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void  *func_kva;
+	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb2_hw_resources rsrcs;
+	struct dlb2_function_resources pf;
+	struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+	struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+	/* Virtualization */
+	int virt_mode;
+	struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
+	unsigned int pasid[DLB2_MAX_NUM_VDEVS];
+};
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
new file mode 100644
index 0000000..ce462c0
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
@@ -0,0 +1,596 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_BASE_DLB2_MBOX_H
+#define __DLB2_BASE_DLB2_MBOX_H
+
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+
+#define DLB2_MBOX_INTERFACE_VERSION 1
+
+/*
+ * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
+ * to send requests of its own (e.g. notifying a VF of an impending FLR).
+ * To avoid communication race conditions, e.g. the PF sends a response and then
+ * sends a request before the VF reads the response, the PF->VF mailbox is
+ * divided into two sections:
+ * - Bytes 0-47: PF responses
+ * - Bytes 48-63: PF requests
+ *
+ * Partitioning the PF->VF mailbox allows responses and requests to occupy the
+ * mailbox simultaneously.
+ */
+#define DLB2_PF2VF_RESP_BYTES	  48
+#define DLB2_PF2VF_RESP_BASE	  0
+#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
+
+#define DLB2_PF2VF_REQ_BYTES	  16
+#define DLB2_PF2VF_REQ_BASE	  (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
+#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
+
+/*
+ * Similarly, the VF->PF mailbox is divided into two sections:
+ * - Bytes 0-239: VF requests
+ * -- (Bytes 0-3 are unused due to a hardware errata)
+ * - Bytes 240-255: VF responses
+ */
+#define DLB2_VF2PF_REQ_BYTES	 236
+#define DLB2_VF2PF_REQ_BASE	 4
+#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
+
+#define DLB2_VF2PF_RESP_BYTES	  16
+#define DLB2_VF2PF_RESP_BASE	  (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
+#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
+
+/* VF-initiated commands */
+enum dlb2_mbox_cmd_type {
+	DLB2_MBOX_CMD_REGISTER,
+	DLB2_MBOX_CMD_UNREGISTER,
+	DLB2_MBOX_CMD_GET_NUM_RESOURCES,
+	DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
+	DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
+	DLB2_MBOX_CMD_CREATE_LDB_PORT,
+	DLB2_MBOX_CMD_CREATE_DIR_PORT,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT,
+	DLB2_MBOX_CMD_DISABLE_LDB_PORT,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT,
+	DLB2_MBOX_CMD_DISABLE_DIR_PORT,
+	DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_MAP_QID,
+	DLB2_MBOX_CMD_UNMAP_QID,
+	DLB2_MBOX_CMD_START_DOMAIN,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
+	DLB2_MBOX_CMD_ARM_CQ_INTR,
+	DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
+	DLB2_MBOX_CMD_GET_SN_ALLOCATION,
+	DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
+	DLB2_MBOX_CMD_GET_COS_BW,
+	DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
+	DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_QE_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_CMD_TYPES,
+};
+
+static const char dlb2_mbox_cmd_type_strings[][128] = {
+	"DLB2_MBOX_CMD_REGISTER",
+	"DLB2_MBOX_CMD_UNREGISTER",
+	"DLB2_MBOX_CMD_GET_NUM_RESOURCES",
+	"DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_LDB_PORT",
+	"DLB2_MBOX_CMD_CREATE_DIR_PORT",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_DISABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_DISABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_MAP_QID",
+	"DLB2_MBOX_CMD_UNMAP_QID",
+	"DLB2_MBOX_CMD_START_DOMAIN",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
+	"DLB2_MBOX_CMD_ARM_CQ_INTR",
+	"DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
+	"DLB2_MBOX_CMD_GET_SN_ALLOCATION",
+	"DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
+	"DLB2_MBOX_CMD_GET_COS_BW",
+	"DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
+	"DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
+};
+
+/* PF-initiated commands */
+enum dlb2_mbox_vf_cmd_type {
+	DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
+	DLB2_MBOX_VF_CMD_NOTIFICATION,
+	DLB2_MBOX_VF_CMD_IN_USE,
+
+	/* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_VF_CMD_TYPES,
+};
+
+static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
+	"DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
+	"DLB2_MBOX_VF_CMD_NOTIFICATION",
+	"DLB2_MBOX_VF_CMD_IN_USE",
+};
+
+#define DLB2_MBOX_CMD_TYPE(hdr) \
+	(((struct dlb2_mbox_req_hdr *)hdr)->type)
+#define DLB2_MBOX_CMD_STRING(hdr) \
+	dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
+
+enum dlb2_mbox_status_type {
+	DLB2_MBOX_ST_SUCCESS,
+	DLB2_MBOX_ST_INVALID_CMD_TYPE,
+	DLB2_MBOX_ST_VERSION_MISMATCH,
+	DLB2_MBOX_ST_INVALID_OWNER_VF,
+};
+
+static const char dlb2_mbox_status_type_strings[][128] = {
+	"DLB2_MBOX_ST_SUCCESS",
+	"DLB2_MBOX_ST_INVALID_CMD_TYPE",
+	"DLB2_MBOX_ST_VERSION_MISMATCH",
+	"DLB2_MBOX_ST_INVALID_OWNER_VF",
+};
+
+#define DLB2_MBOX_ST_TYPE(hdr) \
+	(((struct dlb2_mbox_resp_hdr *)hdr)->status)
+#define DLB2_MBOX_ST_STRING(hdr) \
+	dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
+
+/* This structure is always the first field in a request structure */
+struct dlb2_mbox_req_hdr {
+	u32 type;
+};
+
+/* This structure is always the first field in a response structure */
+struct dlb2_mbox_resp_hdr {
+	u32 status;
+};
+
+struct dlb2_mbox_register_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 min_interface_version;
+	u16 max_interface_version;
+};
+
+struct dlb2_mbox_register_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 interface_version;
+	u8 pf_id;
+	u8 vf_id;
+	u8 is_auxiliary_vf;
+	u8 primary_vf_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u16 num_sched_domains;
+	u16 num_ldb_queues;
+	u16 num_ldb_ports;
+	u16 num_cos_ldb_ports[4];
+	u16 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 max_contiguous_hist_list_entries;
+	u16 num_ldb_credits;
+	u16 num_dir_credits;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 num_ldb_queues;
+	u32 num_ldb_ports;
+	u32 num_cos_ldb_ports[4];
+	u32 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u8 cos_strict;
+	u8 padding0[3];
+	u32 padding1;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 num_sequence_numbers;
+	u32 num_qid_inflights;
+	u32 num_atomic_inflights;
+	u32 lock_id_comp_level;
+	u32 depth_threshold;
+	u32 padding;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 depth_threshold;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u16 cq_depth;
+	u16 cq_history_list_size;
+	u8 cos_id;
+	u8 cos_strict;
+	u16 padding1;
+	u64 cq_base_address;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u64 cq_base_address;
+	u16 cq_depth;
+	u16 padding0;
+	s32 queue_id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_map_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+	u32 priority;
+	u32 padding0;
+};
+
+struct dlb2_mbox_map_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_start_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+};
+
+struct dlb2_mbox_start_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 is_ldb;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding0;
+};
+
+/*
+ * The alert_id and aux_alert_data follows the format of the alerts defined in
+ * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
+ * the aux_alert_data value varies depending on the alert.
+ */
+struct dlb2_mbox_vf_alert_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 alert_id;
+	u32 aux_alert_data;
+};
+
+enum dlb2_mbox_vf_notification_type {
+	DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
+	DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
+
+	/* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
+	NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
+};
+
+struct dlb2_mbox_vf_notification_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 notification;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 in_use;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 num;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 cos_id;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 mode;
+};
+
+#endif /* __DLB2_BASE_DLB2_MBOX_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
new file mode 100644
index 0000000..c4c34eb
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
@@ -0,0 +1,230 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_H
+#define __DLB2_OSDEP_H
+
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <rte_string_fns.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include "../dlb2_main.h"
+#include "dlb2_resource.h"
+#include "../../dlb2_log.h"
+#include "../../dlb2_user.h"
+
+
+#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
+#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
+#define DLB2_CSR_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
+#define DLB2_FUNC_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
+/* Map to PMDs logging interface */
+#define DLB2_ERR(dev, fmt, args...) \
+	DLB2_LOG_ERR(fmt, ## args)
+
+#define DLB2_INFO(dev, fmt, args...) \
+	DLB2_LOG_INFO(fmt, ## args)
+
+#define DLB2_DEBUG(dev, fmt, args...) \
+	DLB2_LOG_DBG(fmt, ## args)
+
+/**
+ * os_udelay() - busy-wait for a number of microseconds
+ * @usecs: delay duration.
+ */
+static inline void os_udelay(int usecs)
+{
+	rte_delay_us(usecs);
+}
+
+/**
+ * os_msleep() - sleep for a number of milliseconds
+ * @usecs: delay duration.
+ */
+static inline void os_msleep(int msecs)
+{
+	rte_delay_ms(msecs);
+}
+
+#define DLB2_PP_BASE(__is_ldb) \
+	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
+
+/**
+ * os_map_producer_port() - map a producer port into the caller's address space
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ *
+ * This function maps the requested producer port memory into the caller's
+ * address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void *os_map_producer_port(struct dlb2_hw *hw,
+					 u8 port_id,
+					 bool is_ldb)
+{
+	uint64_t addr;
+	uint64_t pp_dma_base;
+
+	pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
+	addr = (pp_dma_base + (PAGE_SIZE * port_id));
+
+	return (void *)(uintptr_t)addr;
+}
+
+/**
+ * os_unmap_producer_port() - unmap a producer port
+ * @addr: mapped producer port address
+ *
+ * This function undoes os_map_producer_port() by unmapping the producer port
+ * memory from the caller's address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(addr);
+}
+
+/**
+ * os_fence_hcw() - fence an HCW to ensure it arrives at the device
+ * @hw: dlb2_hw handle for a particular device.
+ * @pp_addr: producer port address
+ */
+static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
+{
+	RTE_SET_USED(hw);
+
+	/* To ensure outstanding HCWs reach the device, read the PP address. IA
+	 * memory ordering prevents reads from passing older writes, and the
+	 * mfence also ensures this.
+	 */
+	rte_mb();
+
+	*(volatile u64 *)pp_addr;
+}
+
+/**
+ * DLB2_HW_ERR() - log an error message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_ERR(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_ERR(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/**
+ * DLB2_HW_DBG() - log an info message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_DBG(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_DEBUG(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/* The callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void *dlb2_complete_queue_map_unmap(void *__args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
+	int ret;
+
+	while (1) {
+		rte_spinlock_lock(&dlb2_dev->resource_mutex);
+
+		ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+		ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+		if (ret != 0) {
+			rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+			/* Relinquish the CPU so the application can process
+			 * its CQs, so this function doesn't deadlock.
+			 */
+			sched_yield();
+		} else {
+			break;
+		}
+	}
+
+	dlb2_dev->worker_launched = false;
+
+	rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+
+	return NULL;
+}
+
+
+/**
+ * os_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static inline void os_schedule_work(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+	pthread_t complete_queue_map_unmap_thread;
+	int ret;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
+				     "dlb_queue_unmap_waiter",
+				     NULL,
+				     dlb2_complete_queue_map_unmap,
+				     dlb2_dev);
+	if (ret)
+		DLB2_ERR(dlb2_dev,
+			 "Could not create queue complete map/unmap thread, err=%d\n",
+			 ret);
+	else
+		dlb2_dev->worker_launched = true;
+}
+
+/**
+ * os_worker_active() - query whether the map/unmap worker thread is active
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether a thread (launched by
+ * os_schedule_work()) is active. This function is used to determine
+ * whether or not to launch a worker thread.
+ */
+static inline bool os_worker_active(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	return dlb2_dev->worker_launched;
+}
+
+#endif /*  __DLB2_OSDEP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
new file mode 100644
index 0000000..423233b
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <rte_bitmap.h>
+#include <rte_string_fns.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include "../dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+	struct rte_bitmap *map;
+	unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+				    unsigned int len)
+{
+	struct dlb2_bitmap *bm;
+	void *mem;
+	uint32_t alloc_size;
+	uint32_t nbits = (uint32_t)len;
+
+	if (bitmap == NULL || nbits == 0)
+		return -EINVAL;
+
+	/* Allocate DLB2 bitmap control struct */
+	bm = rte_malloc("DLB2_PF",
+			sizeof(struct dlb2_bitmap),
+			RTE_CACHE_LINE_SIZE);
+
+	if (bm == NULL)
+		return -ENOMEM;
+
+	/* Allocate bitmap memory */
+	alloc_size = rte_bitmap_get_memory_footprint(nbits);
+	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
+	if (mem == NULL) {
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->map = rte_bitmap_init(len, mem, alloc_size);
+	if (bm->map == NULL) {
+		rte_free(mem);
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap == NULL)
+		return;
+
+	rte_free(bitmap->map);
+	rte_free(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++)
+		rte_bitmap_set(bitmap->map, i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	rte_bitmap_reset(bitmap->map);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set() - set a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
+				  unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_set(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+					unsigned int bit,
+					unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_set(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear() - clear a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
+				    unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_clear(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+					  unsigned int bit,
+					  unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_clear(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+						 unsigned int len)
+{
+	unsigned int i, j = 0;
+
+	if (bitmap  == NULL || bitmap->map == NULL || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			if (++j == len)
+				return i - j + 1;
+		} else {
+			j = 0;
+		}
+	}
+
+	/* No set bit range of length len? */
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			return i;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+	int weight = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			weight++;
+	}
+	return weight;
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				      bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+	int max_len = 0, len = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			len++;
+		} else {
+			if (len > max_len)
+				max_len = len;
+			len = 0;
+		}
+	}
+
+	if (len > max_len)
+		max_len = len;
+
+	return max_len;
+}
+
+/**
+ * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
+ * @dest: pointer to dlb2_bitmap structure, which will contain the results of
+ *	  the 'or' of src1 and src2.
+ * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
+ * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
+ *
+ * This function 'or's two bitmaps together and stores the result in a third
+ * bitmap. The source and destination bitmaps can be the same.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - One of the bitmaps is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
+				 struct dlb2_bitmap *src1,
+				 struct dlb2_bitmap *src2)
+{
+	unsigned int i, min;
+	int numset = 0;
+
+	if (dest  == NULL || dest->map == NULL ||
+	    src1  == NULL || src1->map == NULL ||
+	    src2  == NULL || src2->map == NULL)
+		return -EINVAL;
+
+	min = dest->len;
+	min = (min > src1->len) ? src1->len : min;
+	min = (min > src2->len) ? src2->len : min;
+
+	for (i = 0; i != min; i++) {
+		if  (rte_bitmap_get(src1->map, i) ||
+		     rte_bitmap_get(src2->map, i)) {
+			rte_bitmap_set(dest->map, i);
+			numset++;
+		} else {
+			rte_bitmap_clear(dest->map, i);
+		}
+	}
+
+	return numset;
+}
+
+#endif /*  __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
new file mode 100644
index 0000000..5531739
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_LIST_H
+#define __DLB2_OSDEP_LIST_H
+
+#include <rte_tailq.h>
+
+struct dlb2_list_entry {
+	TAILQ_ENTRY(dlb2_list_entry) node;
+};
+
+/* Dummy - just a struct definition */
+TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
+
+/* =================
+ * TAILQ Supplements
+ * =================
+ */
+
+#ifndef TAILQ_FOREACH_ENTRY
+#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)		\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter)						\
+		&& (ptr = container_of(iter, typeof(*(ptr)), name)); \
+	    (iter) = TAILQ_NEXT((iter), node))
+#endif
+
+#ifndef TAILQ_FOREACH_ENTRY_SAFE
+#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)	\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter) &&						\
+		(ptr = container_of(iter, typeof(*(ptr)), name)) &&\
+		((tvar) = TAILQ_NEXT((iter), node), 1);	\
+	    (iter) = (tvar))
+#endif
+
+/***********************/
+/*** List operations ***/
+/***********************/
+
+/**
+ * dlb2_list_init_head() - initialize the head of a list
+ * @head: list head
+ */
+static inline void dlb2_list_init_head(struct dlb2_list_head *head)
+{
+	TAILQ_INIT(head);
+}
+
+/**
+ * dlb2_list_add() - add an entry to a list
+ * @head: list head
+ * @entry: new list entry
+ */
+static inline void
+dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
+{
+	TAILQ_INSERT_TAIL(head, entry, node);
+}
+
+/**
+ * dlb2_list_del() - delete an entry from a list
+ * @entry: list entry
+ * @head: list head
+ */
+static inline void dlb2_list_del(struct dlb2_list_head *head,
+				 struct dlb2_list_entry *entry)
+{
+	TAILQ_REMOVE(head, entry, node);
+}
+
+/**
+ * dlb2_list_empty() - check if a list is empty
+ * @head: list head
+ *
+ * Return:
+ * Returns 1 if empty, 0 if not.
+ */
+static inline int dlb2_list_empty(struct dlb2_list_head *head)
+{
+	return TAILQ_EMPTY(head);
+}
+
+/**
+ * dlb2_list_splice() - splice a list
+ * @src_head: list to be added
+ * @ head: where src_head will be inserted
+ */
+static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
+				    struct dlb2_list_head *head)
+{
+	TAILQ_CONCAT(head, src_head, node);
+}
+
+/**
+ * DLB2_LIST_HEAD() - retrieve the head of the list
+ * @head: list head
+ * @type: type of the list variable
+ * @name: name of the list field within the containing struct
+ */
+#define DLB2_LIST_HEAD(head, type, name)                       \
+	(TAILQ_FIRST(&head) ?					\
+		container_of(TAILQ_FIRST(&head), type, name) :	\
+		NULL)
+
+/**
+ * DLB2_LIST_FOR_EACH() - iterate over a list
+ * @head: list head
+ * @ptr: pointer to struct containing a struct list
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ */
+#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
+	TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
+
+/**
+ * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
+ * an element is removed from the list while processing it.
+ * @ptr: pointer to struct containing a struct list
+ * @ptr_tmp: pointer to struct containing a struct list (temporary)
+ * @head: list head
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ * @iter_tmp: iterator variable (temporary)
+ */
+#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
+	TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
+
+#endif /*  __DLB2_OSDEP_LIST_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
new file mode 100644
index 0000000..0a48f7e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_TYPES_H
+#define __DLB2_OSDEP_TYPES_H
+
+#include <linux/types.h>
+
+#include <inttypes.h>
+#include <ctype.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Types for user mode PF PMD */
+typedef uint8_t         u8;
+typedef int8_t          s8;
+typedef uint16_t        u16;
+typedef int16_t         s16;
+typedef uint32_t        u32;
+typedef int32_t         s32;
+typedef uint64_t        u64;
+
+#define __iomem
+
+/* END types for user mode PF PMD */
+
+#endif /* __DLB2_OSDEP_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
new file mode 100644
index 0000000..43ecad4
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
@@ -0,0 +1,2527 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "dlb2_osdep_types.h"
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+	struct {
+		u32 isr_pend : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+	struct {
+		u32 vf0_reset_in_progress : 1;
+		u32 vf1_reset_in_progress : 1;
+		u32 vf2_reset_in_progress : 1;
+		u32 vf3_reset_in_progress : 1;
+		u32 vf4_reset_in_progress : 1;
+		u32 vf5_reset_in_progress : 1;
+		u32 vf6_reset_in_progress : 1;
+		u32 vf7_reset_in_progress : 1;
+		u32 vf8_reset_in_progress : 1;
+		u32 vf9_reset_in_progress : 1;
+		u32 vf10_reset_in_progress : 1;
+		u32 vf11_reset_in_progress : 1;
+		u32 vf12_reset_in_progress : 1;
+		u32 vf13_reset_in_progress : 1;
+		u32 vf14_reset_in_progress : 1;
+		u32 vf15_reset_in_progress : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+	struct {
+		u32 vec_mask : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+	struct {
+		u32 func_vf_bar_dis : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+	struct {
+		u32 total_vas : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+	struct {
+		u32 total_dir_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+	struct {
+		u32 total_ldb_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+	struct {
+		u32 total_dir_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+	struct {
+		u32 total_ldb_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+	struct {
+		u32 total_dir_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+	struct {
+		u32 total_ldb_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 meas : 1;
+		u32 debug : 7;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 cq_int_rearm : 1;
+		u32 dsi_error : 1;
+		u32 rsvd0 : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 rsvd0 : 3;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+	struct {
+		u32 qid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+	struct {
+		u32 vqid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+	struct {
+		u32 qid : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 debug : 8;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 isz : 1;
+		u32 dsi_error : 1;
+		u32 dlbrsvd : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 vf_synd0_parity : 1;
+		u32 vf_synd1_parity : 1;
+		u32 vf_synd2_parity : 1;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+	struct {
+		u32 sn_cfg_v : 1;
+		u32 fid_cfg_v : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS  0
+#define DLB2_CQ_ISR_MODE_MSI  1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI  3
+union dlb2_sys_ldb_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+	struct {
+		u32 keep_pf_ppid : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+	struct {
+		u32 illegal_hcw : 1;
+		u32 illegal_pp : 1;
+		u32 illegal_pasid : 1;
+		u32 illegal_qid : 1;
+		u32 disabled_qid : 1;
+		u32 illegal_ldb_qid_cfg : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+	struct {
+		u32 msix_0_ack : 1;
+		u32 msix_1_ack : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+	struct {
+		u32 msix_0_passthru : 1;
+		u32 msix_1_passthru : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED     0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+	struct {
+		u32 mode : 1;
+		u32 poll_mode : 1;
+		u32 poll_mask : 1;
+		u32 poll_lock : 1;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+	struct {
+		u32 cq : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 alarm : 1;
+		u32 cwd : 1;
+		u32 vf_pf_mb : 1;
+		u32 rsvd0 : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+	struct {
+		u32 qid_fid_limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+	struct {
+		u32 compress_code : 3;
+		u32 rsvd0 : 29;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+	(DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+	struct {
+		u32 count : 15;
+		u32 rsvd0 : 17;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+	struct {
+		u32 sn : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+	struct {
+		u32 mode : 3;
+		u32 slot : 4;
+		u32 rsvz0 : 1;
+		u32 grp : 1;
+		u32 rsvz1 : 1;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+	struct {
+		u32 en : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+	struct {
+		u32 write_pointer : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+	struct {
+		u32 base : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+	struct {
+		u32 limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+	struct {
+		u32 pop_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+	struct {
+		u32 push_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+	struct {
+		u32 depth : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+	struct {
+		u32 write_pointer : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 dlb_cor_alarm_enable : 1;
+		u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+		u32 cfg_64bytes_qe_dir_cq_mode : 1;
+		u32 pad_write_ldb : 1;
+		u32 pad_write_dir : 1;
+		u32 pad_first_write_ldb : 1;
+		u32 pad_first_write_dir : 1;
+		u32 rsvz0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+	struct {
+		u32 egress_credit_status_empty : 1;
+		u32 egress_credit_status_afull : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+		u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+		u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+		u32 chp_rop_pipe_credit_status_empty : 1;
+		u32 chp_rop_pipe_credit_status_afull : 1;
+		u32 qed_to_cq_pipe_credit_status_empty : 1;
+		u32 qed_to_cq_pipe_credit_status_afull : 1;
+		u32 egress_lsp_token_credit_status_empty : 1;
+		u32 egress_lsp_token_credit_status_afull : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 rsvz0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+	struct {
+		u32 sn_mode_0 : 3;
+		u32 rszv0 : 5;
+		u32 sn_mode_1 : 3;
+		u32 rszv1 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+	struct {
+		u32 unit_single_step_mode : 1;
+		u32 rr_en : 1;
+		u32 rszv0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+	struct {
+		u32 prio : 24;
+		u32 v : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+	struct {
+		u32 qid_p0 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p1 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p2 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p3 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+	struct {
+		u32 qid_p4 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p5 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p6 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p7 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+	struct {
+		u32 token_depth_select : 4;
+		u32 disable_wb_opt : 1;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+	struct {
+		u32 token_count : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+	struct {
+		u32 thresh : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+	(DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+	(DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+	struct {
+		u32 depth : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+	struct {
+		u32 cq : 8;
+		u32 qidix : 3;
+		u32 value : 1;
+		u32 nalb_haswork_v : 1;
+		u32 rlist_haswork_v : 1;
+		u32 slist_haswork_v : 1;
+		u32 inflight_ok_v : 1;
+		u32 aqed_nfull_v : 1;
+		u32 rsvz0 : 15;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+	struct {
+		u32 transfer : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+	struct {
+		u32 bw_range : 9;
+		u32 rsvz0 : 22;
+		u32 no_extra_credit : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+	struct {
+		u32 disab_atq_empty_arb : 1;
+		u32 inc_tok_unit_idle : 1;
+		u32 disab_rlist_pri : 1;
+		u32 inc_cmp_unit_idle : 1;
+		u32 rsvz0 : 2;
+		u32 dir_single_op : 1;
+		u32 dir_half_bw : 1;
+		u32 dir_single_out : 1;
+		u32 dir_disab_multi : 1;
+		u32 atq_single_op : 1;
+		u32 atq_half_bw : 1;
+		u32 atq_single_out : 1;
+		u32 atq_disab_multi : 1;
+		u32 dirrpl_single_op : 1;
+		u32 dirrpl_half_bw : 1;
+		u32 dirrpl_single_out : 1;
+		u32 lbrpl_single_op : 1;
+		u32 lbrpl_half_bw : 1;
+		u32 lbrpl_single_out : 1;
+		u32 ldb_single_op : 1;
+		u32 ldb_half_bw : 1;
+		u32 ldb_disab_multi : 1;
+		u32 atm_single_sch : 1;
+		u32 atm_single_cmp : 1;
+		u32 ldb_ce_tog_arb : 1;
+		u32 rsvz1 : 1;
+		u32 smon0_valid_sel : 2;
+		u32 smon0_value_sel : 1;
+		u32 smon0_compare_sel : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+	struct {
+		u32 chp_pf_reset_done : 1;
+		u32 rop_pf_reset_done : 1;
+		u32 lsp_pf_reset_done : 1;
+		u32 nalb_pf_reset_done : 1;
+		u32 ap_pf_reset_done : 1;
+		u32 dp_pf_reset_done : 1;
+		u32 qed_pf_reset_done : 1;
+		u32 dqed_pf_reset_done : 1;
+		u32 aqed_pf_reset_done : 1;
+		u32 sys_pf_reset_done : 1;
+		u32 pf_reset_active : 1;
+		u32 flrsm_state : 7;
+		u32 rsvd0 : 13;
+		u32 dlb_proc_reset_done : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+	struct {
+		u32 chp_pipeidle : 1;
+		u32 rop_pipeidle : 1;
+		u32 lsp_pipeidle : 1;
+		u32 nalb_pipeidle : 1;
+		u32 ap_pipeidle : 1;
+		u32 dp_pipeidle : 1;
+		u32 qed_pipeidle : 1;
+		u32 dqed_pipeidle : 1;
+		u32 aqed_pipeidle : 1;
+		u32 sys_pipeidle : 1;
+		u32 chp_unit_idle : 1;
+		u32 rop_unit_idle : 1;
+		u32 lsp_unit_idle : 1;
+		u32 nalb_unit_idle : 1;
+		u32 ap_unit_idle : 1;
+		u32 dp_unit_idle : 1;
+		u32 qed_unit_idle : 1;
+		u32 dqed_unit_idle : 1;
+		u32 aqed_unit_idle : 1;
+		u32 sys_unit_idle : 1;
+		u32 rsvd1 : 4;
+		u32 mstr_cfg_ring_idle : 1;
+		u32 mstr_cfg_mstr_idle : 1;
+		u32 mstr_flr_clkreq_b : 1;
+		u32 mstr_proc_idle : 1;
+		u32 mstr_proc_idle_masked : 1;
+		u32 rsvd0 : 2;
+		u32 dlb_func_idle : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+	struct {
+		u32 prochot : 1;
+		u32 pgcb_dlb_idle : 1;
+		u32 pgcb_dlb_pg_rdy_ack_b : 1;
+		u32 pmsm_pgcb_req_b : 1;
+		u32 pgbc_pmc_pg_req_b : 1;
+		u32 pmc_pgcb_pg_ack_b : 1;
+		u32 pmc_pgcb_fet_en_b : 1;
+		u32 pgcb_fet_en_b : 1;
+		u32 rsvz0 : 1;
+		u32 rsvz1 : 1;
+		u32 fuse_force_on : 1;
+		u32 fuse_proc_disable : 1;
+		u32 rsvz2 : 1;
+		u32 rsvz3 : 1;
+		u32 pm_fsm_d0tod3_ok : 1;
+		u32 pm_fsm_d3tod0_ok : 1;
+		u32 dlb_in_d3 : 1;
+		u32 rsvz4 : 7;
+		u32 pmsm : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+	struct {
+		u32 disable : 1;
+		u32 rsvz0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+	struct {
+		u32 isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+	struct {
+		u32 pf_isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+	struct {
+		u32 isr_pend : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+	struct {
+		u32 reset_in_progress : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+	struct {
+		u32 vf_msi_isr : 32;
+	} field;
+	u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
new file mode 100644
index 0000000..503fdf3
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
@@ -0,0 +1,1913 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_osdep_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_start_domain() - start a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: start domain arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int dlb2_hw_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp,
+			 bool vdev_request,
+			 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_request,
+		    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding unmap procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_finish_map_qid_procedures() - finish any pending map procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding map procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ *					interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+			  int port_id,
+			  bool is_ldb,
+			  bool vdev_request,
+			  unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+					 u32 *ldb_interrupts,
+					 u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+				 u32 *ldb_interrupts,
+				 u32 *dir_interrupts);
+
+/**
+ * dlb2_read_vf_intr_status() - read the VF interrupt status register
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function can be called from a VF's interrupt handler to determine
+ * which interrupts have fired. The first 31 bits correspond to CQ interrupt
+ * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
+ *
+ * Return:
+ * Returns a bit vector indicating which interrupt vectors are active.
+ */
+u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_intr_status() - ack VF interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function ACKs a VF's interrupts. Its interrupts argument should be the
+ * value returned by dlb2_read_vf_intr_status().
+ */
+void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function clears the VF's MSI interrupt pending register. Its interrupts
+ * argument should be contain the MSI vectors to ACK. For example, if MSI MME
+ * is in mode 0, then one bit 0 should ever be set.
+ */
+void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When done processing the PF mailbox request, this function unsets
+ * the PF's mailbox ISR register.
+ */
+void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
+
+/**
+ * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
+ *					vdevs
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the vdev->PF ISR fires, this function can be called to determine which
+ * vdev(s) are requesting service. This bitvector must be passed to
+ * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
+ * vdevs.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested service.
+ */
+u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ *
+ * When done processing all VF mailbox requests, this function unsets the VF's
+ * mailbox ISR register.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
+ *				    FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the VF FLR ISR fires, this function can be called to determine which
+ * VF(s) are requesting FLRs. This bitvector must passed to
+ * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
+ */
+u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF FLR requests, this function unsets the VF's FLR
+ * ISR register.
+ */
+void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF requests, this function communicates to the
+ * hardware that processing is complete.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
+			     u32 mbox_bitvec,
+			     u32 flr_bitvec);
+
+/**
+ * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the watchdog timer interrupt cause registers to
+ * determine which port(s) had a watchdog timeout, and notifies the
+ * application(s) that own the port(s).
+ */
+void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads and logs the alarm syndrome, then acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_request,
+			      unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of resources in use by the PF or a VF. It
+ * fills in the fields that args points to, except the following:
+ * - max_contiguous_atomic_inflights
+ * - max_contiguous_hist_list_entries
+ * - max_contiguous_ldb_credits
+ * - max_contiguous_dir_credits
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
+				   struct dlb2_get_num_resources_args *arg,
+				   bool vdev_request,
+				   unsigned int vdev_id);
+
+/**
+ * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
+ *				       the PF
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function sends a VF->PF mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the PF has processed
+ * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
+ * when the PF has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
+ *				 request
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * processing a VF->PF mailbox request. It should only be called after sending
+ * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vf_flr_complete() - check the status of a VF FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * executing the VF FLR. It should only be called after setting the VF's FLR
+ * bit.
+ */
+bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
+ *					vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function sends a PF->vdev mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the vdev has
+ * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
+ * determine when the vdev has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
+ *			       request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function returns a boolean indicating whether the vdev has finished
+ * processing a PF->vdev mailbox request. It should only be called after
+ * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
+			     unsigned int vf_id,
+			     void *data,
+			     int len);
+
+/**
+ * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
+			       unsigned int vf_id,
+			       void *data,
+			       int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_reset_vdev() - reset the hardware owned by a virtual device
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function resets the hardware owned by a vdev, by resetting the vdev's
+ * domains one by one.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function returns whether or not the vdev's resource assignments are
+ * locked. If locked, no resources can be added to or subtracted from the
+ * group.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_lock_vdev() - lock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sets a flag indicating that the vdev is using its resources.
+ * When the vdev is locked, its resource assignment cannot be changed.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_unlock_vdev() - unlock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function unlocks the vdev's resource assignment, allowing it to be
+ * modified.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of scheduling domains to assign to this vdev
+ *
+ * This function assigns num scheduling domains to the specified vdev. If the
+ * vdev already has domains assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB queues to assign to this vdev
+ *
+ * This function assigns num LDB queues to the specified vdev. If the vdev
+ * already has LDB queues assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports to the specified vdev. If the vdev
+ * already has LDB ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @cos: class-of-service ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports from class-of-service cos to the
+ * specified vdev. If the vdev already has LDB ports from this class-of-service
+ * assigned, this existing assignment is adjusted accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
+				   u32 id,
+				   u32 cos,
+				   u32 num);
+
+/**
+ * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR ports to assign to this vdev
+ *
+ * This function assigns num DIR ports to the specified vdev. If the vdev
+ * already has DIR ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB credit credits to assign to this vdev
+ *
+ * This function assigns num LDB credit to the specified vdev. If the vdev
+ * already has LDB credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR credits to assign to this vdev
+ *
+ * This function assigns num DIR credit to the specified vdev. If the vdev
+ * already has DIR credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of history list entries to assign to this vdev
+ *
+ * This function assigns num history list entries to the specified vdev. If the
+ * vdev already has history list entries assigned, this existing assignment is
+ * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
+ * this function may fail if a sufficiently large contiguous chunk is not
+ * available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of atomic inflights to assign to this vdev
+ *
+ * This function assigns num atomic inflights to the specified vdev. If the vdev
+ * already has atomic inflights assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of entries, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function takes any resources currently assigned to the vdev and
+ * reassigns them to the PF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_notify_vf() - send an alarm to a VF
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID
+ * @notification: notification
+ *
+ * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
+ *
+ * Return:
+ * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
+ */
+int dlb2_notify_vf(struct dlb2_hw *hw,
+		   unsigned int vf_id,
+		   u32 notification);
+
+/**
+ * dlb2_vdev_in_use() - query whether a virtual device is in use
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sends a mailbox request to the vdev to query whether the vdev
+ * is in use.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
+ * an internal error occurs.
+ */
+int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+enum dlb2_virt_mode {
+	DLB2_VIRT_NONE,
+	DLB2_VIRT_SRIOV,
+	DLB2_VIRT_SIOV,
+
+	/* NUM_DLB2_VIRT_MODES must be last */
+	NUM_DLB2_VIRT_MODES,
+};
+
+/**
+ * dlb2_hw_set_virt_mode() - set the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: either none, SR-IOV, or Scalable IOV.
+ *
+ * This function sets the virtualization mode of the device. This controls
+ * whether the device uses a software or hardware mailbox.
+ *
+ * This should be called by the PF driver when either SR-IOV or Scalable IOV is
+ * selected as the virtualization mechanism, and by the VF/VDEV driver during
+ * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
+ *
+ * Errors:
+ * EINVAL - Invalid mode.
+ */
+int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
+
+/**
+ * dlb2_hw_get_virt_mode() - get the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function gets the virtualization mode of the device.
+ */
+enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_register_sw_mbox() - register a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
+ * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
+ * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
+ * @inject_arg: user argument for pf2vdev_inject callback.
+ *
+ * When Scalable IOV is enabled, the VDCM must register a software mailbox for
+ * every virtual device during vdev creation.
+ *
+ * This function notifies the driver to use a software mailbox using the
+ * provided pointers, instead of the device's hardware mailbox. When the driver
+ * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
+ * go to the software mailbox instead of the hardware one. This is used in
+ * Scalable IOV virtualization.
+ */
+void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
+			      unsigned int vdev_id,
+			      u32 *vdev2pf_mbox,
+			      u32 *pf2vdev_mbox,
+			      void (*pf2vdev_inject)(void *),
+			      void *inject_arg);
+
+/**
+ * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function notifies the driver to stop using a previously registered
+ * software mailbox.
+ */
+void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ * @addr_lo: least-significant 32 bits of address.
+ * @data: 32 data bits.
+ *
+ * This sets up the CQ's IMS entry with the provided address and data values.
+ * This function should only be called if the device is configured for Scalable
+ * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
+ * not needed.
+ */
+void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb,
+				u32 addr_lo,
+				u32 data);
+
+/**
+ * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ *
+ * This clears the CQ's IMS entry, reverting it to its reset state.
+ */
+void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb);
+
+/**
+ * dlb2_hw_register_pasid() - register a vdev's PASID
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @pasid: the vdev's PASID.
+ *
+ * This function stores the user-supplied PASID, and uses it when configuring
+ * the vdev's CQs.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+int dlb2_hw_register_pasid(struct dlb2_hw *hw,
+			   unsigned int vdev_id,
+			   unsigned int pasid);
+
+/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vf_request,
+				unsigned int vf_id);
+
+/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ *	to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ *	port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ *	    cause the total bandwidth across all classes of service to exceed
+ *	    100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+enum dlb2_wd_tmo {
+	/* 40s watchdog timeout */
+	DLB2_WD_TMO_40S,
+	/* 10s watchdog timeout */
+	DLB2_WD_TMO_10S,
+	/* 1s watchdog timeout */
+	DLB2_WD_TMO_1S,
+
+	/* Must be last */
+	NUM_DLB2_WD_TMOS,
+};
+
+/**
+ * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
+ *	caller-specified timeout.
+ * @hw: dlb2_hw handle for a particular device.
+ * @tmo: watchdog timeout.
+ *
+ * This function should be called during device initialization and after reset.
+ * The watchdog timer interrupt must also be enabled per-CQ, using either
+ * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid timeout.
+ */
+int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
+
+/**
+ * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid directed port ID.
+ */
+int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid load-balanced port ID.
+ */
+int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *	ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * load-balanced CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_ldb_cq_interrupt().
+ */
+void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical directed port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * directed CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_dir_cq_interrupt().
+ */
+void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+#endif /* __DLB2_RESOURCE_H */
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
new file mode 100644
index 0000000..3c51cab
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -0,0 +1,621 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "base/dlb2_resource.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_regs.h"
+#include "dlb2_main.h"
+#include "../dlb2_user.h"
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+
+#define PF_ID_ZERO 0	/* PF ONLY! */
+#define NO_OWNER_VF 0	/* PF ONLY! */
+#define NOT_VF_REQ false /* PF ONLY! */
+
+#define DLB2_PCI_CFG_SPACE_SIZE 256
+#define DLB2_PCI_CAP_POINTER 0x34
+#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
+#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
+#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
+#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
+#define DLB2_PCI_EXT_CAP_ID_ERR 1
+#define DLB2_PCI_ERR_UNCOR_MASK 8
+#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
+
+#define DLB2_PCI_EXP_DEVCTL 8
+#define DLB2_PCI_LNKCTL 16
+#define DLB2_PCI_SLTCTL 24
+#define DLB2_PCI_RTCTL 28
+#define DLB2_PCI_EXP_DEVCTL2 40
+#define DLB2_PCI_LNKCTL2 48
+#define DLB2_PCI_SLTCTL2 56
+#define DLB2_PCI_CMD 4
+#define DLB2_PCI_X_CMD 2
+#define DLB2_PCI_EXP_DEVSTA 10
+#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
+#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
+
+#define DLB2_PCI_CAP_ID_EXP       0x10
+#define DLB2_PCI_CAP_ID_MSIX      0x11
+#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
+#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
+#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
+
+#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
+#define DLB2_PCI_PRI_ALLOC_REQ           0xC
+#define DLB2_PCI_PRI_CTRL                0x4
+#define DLB2_PCI_MSIX_FLAGS              0x2
+#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
+#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
+#define DLB2_PCI_ERR_ROOT_STATUS         0x30
+#define DLB2_PCI_ERR_COR_STATUS          0x10
+#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
+#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
+#define DLB2_PCI_ACS_CAP                 0x4
+#define DLB2_PCI_ACS_CTRL                0x6
+#define DLB2_PCI_ACS_SV                  0x1
+#define DLB2_PCI_ACS_RR                  0x4
+#define DLB2_PCI_ACS_CR                  0x8
+#define DLB2_PCI_ACS_UF                  0x10
+#define DLB2_PCI_ACS_EC                  0x20
+
+/* Stubs: Allow building partial probe patch */
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	int ret = 0;
+	RTE_SET_USED(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+/* End stubs */
+
+static int
+dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint32_t hdr;
+	size_t sz;
+	int pos;
+
+	pos = DLB2_PCI_CFG_SPACE_SIZE;
+	sz = sizeof(hdr);
+
+	while (pos > 0xFF) {
+		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
+			return -1;
+
+		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
+			return pos;
+
+		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint8_t pos;
+	int ret;
+	uint16_t hdr;
+
+	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
+	pos &= 0xFC;
+
+	if (ret != 1)
+		return -1;
+
+	while (pos > 0x3F) {
+		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
+		if (ret != 2)
+			return -1;
+
+		if (DLB2_PCI_CAP_ID(hdr) == id)
+			return pos;
+
+		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
+			return -1;
+
+		pos = DLB2_PCI_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+	rte_spinlock_init(&dlb2_dev->resource_mutex);
+	rte_spinlock_init(&dlb2_dev->measurement_lock);
+
+	return 0;
+}
+
+static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
+{
+	u32 retries = 0;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
+		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+		union dlb2_cfg_mstr_cfg_pm_status pm_st;
+		u32 addr;
+
+		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+			break;
+
+		rte_delay_ms(1);
+	};
+
+	if (retries == DLB2_READY_RETRY_LIMIT) {
+		printf("[%s()] wait for device ready timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+struct dlb2_dev *
+dlb2_probe(struct rte_pci_device *pdev)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret = 0;
+
+	DLB2_INFO(dlb2_dev, "probe\n");
+
+	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
+			      RTE_CACHE_LINE_SIZE);
+
+	if (dlb2_dev == NULL) {
+		ret = -ENOMEM;
+		goto dlb2_dev_malloc_fail;
+	}
+
+	/* PCI Bus driver has already mapped bar space into process.
+	 * Save off our IO register and FUNC addresses.
+	 */
+
+	/* BAR 0 */
+	if (pdev->mem_resource[0].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
+	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.func_kva,
+		  (void *)dlb2_dev->hw.func_phys_addr,
+		  (void *)(pdev->mem_resource[0].len));
+
+	/* BAR 2 */
+	if (pdev->mem_resource[2].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
+	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.csr_kva,
+		  (void *)dlb2_dev->hw.csr_phys_addr,
+		  (void *)(pdev->mem_resource[2].len));
+
+	dlb2_dev->pdev = pdev;
+
+	/* PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb2_pf_enable_pm(dlb2_dev);
+
+	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb2_pf_reset(dlb2_dev);
+	if (ret)
+		goto dlb2_reset_fail;
+
+	ret = dlb2_pf_init_driver_state(dlb2_dev);
+	if (ret)
+		goto init_driver_state_fail;
+
+	ret = dlb2_resource_init(&dlb2_dev->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	return dlb2_dev;
+
+resource_init_fail:
+	dlb2_resource_free(&dlb2_dev->hw);
+init_driver_state_fail:
+dlb2_reset_fail:
+pci_mmap_bad_addr:
+wait_for_device_ready_fail:
+	rte_free(dlb2_dev);
+dlb2_dev_malloc_fail:
+	rte_errno = ret;
+	return NULL;
+}
+
+int
+dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
+{
+	int ret = 0;
+	int i = 0;
+	uint32_t dword[16];
+	uint16_t cmd;
+	off_t off;
+
+	uint16_t dev_ctl_word;
+	uint16_t dev_ctl2_word;
+	uint16_t lnk_word;
+	uint16_t lnk_word2;
+	uint16_t slt_word;
+	uint16_t slt_word2;
+	uint16_t rt_ctl_word;
+	uint32_t pri_reqs_dword;
+	uint16_t pri_ctrl_word;
+
+	int pcie_cap_offset;
+	int pri_cap_offset;
+	int msix_cap_offset;
+	int err_cap_offset;
+	int acs_cap_offset;
+	int wait_count;
+
+	uint16_t devsta_busy_word;
+	uint16_t devctl_word;
+
+	struct rte_pci_device *pdev = dlb2_dev->pdev;
+
+	/* Save PCI config state */
+
+	for (i = 0; i < 16; i++) {
+		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
+			return ret;
+	}
+
+	pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
+
+	if (pcie_cap_offset < 0) {
+		printf("[%s()] failed to find the pcie capability\n",
+		       __func__);
+		return pcie_cap_offset;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
+		dev_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
+		lnk_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
+		slt_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_RTCTL;
+	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
+		rt_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
+		dev_ctl2_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
+		lnk_word2 = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
+		slt_word2 = 0;
+
+	off = DLB2_PCI_EXT_CAP_ID_PRI;
+	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (pri_cap_offset >= 0) {
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
+			pri_reqs_dword = 0;
+	}
+
+	/* clear the PCI command register before issuing the FLR */
+
+	off = DLB2_PCI_CMD;
+	cmd = 0;
+	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+		printf("[%s()] failed to write the pci command\n",
+		       __func__);
+		return ret;
+	}
+
+	/* issue the FLR */
+	for (wait_count = 0; wait_count < 4; wait_count++) {
+		int sleep_time;
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
+		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to read the pci device status\n",
+			       __func__);
+			return ret;
+		}
+
+		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
+			break;
+
+		sleep_time = (1 << (wait_count)) * 100;
+		rte_delay_ms(sleep_time);
+	}
+
+	if (wait_count == 4) {
+		printf("[%s()] wait for pci pending transactions timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to read the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
+
+	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to write the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	rte_delay_ms(100);
+
+	/* Restore PCI config state */
+
+	if (pcie_cap_offset >= 0) {
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie device control at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_RTCTL;
+		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	if (pri_cap_offset >= 0) {
+		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
+
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
+		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ERR;
+	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (err_cap_offset >= 0) {
+		uint32_t tmp;
+
+		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	for (i = 16; i > 0; i--) {
+		off = (i - 1) * 4;
+		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_CMD;
+	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
+		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+			printf("[%s()] failed to write the pci command\n",
+			       __func__);
+			return ret;
+		}
+	}
+
+	msix_cap_offset = dlb2_pci_find_capability(pdev,
+						   DLB2_PCI_CAP_ID_MSIX);
+	if (msix_cap_offset >= 0) {
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
+			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ACS;
+	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (acs_cap_offset >= 0) {
+		uint16_t acs_cap, acs_ctrl, acs_mask;
+		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
+		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
+			acs_cap = 0;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
+		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
+		acs_ctrl |= (acs_cap & acs_mask);
+
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
+		acs_mask |= DLB2_PCI_ACS_EC;
+		acs_ctrl &= ~acs_mask;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
new file mode 100644
index 0000000..f082a94
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -0,0 +1,98 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+#endif
+
+#include "base/dlb2_hw_types.h"
+#include "../dlb2_user.h"
+
+#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
+
+struct dlb2_dev;
+
+struct dlb2_port_memory {
+	struct dlb2_list_head list;
+	void *cq_base;
+	bool valid;
+};
+
+struct dlb2_dev {
+	struct rte_pci_device *pdev;
+	struct dlb2_hw hw;
+	/* struct list_head list; */
+	struct device *dlb2_device;
+	bool domain_reset_failed;
+	/* The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	rte_spinlock_t resource_mutex;
+	rte_spinlock_t measurement_lock;
+	bool worker_launched;
+	u8 revision;
+};
+
+struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
+int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
+int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+
+#endif /* __DLB2_MAIN_H */
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
new file mode 100644
index 0000000..ae4d785
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -0,0 +1,196 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_devargs.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_pci.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+#include "dlb2_main.h"
+#include "base/dlb2_hw_types.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_resource.h"
+
+static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
+
+/* Stubs: Allow building partial probe patch */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(arg);
+	RTE_SET_USED(vdev_req);
+	RTE_SET_USED(vdev_id);
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+/* End stubs */
+
+static void
+dlb2_pf_iface_fn_ptrs_init(void)
+{
+/* flexible iface fn ptr assignments will go here */
+}
+
+/* PCI DEV HOOKS */
+static int
+dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
+{
+	int ret = 0;
+	struct rte_pci_device *pci_dev;
+	struct dlb2_devargs dlb2_args = {
+		.socket_id = rte_socket_id(),
+		.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+		.num_dir_credits_override = -1,
+		.qid_depth_thresholds = { {0} },
+		.cos_id = DLB2_COS_DEFAULT
+	};
+	struct dlb2_eventdev *dlb2;
+
+	DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
+		     eventdev->data->dev_id, eventdev->data->socket_id);
+
+	dlb2_pf_iface_fn_ptrs_init();
+
+	pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
+
+		/* Probe the DLB2 PF layer */
+		dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
+
+		if (dlb2->qm_instance.pf_dev == NULL) {
+			DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
+				     rte_errno);
+			ret = -rte_errno;
+			goto dlb2_probe_failed;
+		}
+
+		/* Were we invoked with runtime parameters? */
+		if (pci_dev->device.devargs) {
+			ret = dlb2_parse_params(pci_dev->device.devargs->args,
+						pci_dev->device.devargs->name,
+						&dlb2_args);
+			if (ret) {
+				DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
+					     ret, rte_errno);
+				goto dlb2_probe_failed;
+			}
+		}
+
+		ret = dlb2_primary_eventdev_probe(eventdev,
+						  event_dlb2_pf_name,
+						  &dlb2_args);
+	} else {
+		ret = dlb2_secondary_eventdev_probe(eventdev,
+						    event_dlb2_pf_name);
+	}
+	if (ret)
+		goto dlb2_probe_failed;
+
+	DLB2_LOG_INFO("DLB2 PF Probe success\n");
+
+	return 0;
+
+dlb2_probe_failed:
+
+	DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
+
+	return ret;
+}
+
+#define EVENTDEV_INTEL_VENDOR_ID 0x8086
+
+static const struct rte_pci_id pci_id_dlb2_map[] = {
+	{
+		RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
+			       PCI_DEVICE_ID_INTEL_DLB2_PF)
+	},
+	{
+		.vendor_id = 0,
+	},
+};
+
+static int
+event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
+		     struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
+					     sizeof(struct dlb2_eventdev),
+					     dlb2_eventdev_pci_init,
+					     event_dlb2_pf_name);
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+}
+
+static int
+event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_remove(pci_dev, NULL);
+
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+
+}
+
+static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
+	.id_table = pci_id_dlb2_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe = event_dlb2_pci_probe,
+	.remove = event_dlb2_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
-- 
2.6.4

[dpdk-dev] [PATCH v3 07/23] event/dlb2: add flexible interface

[Timothy McDaniel]

This commit introduces the flexible interface. This
interface allows the core code to operate in PF mode (direct
hardware access) or bifurcated mode (hardware configured via
kernel driver). This driver currently only supports PF mode
but bifurcated mode will be added in a future DPDK patch-set.
Note that the flexible interface is not used for data path
operations, and thus there are no performance concerns
related to the use of function pointers.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_iface.c | 28 ++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.h | 29 +++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build  |  1 +
 3 files changed, 58 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h

diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
new file mode 100644
index 0000000..0fc9991
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+
+#include "dlb2_priv.h"
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+void (*dlb2_iface_low_level_io_init)(void);
+
+int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+				     uint8_t *revision);
+
+void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+				   enum dlb2_cq_poll_modes *mode);
+
+int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
new file mode 100644
index 0000000..4fb416e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_IFACE_H_
+#define _DLB2_IFACE_H_
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+extern void (*dlb2_iface_low_level_io_init)(void);
+
+extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+					    uint8_t *revision);
+
+extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+					  enum dlb2_cq_poll_modes *mode);
+
+extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
+
+#endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 1eab9b9..491e76d 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -8,6 +8,7 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
 endif
 
 sources = files('dlb2.c',
+		'dlb2_iface.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c'
 )
-- 
2.6.4

[dpdk-dev] [PATCH v3 08/23] event/dlb2: add probe-time hardware init

[Timothy McDaniel]

This commit adds probe-time low level hardware
initialization.  It also adds probe-time init for both
primary and secondary DPDK processes.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 188 +++++++++++++++++++-
 drivers/event/dlb2/meson.build             |   3 +-
 drivers/event/dlb2/pf/base/dlb2_resource.c | 274 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  23 +--
 drivers/event/dlb2/pf/dlb2_main.h          |   1 -
 drivers/event/dlb2/pf/dlb2_pf.c            |  78 ++++++--
 6 files changed, 523 insertions(+), 44 deletions(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 1b11979..16653db 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -31,6 +31,7 @@
 #include <rte_string_fns.h>
 
 #include "dlb2_priv.h"
+#include "dlb2_iface.h"
 #include "dlb2_inline_fns.h"
 
 /*
@@ -40,10 +41,108 @@
 #if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
 #error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
 #endif
+static struct rte_event_dev_info evdev_dlb2_default_info = {
+	.driver_name = "", /* probe will set */
+	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
+	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
+	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
+#else
+	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
+#endif
+	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
+	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
+	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
+	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
+	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
+	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+	.max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
+	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
+			  RTE_EVENT_DEV_CAP_EVENT_QOS |
+			  RTE_EVENT_DEV_CAP_BURST_MODE |
+			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
+			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
+			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
+};
 
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/* override defaults with value(s) provided on command line */
+static void
+dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
+				 int *qid_depth_thresholds)
+{
+	int q;
+
+	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
+		if (qid_depth_thresholds[q] != 0)
+			dlb2->ev_queues[q].depth_threshold =
+				qid_depth_thresholds[q];
+	}
+}
+
+static int
+dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
+	int ret;
+
+	/* Query driver resources provisioned for this device */
+
+	ret = dlb2_iface_get_num_resources(handle,
+					   &dlb2->hw_rsrc_query_results);
+	if (ret) {
+		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
+		return ret;
+	}
+
+	/* Complete filling in device resource info returned to evdev app,
+	 * overriding any default values.
+	 * The capabilities (CAPs) were set at compile time.
+	 */
+
+	evdev_dlb2_default_info.max_event_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	evdev_dlb2_default_info.max_event_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	evdev_dlb2_default_info.max_num_events =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	/* Save off values used when creating the scheduling domain. */
+
+	handle->info.num_sched_domains =
+		dlb2->hw_rsrc_query_results.num_sched_domains;
+
+	handle->info.hw_rsrc_max.nb_events_limit =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	handle->info.hw_rsrc_max.num_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues +
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.num_ldb_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	handle->info.hw_rsrc_max.num_ldb_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	handle->info.hw_rsrc_max.num_dir_ports =
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.reorder_window_size =
+		dlb2->hw_rsrc_query_results.num_hist_list_entries;
+
+	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
+
+	return 0;
+}
+
 #define DLB2_BASE_10 10
 
 static int
@@ -235,14 +334,71 @@ set_qid_depth_thresh(const char *key __rte_unused,
 	return 0;
 }
 
+static void
+dlb2_entry_points_init(struct rte_eventdev *dev)
+{
+	RTE_SET_USED(dev);
+
+	/* Eventdev PMD entry points */
+}
+
 int
 dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    const char *name,
 			    struct dlb2_devargs *dlb2_args)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
-	RTE_SET_USED(dlb2_args);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	dlb2->event_dev = dev; /* backlink */
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	dlb2->max_num_events_override = dlb2_args->max_num_events;
+	dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
+	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
+					    &dlb2->revision);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_hardware_init(&dlb2->qm_instance);
+
+	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
+			     err);
+		return err;
+	}
+
+	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	dlb2_init_queue_depth_thresholds(dlb2,
+					 dlb2_args->qid_depth_thresholds.val);
 
 	return 0;
 }
@@ -251,8 +407,30 @@ int
 dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
 			      const char *name)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 491e76d..28fef3f 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -10,7 +10,8 @@ endif
 sources = files('dlb2.c',
 		'dlb2_iface.c',
 		'pf/dlb2_main.c',
-		'pf/dlb2_pf.c'
+		'pf/dlb2_pf.c',
+		'pf/base/dlb2_resource.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
new file mode 100644
index 0000000..6de8b95
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_mbox.h"
+#include "dlb2_osdep.h"
+#include "dlb2_osdep_bitmap.h"
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+	int i;
+
+	dlb2_list_init_head(&domain->used_ldb_queues);
+	dlb2_list_init_head(&domain->used_dir_pq_pairs);
+	dlb2_list_init_head(&domain->avail_ldb_queues);
+	dlb2_list_init_head(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+	int i;
+
+	dlb2_list_init_head(&rsrc->avail_domains);
+	dlb2_list_init_head(&rsrc->used_domains);
+	dlb2_list_init_head(&rsrc->avail_ldb_queues);
+	dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
+	arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
+	arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
+	arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb2_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	struct dlb2_list_entry *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. This is application
+	 * dependent, but the driver interleaves port IDs as much as possible
+	 * to reduce the likelihood of this. This initial allocation maximizes
+	 * the average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	/* Zero-out resource tracking data structures */
+	memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+	memset(&hw->pf, 0, sizeof(hw->pf));
+
+	dlb2_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+		dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+		dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_domains, list);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_ldb_queues, list);
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+		struct dlb2_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
+			      &port->func_list);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+				DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+	if (ret)
+		goto unwind;
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+					DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+		if (ret)
+			goto unwind;
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb2_resource_free(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+	r0.field.disable = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 3c51cab..210b3186 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -19,6 +19,7 @@
 #include "dlb2_main.h"
 #include "../dlb2_user.h"
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 
 #define PF_ID_ZERO 0	/* PF ONLY! */
@@ -72,27 +73,6 @@
 #define DLB2_PCI_ACS_UF                  0x10
 #define DLB2_PCI_ACS_EC                  0x20
 
-/* Stubs: Allow building partial probe patch */
-void dlb2_resource_free(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-int dlb2_resource_init(struct dlb2_hw *hw)
-{
-	int ret = 0;
-	RTE_SET_USED(hw);
-
-	return ret;
-}
-
-void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-/* End stubs */
-
 static int
 dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
 {
@@ -149,7 +129,6 @@ static int
 dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
 {
 	rte_spinlock_init(&dlb2_dev->resource_mutex);
-	rte_spinlock_init(&dlb2_dev->measurement_lock);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index f082a94..f3bee71 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -38,7 +38,6 @@ struct dlb2_dev {
 	 * hardware registers.
 	 */
 	rte_spinlock_t resource_mutex;
-	rte_spinlock_t measurement_lock;
 	bool worker_launched;
 	u8 revision;
 };
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index ae4d785..7d64309 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -32,6 +32,7 @@
 #include <rte_string_fns.h>
 
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 #include "dlb2_main.h"
 #include "base/dlb2_hw_types.h"
@@ -40,35 +41,82 @@
 
 static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
 
-/* Stubs: Allow building partial probe patch */
-int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
-			      struct dlb2_get_num_resources_args *arg,
-			      bool vdev_req,
-			      unsigned int vdev_id)
+static void
+dlb2_pf_low_level_io_init(void)
+{
+	int i;
+	/* Addresses will be initialized at port create */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		/* First directed ports */
+		dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
+
+		/* Now load balanced ports */
+		dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
+	}
+}
+
+static int
+dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
+{
+	RTE_SET_USED(handle);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+static int
+dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
+			   uint8_t *revision)
 {
-	RTE_SET_USED(hw);
-	RTE_SET_USED(arg);
-	RTE_SET_USED(vdev_req);
-	RTE_SET_USED(vdev_id);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	*revision = dlb2_dev->revision;
 
 	return 0;
 }
 
-void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+static void
+dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+	dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_num_resources_args *rsrcs)
 {
-	RTE_SET_USED(hw);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
 }
 
-void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+static int
+dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
+			 enum dlb2_cq_poll_modes *mode)
 {
-	RTE_SET_USED(hw);
+	RTE_SET_USED(handle);
+
+	*mode = DLB2_CQ_POLL_MODE_SPARSE;
+
+	return 0;
 }
-/* End stubs */
 
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
-/* flexible iface fn ptr assignments will go here */
+	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
+	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
+	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
+	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
+	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v3 09/23] event/dlb2: add xstats

[Timothy McDaniel]

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
Reviewed-by: Chen, Mike Ximing <mike.ximing.chen@intel.com>
---
 drivers/event/dlb2/dlb2.c        |   35 +-
 drivers/event/dlb2/dlb2_xstats.c | 1240 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build   |    1 +
 3 files changed, 1273 insertions(+), 3 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 16653db..925824e 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,6 +70,21 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/*
+ * DUMMY - added so that xstats path will compile/link.
+ * Will be replaced by real version in a subsequent
+ * patch.
+ */
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	RTE_SET_USED(dlb2);
+	RTE_SET_USED(queue);
+
+	return 0;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -337,9 +352,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
-	RTE_SET_USED(dev);
-
-	/* Eventdev PMD entry points */
+	/* Expose PMD's eventdev interface */
+	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dump             = dlb2_eventdev_dump,
+		.xstats_get       = dlb2_eventdev_xstats_get,
+		.xstats_get_names = dlb2_eventdev_xstats_get_names,
+		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
+		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+	};
+
+	dev->dev_ops = &dlb2_eventdev_entry_ops;
 }
 
 int
@@ -391,6 +413,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Complete xtstats runtime initialization */
+	err = dlb2_xstats_init(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
+		return err;
+	}
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
new file mode 100644
index 0000000..21391f7
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_xstats.c
@@ -0,0 +1,1240 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eventdev.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+enum dlb2_xstats_type {
+	/* common to device and port */
+	rx_ok,				/**< Receive an event */
+	rx_drop,                        /**< Error bit set in received QE */
+	rx_interrupt_wait,		/**< Wait on an interrupt */
+	rx_umonitor_umwait,		/**< Block using umwait */
+	tx_ok,				/**< Transmit an event */
+	total_polls,			/**< Call dequeue_burst */
+	zero_polls,			/**< Call dequeue burst and return 0 */
+	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
+	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
+	tx_nospc_inflight_max,		/**< Reach the new_event_threshold */
+	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
+	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
+	/* device specific */
+	nb_events_limit,
+	inflight_events,
+	ldb_pool_size,
+	dir_pool_size,
+	/* port specific */
+	tx_new,				/**< Send an OP_NEW event */
+	tx_fwd,				/**< Send an OP_FORWARD event */
+	tx_rel,				/**< Send an OP_RELEASE event */
+	tx_implicit_rel,		/**< Issue an implicit event release */
+	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED event */
+	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL event */
+	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event */
+	tx_sched_directed,		/**< Send a directed event */
+	tx_invalid,                     /**< Send an event with an invalid op */
+	outstanding_releases,		/**< # of releases a port owes */
+	max_outstanding_releases,	/**< max # of releases a port can owe */
+	rx_sched_ordered,		/**< Dequeue an ordered event */
+	rx_sched_unordered,		/**< Dequeue an unordered event */
+	rx_sched_atomic,		/**< Dequeue an atomic event */
+	rx_sched_directed,		/**< Dequeue an directed event */
+	rx_sched_invalid,               /**< Dequeue event sched type invalid */
+	/* common to port and queue */
+	is_configured,			/**< Port is configured */
+	is_load_balanced,		/**< Port is LDB */
+	hw_id,				/**< Hardware ID */
+	/* queue specific */
+	num_links,			/**< Number of ports linked */
+	sched_type,			/**< Queue sched type */
+	enq_ok,				/**< # events enqueued to the queue */
+	current_depth,			/**< Current queue depth */
+	depth_threshold,		/**< Programmed depth threshold */
+	depth_le50_threshold,
+	/**< Depth LE to 50% of the configured hardware threshold */
+	depth_gt50_le75_threshold,
+	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold */
+	depth_gt75_le100_threshold,
+	/**< Depth GT 75%. but LE to the configured hardware threshold */
+	depth_gt100_threshold
+	/**< Depth GT 100% of the configured hw threshold */
+};
+
+typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
+		uint16_t obj_idx, /* port or queue id */
+		enum dlb2_xstats_type stat, int extra_arg);
+
+enum dlb2_xstats_fn_type {
+	DLB2_XSTATS_FN_DEV,
+	DLB2_XSTATS_FN_PORT,
+	DLB2_XSTATS_FN_QUEUE
+};
+
+struct dlb2_xstats_entry {
+	struct rte_event_dev_xstats_name name;
+	uint64_t reset_value; /* an offset to be taken away to emulate resets */
+	enum dlb2_xstats_fn_type fn_id;
+	enum dlb2_xstats_type stat;
+	enum rte_event_dev_xstats_mode mode;
+	int extra_arg;
+	uint16_t obj_idx;
+	uint8_t reset_allowed; /* when set, this value can be reset */
+};
+
+/* Some device stats are simply a summation of the corresponding port values */
+static uint64_t
+dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
+			     int which_stat)
+{
+	int i;
+	uint64_t val = 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
+
+		if (!port->setup_done)
+			continue;
+
+		switch (which_stat) {
+		case rx_ok:
+			val += port->stats.traffic.rx_ok;
+			break;
+		case rx_drop:
+			val += port->stats.traffic.rx_drop;
+			break;
+		case rx_interrupt_wait:
+			val += port->stats.traffic.rx_interrupt_wait;
+			break;
+		case rx_umonitor_umwait:
+			val += port->stats.traffic.rx_umonitor_umwait;
+			break;
+		case tx_ok:
+			val += port->stats.traffic.tx_ok;
+			break;
+		case total_polls:
+			val += port->stats.traffic.total_polls;
+			break;
+		case zero_polls:
+			val += port->stats.traffic.zero_polls;
+			break;
+		case tx_nospc_ldb_hw_credits:
+			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
+			break;
+		case tx_nospc_dir_hw_credits:
+			val += port->stats.traffic.tx_nospc_dir_hw_credits;
+			break;
+		case tx_nospc_inflight_max:
+			val += port->stats.traffic.tx_nospc_inflight_max;
+			break;
+		case tx_nospc_new_event_limit:
+			val += port->stats.traffic.tx_nospc_new_event_limit;
+			break;
+		case tx_nospc_inflight_credits:
+			val += port->stats.traffic.tx_nospc_inflight_credits;
+			break;
+		default:
+			return -1;
+		}
+	}
+	return val;
+}
+
+static uint64_t
+get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
+	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	switch (type) {
+	case rx_ok:
+	case rx_drop:
+	case rx_interrupt_wait:
+	case rx_umonitor_umwait:
+	case tx_ok:
+	case total_polls:
+	case zero_polls:
+	case tx_nospc_ldb_hw_credits:
+	case tx_nospc_dir_hw_credits:
+	case tx_nospc_inflight_max:
+	case tx_nospc_new_event_limit:
+	case tx_nospc_inflight_credits:
+		return dlb2_device_traffic_stat_get(dlb2, type);
+	case nb_events_limit:
+		return dlb2->new_event_limit;
+	case inflight_events:
+		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
+	case ldb_pool_size:
+		return dlb2->num_ldb_credits;
+	case dir_pool_size:
+		return dlb2->num_dir_credits;
+	default: return -1;
+	}
+}
+
+static uint64_t
+get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
+
+	switch (type) {
+	case rx_ok: return ev_port->stats.traffic.rx_ok;
+
+	case rx_drop: return ev_port->stats.traffic.rx_drop;
+
+	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
+
+	case rx_umonitor_umwait:
+		return ev_port->stats.traffic.rx_umonitor_umwait;
+
+	case tx_ok: return ev_port->stats.traffic.tx_ok;
+
+	case total_polls: return ev_port->stats.traffic.total_polls;
+
+	case zero_polls: return ev_port->stats.traffic.zero_polls;
+
+	case tx_nospc_ldb_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
+
+	case tx_nospc_dir_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
+
+	case tx_nospc_inflight_max:
+		return ev_port->stats.traffic.tx_nospc_inflight_max;
+
+	case tx_nospc_new_event_limit:
+		return ev_port->stats.traffic.tx_nospc_new_event_limit;
+
+	case tx_nospc_inflight_credits:
+		return ev_port->stats.traffic.tx_nospc_inflight_credits;
+
+	case is_configured: return ev_port->setup_done;
+
+	case is_load_balanced: return !ev_port->qm_port.is_directed;
+
+	case hw_id: return ev_port->qm_port.id;
+
+	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
+
+	case tx_fwd: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
+
+	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
+
+	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
+
+	case tx_sched_ordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case tx_sched_unordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case tx_sched_atomic:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case tx_sched_directed:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case tx_invalid: return ev_port->stats.tx_invalid;
+
+	case outstanding_releases: return ev_port->outstanding_releases;
+
+	case max_outstanding_releases:
+		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	case rx_sched_ordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case rx_sched_unordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case rx_sched_atomic:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case rx_sched_directed:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
+
+	default: return -1;
+	}
+}
+
+static uint64_t
+dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
+{
+	int port = 0;
+	uint64_t tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
+
+	return tally;
+}
+
+static uint64_t
+dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
+{
+	int port = 0;
+	uint64_t enq_ok_tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
+
+	return enq_ok_tally;
+}
+
+static uint64_t
+get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_queue *ev_queue =
+		&dlb2->ev_queues[obj_idx];
+
+	switch (type) {
+	case is_configured: return ev_queue->setup_done;
+
+	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
+
+	case hw_id: return ev_queue->qm_queue.id;
+
+	case num_links: return ev_queue->num_links;
+
+	case sched_type: return ev_queue->qm_queue.sched_type;
+
+	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
+
+	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
+
+	case depth_threshold: return ev_queue->depth_threshold;
+
+	case depth_le50_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_LE50);
+
+	case depth_gt50_le75_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT50_LE75);
+
+	case depth_gt75_le100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT75_LE100);
+
+	case depth_gt100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT100);
+
+	default: return -1;
+	}
+}
+
+int
+dlb2_xstats_init(struct dlb2_eventdev *dlb2)
+{
+	/*
+	 * define the stats names and types. Used to build up the device
+	 * xstats array
+	 * There are multiple set of stats:
+	 *   - device-level,
+	 *   - per-port,
+	 *   - per-qid,
+	 *
+	 * For each of these sets, we have three parallel arrays, one for the
+	 * names, the other for the stat type parameter to be passed in the fn
+	 * call to get that stat. The third array allows resetting or not.
+	 * All these arrays must be kept in sync
+	 */
+	static const char * const dev_stats[] = {
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"nb_events_limit",
+		"inflight_events",
+		"ldb_pool_size",
+		"dir_pool_size",
+	};
+	static const enum dlb2_xstats_type dev_types[] = {
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		nb_events_limit,
+		inflight_events,
+		ldb_pool_size,
+		dir_pool_size,
+	};
+	/* Note: generated device stats are not allowed to be reset. */
+	static const uint8_t dev_reset_allowed[] = {
+		0, /* rx_ok */
+		0, /* rx_drop */
+		0, /* rx_interrupt_wait */
+		0, /* rx_umonitor_umwait */
+		0, /* tx_ok */
+		0, /* total_polls */
+		0, /* zero_polls */
+		0, /* tx_nospc_ldb_hw_credits */
+		0, /* tx_nospc_dir_hw_credits */
+		0, /* tx_nospc_inflight_max */
+		0, /* tx_nospc_new_event_limit */
+		0, /* tx_nospc_inflight_credits */
+		0, /* nb_events_limit */
+		0, /* inflight_events */
+		0, /* ldb_pool_size */
+		0, /* dir_pool_size */
+	};
+	static const char * const port_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"tx_new",
+		"tx_fwd",
+		"tx_rel",
+		"tx_implicit_rel",
+		"tx_sched_ordered",
+		"tx_sched_unordered",
+		"tx_sched_atomic",
+		"tx_sched_directed",
+		"tx_invalid",
+		"outstanding_releases",
+		"max_outstanding_releases",
+		"rx_sched_ordered",
+		"rx_sched_unordered",
+		"rx_sched_atomic",
+		"rx_sched_directed",
+		"rx_sched_invalid"
+	};
+	static const enum dlb2_xstats_type port_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		tx_new,
+		tx_fwd,
+		tx_rel,
+		tx_implicit_rel,
+		tx_sched_ordered,
+		tx_sched_unordered,
+		tx_sched_atomic,
+		tx_sched_directed,
+		tx_invalid,
+		outstanding_releases,
+		max_outstanding_releases,
+		rx_sched_ordered,
+		rx_sched_unordered,
+		rx_sched_atomic,
+		rx_sched_directed,
+		rx_sched_invalid
+	};
+	static const uint8_t port_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		1, /* rx_ok */
+		1, /* rx_drop */
+		1, /* rx_interrupt_wait */
+		1, /* rx_umonitor_umwait */
+		1, /* tx_ok */
+		1, /* total_polls */
+		1, /* zero_polls */
+		1, /* tx_nospc_ldb_hw_credits */
+		1, /* tx_nospc_dir_hw_credits */
+		1, /* tx_nospc_inflight_max */
+		1, /* tx_nospc_new_event_limit */
+		1, /* tx_nospc_inflight_credits */
+		1, /* tx_new */
+		1, /* tx_fwd */
+		1, /* tx_rel */
+		1, /* tx_implicit_rel */
+		1, /* tx_sched_ordered */
+		1, /* tx_sched_unordered */
+		1, /* tx_sched_atomic */
+		1, /* tx_sched_directed */
+		1, /* tx_invalid */
+		0, /* outstanding_releases */
+		0, /* max_outstanding_releases */
+		1, /* rx_sched_ordered */
+		1, /* rx_sched_unordered */
+		1, /* rx_sched_atomic */
+		1, /* rx_sched_directed */
+		1  /* rx_sched_invalid */
+	};
+
+	/* QID specific stats */
+	static const char * const qid_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"num_links",
+		"sched_type",
+		"enq_ok",
+		"current_depth",
+		"depth_threshold",
+		"depth_le50_threshold",
+		"depth_gt50_le75_threshold",
+		"depth_gt75_le100_threshold",
+		"depth_gt100_threshold",
+	};
+	static const enum dlb2_xstats_type qid_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		num_links,
+		sched_type,
+		enq_ok,
+		current_depth,
+		depth_threshold,
+		depth_le50_threshold,
+		depth_gt50_le75_threshold,
+		depth_gt75_le100_threshold,
+		depth_gt100_threshold,
+	};
+	static const uint8_t qid_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		0, /* num_links */
+		0, /* sched_type */
+		1, /* enq_ok */
+		0, /* current_depth */
+		0, /* depth_threshold */
+		1, /* depth_le50_threshold */
+		1, /* depth_gt50_le75_threshold */
+		1, /* depth_gt75_le100_threshold */
+		1, /* depth_gt100_threshold */
+	};
+
+	/* ---- end of stat definitions ---- */
+
+	/* check sizes, since a missed comma can lead to strings being
+	 * joined by the compiler.
+	 */
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
+
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_reset_allowed));
+
+	/* other vars */
+	const unsigned int count = RTE_DIM(dev_stats) +
+			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
+			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
+	unsigned int i, port, qid, stat_id = 0;
+
+	dlb2->xstats = rte_zmalloc_socket(NULL,
+			sizeof(dlb2->xstats[0]) * count, 0,
+			dlb2->qm_instance.info.socket_id);
+	if (dlb2->xstats == NULL)
+		return -ENOMEM;
+
+#define sname dlb2->xstats[stat_id].name.name
+	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
+		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
+			.fn_id = DLB2_XSTATS_FN_DEV,
+			.stat = dev_types[i],
+			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
+			.reset_allowed = dev_reset_allowed[i],
+		};
+		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
+	}
+	dlb2->xstats_count_mode_dev = stat_id;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
+		dlb2->xstats_offset_for_port[port] = stat_id;
+
+		uint32_t count_offset = stat_id;
+
+		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_PORT,
+				.obj_idx = port,
+				.stat = port_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_PORT,
+				.reset_allowed = port_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "port_%u_%s",
+				 port, port_stats[i]);
+		}
+
+		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_port = stat_id - dlb2->xstats_count_mode_dev;
+
+	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
+		uint32_t count_offset = stat_id;
+
+		dlb2->xstats_offset_for_qid[qid] = stat_id;
+
+		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_QUEUE,
+				.obj_idx = qid,
+				.stat = qid_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
+				.reset_allowed = qid_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "qid_%u_%s",
+				 qid, qid_stats[i]);
+		}
+
+		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_queue = stat_id -
+		(dlb2->xstats_count_mode_dev + dlb2->xstats_count_mode_port);
+#undef sname
+
+	dlb2->xstats_count = stat_id;
+
+	return 0;
+}
+
+void
+dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
+{
+	rte_free(dlb2->xstats);
+	dlb2->xstats_count = 0;
+}
+
+int
+dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstats_names,
+		unsigned int *ids, unsigned int size)
+{
+	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+	uint32_t start_offset = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			break;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			break;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
+		break;
+	default:
+		return -EINVAL;
+	};
+
+	if (xstats_mode_count > size || ids == NULL || xstats_names == NULL)
+		return xstats_mode_count;
+
+	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
+		if (dlb2->xstats[i].mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != dlb2->xstats[i].obj_idx)
+			continue;
+
+		xstats_names[xidx] = dlb2->xstats[i].name;
+		if (ids)
+			ids[xidx] = start_offset + xidx;
+		xidx++;
+	}
+	return xidx;
+}
+
+static int
+dlb2_xstats_update(struct dlb2_eventdev *dlb2,
+		enum rte_event_dev_xstats_mode mode,
+		uint8_t queue_port_id, const unsigned int ids[],
+		uint64_t values[], unsigned int n, const uint32_t reset)
+{
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			goto invalid_value;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			goto invalid_value;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		break;
+	default:
+		goto invalid_value;
+	};
+
+	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
+		dlb2_xstats_fn fn;
+
+		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != xs->obj_idx)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			goto invalid_value;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+
+		if (values)
+			values[xidx] = val;
+
+		if (xs->reset_allowed && reset)
+			xs->reset_value += val;
+
+		xidx++;
+	}
+
+	return xidx;
+
+invalid_value:
+	return -EINVAL;
+}
+
+int
+dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	const uint32_t reset = 0;
+
+	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
+				  reset);
+}
+
+uint64_t
+dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+				 const char *name, unsigned int *id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	dlb2_xstats_fn fn;
+
+	for (i = 0; i < dlb2->xstats_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (strncmp(xs->name.name, name,
+			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
+			if (id != NULL)
+				*id = i;
+
+			switch (xs->fn_id) {
+			case DLB2_XSTATS_FN_DEV:
+				fn = get_dev_stat;
+				break;
+			case DLB2_XSTATS_FN_PORT:
+				fn = get_port_stat;
+				break;
+			case DLB2_XSTATS_FN_QUEUE:
+				fn = get_queue_stat;
+				break;
+			default:
+				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
+					  xs->fn_id);
+				return (uint64_t)-1;
+			}
+
+			return fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+		}
+	}
+	if (id != NULL)
+		*id = (uint32_t)-1;
+	return (uint64_t)-1;
+}
+
+static void
+dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
+			uint32_t num)
+{
+	uint32_t i;
+	dlb2_xstats_fn fn;
+
+	for (i = start; i < start + num; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (!xs->reset_allowed)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			return;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
+		xs->reset_value = val;
+	}
+}
+
+static int
+dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
+			const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_QUEUE,
+					queue_id, ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	if (ids == NULL)
+		dlb2_xstats_reset_range(dlb2,
+			dlb2->xstats_offset_for_qid[queue_id],
+			dlb2->xstats_count_per_qid[queue_id]);
+
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
+		       const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+	int offset = dlb2->xstats_offset_for_port[port_id];
+	int nb_stat = dlb2->xstats_count_per_port[port_id];
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_PORT, port_id,
+					ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
+		      uint32_t nb_ids)
+{
+	uint32_t i;
+
+	if (ids) {
+		for (i = 0; i < nb_ids; i++) {
+			uint32_t id = ids[i];
+
+			if (id >= dlb2->xstats_count_mode_dev)
+				return -EINVAL;
+			dlb2_xstats_reset_range(dlb2, id, 1);
+		}
+	} else {
+		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
+			dlb2_xstats_reset_range(dlb2, i, 1);
+	}
+
+	return 0;
+}
+
+int
+dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+			   enum rte_event_dev_xstats_mode mode,
+			   int16_t queue_port_id,
+			   const uint32_t ids[],
+			   uint32_t nb_ids)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	uint32_t i;
+
+	/* handle -1 for queue_port_id here, looping over all ports/queues */
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
+			return -EINVAL;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+				if (dlb2_xstats_reset_port(dlb2, i,
+							   ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
+			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
+						   ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
+				if (dlb2_xstats_reset_queue(dlb2, i,
+							    ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
+			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
+						    ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	};
+
+	return 0;
+}
+
+void
+dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	int i;
+
+	if (f == NULL) {
+		printf("Invalid file pointer\n");
+		return;
+	}
+
+	if (dev == NULL) {
+		fprintf(f, "Invalid event device\n");
+		return;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (dlb2 == NULL) {
+		fprintf(f, "DLB2 Event device cannot be dumped!\n");
+		return;
+	}
+
+	if (!dlb2->configured)
+		fprintf(f, "DLB2 Event device is not configured\n");
+
+	handle = &dlb2->qm_instance;
+
+	fprintf(f, "================\n");
+	fprintf(f, "DLB2 Device Dump\n");
+	fprintf(f, "================\n");
+
+	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
+		dlb2->umwait_allowed ? "yes" : "no");
+
+	/* Generic top level device information */
+
+	fprintf(f, "device is configured and run state =");
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		fprintf(f, "STOPPED\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
+		fprintf(f, "STOPPING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
+		fprintf(f, "STARTING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
+		fprintf(f, "STARTED\n");
+	else
+		fprintf(f, "UNEXPECTED\n");
+
+	fprintf(f, "domain ID=%u, socket_id=%u, evdev=%p\n",
+		handle->domain_id, handle->info.socket_id, dlb2->event_dev);
+
+	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
+		dlb2->num_dir_ports, dlb2->num_dir_queues);
+
+	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
+		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
+
+	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
+		handle->cfg.resources.num_atomic_inflights,
+		handle->cfg.resources.num_hist_list_entries);
+
+	fprintf(f, "results from most recent hw resource query:\n");
+
+	fprintf(f, "\tnum_sched_domains = %u\n",
+		dlb2->hw_rsrc_query_results.num_sched_domains);
+
+	fprintf(f, "\tnum_ldb_queues = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_queues);
+
+	fprintf(f, "\tnum_ldb_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_ports);
+
+	fprintf(f, "\tnum_dir_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_ports);
+
+	fprintf(f, "\tnum_atomic_inflights = %u\n",
+		dlb2->hw_rsrc_query_results.num_atomic_inflights);
+
+	fprintf(f, "\tnum_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.num_hist_list_entries);
+
+	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
+
+	fprintf(f, "\tnum_ldb_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_credits);
+
+	fprintf(f, "\tnum_dir_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_credits);
+
+	/* Port level information */
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
+		int j;
+
+		if (!p->enq_configured)
+			fprintf(f, "Port_%d is not configured\n", i);
+
+		fprintf(f, "Port_%d\n", i);
+		fprintf(f, "=======\n");
+
+		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
+			p->id, p->setup_done);
+
+		fprintf(f, "\tconfig state=%d, port state=%d\n",
+			p->qm_port.config_state, p->qm_port.state);
+
+		fprintf(f, "\tport is %s\n",
+			p->qm_port.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\toutstanding releases=%u\n",
+			p->outstanding_releases);
+
+		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
+			p->inflight_max, p->inflight_credits);
+
+		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
+			p->credit_update_quanta, p->implicit_release);
+
+		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (p->link[j].valid)
+				fprintf(f, "id=%u prio=%u ",
+					p->link[j].queue_id,
+					p->link[j].priority);
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
+
+		fprintf(f, "\tcached_ldb_credits=%u\n",
+			p->qm_port.cached_ldb_credits);
+
+		fprintf(f, "\tldb_credits = %u\n",
+			p->qm_port.ldb_credits);
+
+		fprintf(f, "\tcached_dir_credits = %u\n",
+			p->qm_port.cached_dir_credits);
+
+		fprintf(f, "\tdir_credits = %u\n",
+			p->qm_port.dir_credits);
+
+		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
+			p->qm_port.gen_bit,
+			p->qm_port.cq_idx,
+			p->qm_port.cq_depth);
+
+		fprintf(f, "\tinterrupt armed=%d\n",
+			p->qm_port.int_armed);
+
+		fprintf(f, "\tPort statistics\n");
+
+		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
+			p->stats.traffic.rx_ok);
+
+		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
+			p->stats.traffic.rx_drop);
+
+		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
+			p->stats.traffic.rx_interrupt_wait);
+
+		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
+			p->stats.traffic.rx_umonitor_umwait);
+
+		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
+			p->stats.traffic.tx_ok);
+
+		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
+			p->stats.traffic.total_polls);
+
+		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
+			p->stats.traffic.zero_polls);
+
+		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_ldb_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_dir_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_max);
+
+		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_new_event_limit);
+
+		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_credits);
+
+		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
+
+		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
+
+		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
+
+		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
+			p->stats.tx_implicit_rel);
+
+		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
+			p->stats.tx_invalid);
+
+		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
+			p->stats.rx_sched_invalid);
+	}
+
+	/* Queue level information */
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
+		int j, k;
+
+		if (!q->setup_done)
+			fprintf(f, "Queue_%d is not configured\n", i);
+
+		fprintf(f, "Queue_%d\n", i);
+		fprintf(f, "========\n");
+
+		fprintf(f, "\tevqueue_%u is set up\n", q->id);
+
+		fprintf(f, "\tqueue is %s\n",
+			q->qm_queue.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
+
+		for (j = 0; j < dlb2->num_ports; j++) {
+			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
+
+			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++) {
+				if (p->link[k].valid &&
+				    p->link[k].queue_id == q->id)
+					fprintf(f, "id=%u prio=%u ",
+						p->id, p->link[k].priority);
+			}
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\tcurrent depth: %u events\n",
+			dlb2_get_queue_depth(dlb2, q));
+
+		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
+			q->qm_queue.num_qid_inflights, q->qm_queue.sched_type);
+	}
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 28fef3f..dc16c96 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files('dlb2.c',
 		'dlb2_iface.c',
+		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c'
-- 
2.6.4

[dpdk-dev] [PATCH v3 10/23] event/dlb2: add infos get and configure

[Timothy McDaniel]

Add support for configuring the DLB2 hardware.
In particular, this patch configures the DLB2
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  116 +
 drivers/event/dlb2/dlb2.c                  |  313 +++
 drivers/event/dlb2/dlb2_iface.c            |    5 +
 drivers/event/dlb2/dlb2_iface.h            |    4 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 3237 ++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |   15 +
 drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
 7 files changed, 3734 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index e3091c9..5f6c486 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -32,9 +32,125 @@ detailed understanding of the hardware, but these details are important when
 writing high-performance code. This section describes the places where the
 eventdev API and DLB2 misalign.
 
+Scheduling Domain Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are 32 scheduling domainis the DLB2.
+When one is configured, it allocates load-balanced and
+directed queues, ports, credits, and other hardware resources. Some
+resource allocations are user-controlled -- the number of queues, for example
+-- and others, like credit pools (one directed and one load-balanced pool per
+scheduling domain), are not.
+
+The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
+setup argument and the per-port ``new_event_threshold`` argument apply as
+defined in the eventdev header file. The limit is applied to all enqueues,
+regardless of whether it will consume a directed or load-balanced credit.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Reconfiguration
+~~~~~~~~~~~~~~~
+
+The Eventdev API allows one to reconfigure a device, its ports, and its queues
+by first stopping the device, calling the configuration function(s), then
+restarting the device. The DLB2 does not support configuring an individual queue
+or port without first reconfiguring the entire device, however, so there are
+certain reconfiguration sequences that are valid in the eventdev API but not
+supported by the PMD.
+
+Specifically, the PMD supports the following configuration sequence:
+1. Configure and start the device
+2. Stop the device
+3. (Optional) Reconfigure the device
+4. (Optional) If step 3 is run:
+
+   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
+   b. The reconfigured port(s) lose their previous queue links.
+
+5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
+6. Restart the device. If the device is reconfigured in step 3 but one or more
+   of its ports or queues are not, the PMD will apply their previous
+   configuration (including port->queue links) at this time.
+
+The PMD does not support the following configuration sequences:
+1. Configure and start the device
+2. Stop the device
+3. Setup queue or setup port
+4. Start the device
+
+This sequence is not supported because the event device must be reconfigured
+before its ports or queues can be.
+
+Atomic Inflights Allocation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
+inflight event in a temporary buffer that is divided among load-balanced
+queues. If a queue's atomic buffer storage fills up, this can result in
+head-of-line-blocking. For example:
+
+- An LDB queue allocated N atomic buffer entries
+- All N entries are filled with events from flow X, which is pinned to CQ 0.
+
+Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
+scheduled. The likelihood of this case depends on the eventdev configuration,
+traffic behavior, event processing latency, potential for a worker to be
+interrupted or otherwise delayed, etc.
+
+By default, the PMD allocates 16 buffer entries for each load-balanced queue,
+which provides an even division across all 128 queues but potentially wastes
+buffer space (e.g. if not all queues are used, or aren't used for atomic
+scheduling).
+
+The PMD provides a dev arg to override the default per-queue allocation. To
+increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,atm_inflights=64
+
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 925824e..6798e66 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -85,6 +85,25 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
 	return 0;
 }
 
+static void
+dlb2_free_qe_mem(struct dlb2_port *qm_port)
+{
+	if (qm_port == NULL)
+		return;
+
+	rte_free(qm_port->qe4);
+	qm_port->qe4 = NULL;
+
+	rte_free(qm_port->int_arm_qe);
+	qm_port->int_arm_qe = NULL;
+
+	rte_free(qm_port->consume_qe);
+	qm_port->consume_qe = NULL;
+
+	rte_memzone_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz);
+	dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz = NULL;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -350,10 +369,304 @@ set_qid_depth_thresh(const char *key __rte_unused,
 }
 
 static void
+dlb2_eventdev_info_get(struct rte_eventdev *dev,
+		       struct rte_event_dev_info *dev_info)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret;
+
+	ret = dlb2_hw_query_resources(dlb2);
+	if (ret) {
+		const struct rte_eventdev_data *data = dev->data;
+
+		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+			     ret, data->dev_id);
+		/* fn is void, so fall through and return values set up in
+		 * probe
+		 */
+	}
+
+	/* Add num resources currently owned by this domain.
+	 * These would become available if the scheduling domain were reset due
+	 * to the application recalling eventdev_configure to *reconfigure* the
+	 * domain.
+	 */
+	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
+	evdev_dlb2_default_info.max_event_queues += dlb2->num_ldb_queues;
+	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
+
+	evdev_dlb2_default_info.max_event_queues =
+		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
+			RTE_EVENT_MAX_QUEUES_PER_DEV);
+
+	evdev_dlb2_default_info.max_num_events =
+		RTE_MIN(evdev_dlb2_default_info.max_num_events,
+			dlb2->max_num_events_override);
+
+	*dev_info = evdev_dlb2_default_info;
+}
+
+static int
+dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
+			    const struct dlb2_hw_rsrcs *resources_asked)
+{
+	int ret = 0;
+	struct dlb2_create_sched_domain_args *cfg;
+
+	if (resources_asked == NULL) {
+		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
+		ret = EINVAL;
+		goto error_exit;
+	}
+
+	/* Map generic qm resources to dlb2 resources */
+	cfg = &handle->cfg.resources;
+
+	/* DIR ports and queues */
+
+	cfg->num_dir_ports = resources_asked->num_dir_ports;
+
+	cfg->num_dir_credits = resources_asked->num_dir_credits;
+
+	/* LDB queues */
+
+	cfg->num_ldb_queues = resources_asked->num_ldb_queues;
+
+	/* LDB ports */
+
+	cfg->cos_strict = 0; /* Best effort */
+	cfg->num_cos_ldb_ports[0] = 0;
+	cfg->num_cos_ldb_ports[1] = 0;
+	cfg->num_cos_ldb_ports[2] = 0;
+	cfg->num_cos_ldb_ports[3] = 0;
+
+	switch (handle->cos_id) {
+	case DLB2_COS_0:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[0] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_1:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[1] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_2:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[2] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_3:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[3] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_DEFAULT:
+		/* all ldb ports are don't care ports from a cos perspective */
+		cfg->num_ldb_ports =
+			resources_asked->num_ldb_ports;
+		break;
+	}
+
+	cfg->num_ldb_credits =
+		resources_asked->num_ldb_credits;
+
+	cfg->num_atomic_inflights =
+		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
+		cfg->num_ldb_queues;
+
+	cfg->num_hist_list_entries = resources_asked->num_ldb_ports *
+		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d\n",
+		     cfg->num_ldb_queues,
+		     resources_asked->num_ldb_ports,
+		     cfg->num_dir_ports,
+		     cfg->num_atomic_inflights,
+		     cfg->num_hist_list_entries,
+		     cfg->num_ldb_credits,
+		     cfg->num_dir_credits);
+
+	/* Configure the QM */
+
+	ret = dlb2_iface_sched_domain_create(handle, cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: domain create failed, ret = %d, extra status: %s\n",
+			     ret,
+			     dlb2_error_strings[cfg->response.status]);
+
+		goto error_exit;
+	}
+
+	handle->domain_id = cfg->response.id;
+	handle->cfg.configured = true;
+
+error_exit:
+
+	return ret;
+}
+
+static void
+dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	enum dlb2_configuration_state config_state;
+	int i, j;
+
+	dlb2_iface_domain_reset(dlb2);
+
+	/* Free all dynamically allocated port memory */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
+
+	/* If reconfiguring, mark the device's queues and ports as "previously
+	 * configured." If the user doesn't reconfigure them, the PMD will
+	 * reapply their previous configuration when the device is started.
+	 */
+	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
+		DLB2_NOT_CONFIGURED;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		dlb2->ev_ports[i].qm_port.config_state = config_state;
+		/* Reset setup_done so ports can be reconfigured */
+		dlb2->ev_ports[i].setup_done = false;
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			dlb2->ev_ports[i].link[j].mapped = false;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++)
+		dlb2->ev_queues[i].qm_queue.config_state = config_state;
+
+	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
+		dlb2->ev_queues[i].setup_done = false;
+
+	dlb2->num_ports = 0;
+	dlb2->num_ldb_ports = 0;
+	dlb2->num_dir_ports = 0;
+	dlb2->num_queues = 0;
+	dlb2->num_ldb_queues = 0;
+	dlb2->num_dir_queues = 0;
+	dlb2->configured = false;
+}
+
+/* Note: 1 QM instance per QM device, QM instance/device == event device */
+static int
+dlb2_eventdev_configure(const struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
+	const struct rte_eventdev_data *data = dev->data;
+	const struct rte_event_dev_config *config = &data->dev_conf;
+	int ret;
+
+	/* If this eventdev is already configured, we must release the current
+	 * scheduling domain before attempting to configure a new one.
+	 */
+	if (dlb2->configured) {
+		dlb2_hw_reset_sched_domain(dev, true);
+
+		ret = dlb2_hw_query_resources(dlb2);
+		if (ret) {
+			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+				     ret, data->dev_id);
+			return ret;
+		}
+	}
+
+	if (config->nb_event_queues > rsrcs->num_queues) {
+		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_queues,
+			     rsrcs->num_queues);
+		return -EINVAL;
+	}
+	if (config->nb_event_ports > (rsrcs->num_ldb_ports
+			+ rsrcs->num_dir_ports)) {
+		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_ports,
+			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
+		return -EINVAL;
+	}
+	if (config->nb_events_limit > rsrcs->nb_events_limit) {
+		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_events_limit,
+			     rsrcs->nb_events_limit);
+		return -EINVAL;
+	}
+
+	if (config->event_dev_cfg & RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT)
+		dlb2->global_dequeue_wait = false;
+	else {
+		uint32_t timeout32;
+
+		dlb2->global_dequeue_wait = true;
+
+		/* note size mismatch of timeout vals in eventdev lib. */
+		timeout32 = config->dequeue_timeout_ns;
+
+		dlb2->global_dequeue_wait_ticks =
+			timeout32 * (rte_get_timer_hz() / 1E9);
+	}
+
+	/* Does this platform support umonitor/umwait? */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
+		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
+			DLB2_LOG_ERR("invalid value (%d) for RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
+				     RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
+			return -EINVAL;
+		}
+		dlb2->umwait_allowed = true;
+	}
+
+	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
+	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs->num_dir_ports;
+	/* 1 dir queue per dir port */
+	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports;
+
+	/* Scale down nb_events_limit by 4 for directed credits, since there
+	 * are 4x as many load-balanced credits.
+	 */
+	rsrcs->num_ldb_credits = 0;
+	rsrcs->num_dir_credits = 0;
+
+	if (rsrcs->num_ldb_queues)
+		rsrcs->num_ldb_credits = config->nb_events_limit;
+	if (rsrcs->num_dir_ports)
+		rsrcs->num_dir_credits = config->nb_events_limit / 4;
+	if (dlb2->num_dir_credits_override != -1)
+		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
+
+	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
+		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
+		return -ENODEV;
+	}
+
+	dlb2->new_event_limit = config->nb_events_limit;
+	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
+
+	/* Save number of ports/queues for this event dev */
+	dlb2->num_ports = config->nb_event_ports;
+	dlb2->num_queues = config->nb_event_queues;
+	dlb2->num_dir_ports = rsrcs->num_dir_ports;
+	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
+	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
+	dlb2->num_dir_queues = dlb2->num_dir_ports;
+	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
+	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
+	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
+	dlb2->max_dir_credits = rsrcs->num_dir_credits;
+
+	dlb2->configured = true;
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dev_infos_get    = dlb2_eventdev_info_get,
+		.dev_configure    = dlb2_eventdev_configure,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 0fc9991..a829b9b 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -26,3 +26,8 @@ int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
+
+int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				struct dlb2_create_sched_domain_args *args);
+
+void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4fb416e..6663dab 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -26,4 +26,8 @@ extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
 
+extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				 struct dlb2_create_sched_domain_args *args);
+
+extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 6de8b95..63a68a6 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -12,6 +12,27 @@
 #include "dlb2_regs.h"
 #include "dlb2_resource.h"
 
+#include "../../dlb2_priv.h"
+#include "../../dlb2_inline_fns.h"
+
+#define DLB2_DOM_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, domain_list, iter)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, func_list, iter)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, domain_list, it, it_tmp)
+
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, func_list, it, it_tmp)
+
 static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
 {
 	int i;
@@ -272,3 +293,3219 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
 
 	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
 }
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+	union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+	r0.field.count = domain->num_ldb_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+	r1.field.count = domain->num_dir_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+		       struct dlb2_function_resources *rsrcs,
+		       u32 domain_id,
+		       u32 cos_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	RTE_SET_USED(iter);
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+				   typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				   struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_ports,
+				   u32 cos_id,
+				   struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_ldb_port *port;
+
+		port = dlb2_get_next_ldb_port(hw, rsrcs,
+					      domain->id.phys_id, cos_id);
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_ports[cos_id],
+			      &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_ports[cos_id],
+			      &port->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_create_sched_domain_args *args,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      num,
+						      i,
+						      resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+					ret = __dlb2_attach_ldb_ports(hw,
+								      rsrcs,
+								      domain,
+								      1,
+								      cos_id,
+								      resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret < 0)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      1,
+						      j,
+						      resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 u32 num_ports,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+					   typeof(*port));
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_dir_pq_pairs, &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_dir_pq_pairs, &port->domain_list);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+					struct dlb2_hw_domain *domain,
+					u32 num_atomic_inflights,
+					struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+				     struct dlb2_hw_domain *domain,
+				     u32 num_hist_list_entries,
+				     struct dlb2_cmd_response *resp)
+{
+	struct dlb2_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb2_bitmap_find_set_bit_range(bitmap,
+						      num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+				  struct dlb2_function_resources *rsrcs,
+				  struct dlb2_hw_domain *domain,
+				  u32 num_queues,
+				  struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb2_ldb_queue *queue;
+
+		queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+					    typeof(*queue));
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_queues, &queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_queues, &queue->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+			     struct dlb2_function_resources *rsrcs,
+			     struct dlb2_hw_domain *domain,
+			     struct dlb2_create_sched_domain_args *args,
+			     struct dlb2_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb2_attach_ldb_queues(hw,
+				     rsrcs,
+				     domain,
+				     args->num_ldb_queues,
+				     resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_ports(hw,
+				    rsrcs,
+				    domain,
+				    args,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_ports(hw,
+				    rsrcs,
+				    domain,
+				    args->num_dir_ports,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_credits(rsrcs,
+				      domain,
+				      args->num_ldb_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_credits(rsrcs,
+				      domain,
+				      args->num_dir_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+						   domain,
+						   args->num_hist_list_entries,
+						   resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_atomic_inflights(rsrcs,
+					   domain,
+					   args->num_atomic_inflights,
+					   resp);
+	if (ret < 0)
+		return ret;
+
+	dlb2_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+				  struct dlb2_create_sched_domain_args *args,
+				  struct dlb2_cmd_response *resp)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb2_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+				  struct dlb2_create_sched_domain_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		    args->num_ldb_queues);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		    args->num_ldb_ports);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		    args->num_cos_ldb_ports[0]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		    args->cos_strict);
+	DLB2_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		    args->num_dir_ports);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		    args->num_atomic_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		    args->num_hist_list_entries);
+	DLB2_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		    args->num_ldb_credits);
+	DLB2_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		    args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ *	domain and its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+	if (ret)
+		return ret;
+
+	domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available domains\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (domain->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_domains contains configured domains.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	dlb2_init_domain_rsrc_lists(domain);
+
+	ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to verify args.\n",
+			    __func__);
+
+		return ret;
+	}
+
+	dlb2_list_del(&rsrcs->avail_domains, &domain->func_list);
+
+	dlb2_list_add(&rsrcs->used_domains, &domain->func_list);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+	DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+			     struct dlb2_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb2_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	int ret;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_disable(hw, port);
+
+		ret = dlb2_drain_dir_cq(hw, port);
+		if (ret < 0)
+			return ret;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_dir_pq_pair *queue)
+{
+	union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *queue)
+{
+	return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+				      struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		dlb2_movdir64b(pp_addr, hcw);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int ret, i;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			if (toggle_port)
+				dlb2_ldb_port_cq_disable(hw, port);
+
+			ret = dlb2_drain_ldb_cq(hw, port);
+			if (ret < 0)
+				return ret;
+
+			if (toggle_port)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+
+	return 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_atm_active r1;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_ldb_queue *queue)
+{
+	return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb2_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+					   struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to unmap domain queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = true;
+
+			dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+			   u32 id,
+			   bool vdev_req,
+			   unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iter1) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter2)
+			if (queue->id.virt_id == id)
+				return queue;
+	}
+
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue, iter1)
+		if (queue->id.virt_id == id)
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+						      u32 id,
+						      bool vdev_req,
+						      unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iteration;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	RTE_SET_USED(iteration);
+
+	if (id >= DLB2_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iteration)
+		if (domain->id.virt_id == id)
+			return domain;
+
+	return NULL;
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot,
+					   enum dlb2_qid_map_state new_state)
+{
+	enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb2_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB2_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB2_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB2_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB2_HW_DBG(hw,
+		    "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB2_HW_ERR(hw,
+		    "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+				enum dlb2_qid_map_state state,
+				int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+				      enum dlb2_qid_map_state state,
+				      struct dlb2_ldb_queue *queue,
+				      int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+					      struct dlb2_hw_domain *domain,
+					      struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain,
+					     struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+						struct dlb2_ldb_port *port,
+						int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 0;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 1;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+					struct dlb2_ldb_port *p,
+					struct dlb2_ldb_queue *q,
+					u8 priority)
+{
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_lsp_cq2qid0 r1;
+	union dlb2_atm_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix_00 r3;
+	union dlb2_lsp_qid2cqidix2_00 r4;
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+	    !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+	r0.field.v |= 1 << i;
+	r0.field.prio |= (priority & 0x7) << i * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+	else
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		r1.field.qid_p0 = q->id.phys_id;
+	if (i == 1 || i == 5)
+		r1.field.qid_p1 = q->id.phys_id;
+	if (i == 2 || i == 6)
+		r1.field.qid_p2 = q->id.phys_id;
+	if (i == 3 || i == 7)
+		r1.field.qid_p3 = q->id.phys_id;
+
+	if (i < 4)
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+	else
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r4.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+						  p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		r2.field.cq_p0 |= 1 << i;
+		r3.field.cq_p0 |= 1 << i;
+		r4.field.cq_p0 |= 1 << i;
+		break;
+
+	case 1:
+		r2.field.cq_p1 |= 1 << i;
+		r3.field.cq_p1 |= 1 << i;
+		r4.field.cq_p1 |= 1 << i;
+		break;
+
+	case 2:
+		r2.field.cq_p2 |= 1 << i;
+		r3.field.cq_p2 |= 1 << i;
+		r4.field.cq_p2 |= 1 << i;
+		break;
+
+	case 3:
+		r2.field.cq_p3 |= 1 << i;
+		r3.field.cq_p3 |= 1 << i;
+		r4.field.cq_p3 |= 1 << i;
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r3.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		    r4.val);
+
+	dlb2_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB2_QUEUE_MAPPED;
+
+	return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	/* Set the atomic scheduling haswork bit */
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.rlist_haswork_v = r0.field.count > 0;
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      u8 slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.rlist_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.nalb_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+					      struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+	r0.field.limit = queue->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+						struct dlb2_ldb_queue *queue)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+						struct dlb2_hw_domain *domain,
+						struct dlb2_ldb_port *port,
+						struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_lsp_qid_ldb_infl_cnt r0;
+	enum dlb2_qid_map_state state;
+	int slot, ret, i;
+	u8 prio;
+	RTE_SET_USED(iter);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: non-zero QID inflight count\n",
+			    __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			state = DLB2_QUEUE_MAPPED;
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+					 struct dlb2_ldb_port *port,
+					 struct dlb2_ldb_queue *queue,
+					 u8 priority)
+{
+	union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	int domain_id, slot, ret;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+		DLB2_HW_ERR(hw,
+			    "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		if (port->enabled)
+			dlb2_ldb_port_cq_enable(hw, port);
+
+		dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain,
+					struct dlb2_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		union dlb2_lsp_qid_ldb_infl_cnt r0;
+		struct dlb2_ldb_queue *queue;
+		int qid;
+
+		if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: unable to find queue %d\n",
+				    __func__, qid);
+			continue;
+		}
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count)
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb2_ldb_port_cq_disable(hw, port);
+
+		dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count) {
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+
+			dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port,
+				   struct dlb2_ldb_queue *queue)
+{
+	enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_atm_qid2cqidix_00 r1;
+	union dlb2_lsp_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix2_00 r3;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB2_QUEUE_MAPPED;
+	in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: QID %d isn't mapped\n",
+			    __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+	r0.field.v &= ~(1 << i);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		r1.field.cq_p0 &= ~(1 << i);
+		r2.field.cq_p0 &= ~(1 << i);
+		r3.field.cq_p0 &= ~(1 << i);
+		break;
+
+	case 1:
+		r1.field.cq_p1 &= ~(1 << i);
+		r2.field.cq_p1 &= ~(1 << i);
+		r3.field.cq_p1 &= ~(1 << i);
+		break;
+
+	case 2:
+		r1.field.cq_p2 &= ~(1 << i);
+		r2.field.cq_p2 &= ~(1 << i);
+		r3.field.cq_p2 &= ~(1 << i);
+		break;
+
+	case 3:
+		r1.field.cq_p3 &= ~(1 << i);
+		r2.field.cq_p3 &= ~(1 << i);
+		r3.field.cq_p3 &= ~(1 << i);
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+		    r1.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+		    r3.val);
+
+	dlb2_flush_csr(hw);
+
+	unmapped = DLB2_QUEUE_UNMAPPED;
+
+	return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_ldb_port *port,
+				 struct dlb2_ldb_queue *queue,
+				 u8 prio)
+{
+	if (domain->started)
+		return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   int slot)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb2_ldb_port_qid_map *map;
+		struct dlb2_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain,
+					  struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (r0.field.count > 0)
+		return false;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = false;
+
+			dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_dir_vpp_v r1;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		unsigned int offs;
+		u32 virt_id;
+
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+	}
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_ldb_vpp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			unsigned int offs;
+			u32 virt_id;
+
+			if (hw->virt_mode == DLB2_VIRT_SRIOV)
+				virt_id = port->id.virt_id;
+			else
+				virt_id = port->id.phys_id;
+
+			offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+			DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+	union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+				    r0.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+				    r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+	union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+			    r0.val);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+			    r1.val);
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+		union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+		union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+				    r1.val);
+
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID_V(idx),
+				    r2.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID2QID(idx),
+				    r3.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID_V(idx),
+				    r1.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID2QID(idx),
+				    r2.val);
+		}
+	}
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+					       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_sn_chk_enbl r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.en = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			int i;
+
+			for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+				if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					    __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		port->enabled = false;
+
+		dlb2_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	union dlb2_sys_dir_pp_v r1;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+			    r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_ldb_pp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_ldb_queue *queue;
+	int i;
+	RTE_SET_USED(iter);
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty ldb queue %d\n",
+				    __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port, iter) {
+			if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb2_ldb_cq_token_count(hw, ldb_port)) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to empty ldb port %d\n",
+					    __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir queue %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb2_dir_cq_token_count(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir port %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						   struct dlb2_ldb_port *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP2PP(offs),
+			    DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP_V(offs),
+			    DLB2_SYS_VF_LDB_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    DLB2_SYS_LDB_PP_V_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_DSBL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_BASE_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID0(port->id.phys_id),
+		    DLB2_LSP_CQ2QID0_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID1(port->id.phys_id),
+		    DLB2_LSP_CQ2QID1_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+		    DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			__dlb2_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+				       struct dlb2_dir_pq_pair *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_DSBL_RST);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_FMT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP2PP(offs),
+			    DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP_V(offs),
+			    DLB2_SYS_VF_DIR_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		__dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+			    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_ITS(queue_id),
+			    DLB2_SYS_LDB_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN(queue_id),
+			    DLB2_CHP_ORD_QID_SN_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+			    DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_V(queue_id),
+			    DLB2_SYS_LDB_QID_V_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_CFG_V(queue_id),
+			    DLB2_SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB2_CSR_WR(hw,
+				    offs[queue->sn_group],
+				    DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX2(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX2_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_ATM_QID2CQIDIX(queue_id, i),
+				    DLB2_ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb2_domain_reset_dir_port_registers(hw, domain);
+
+	dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_dir_pq_pair *tmp_dir_port;
+	struct dlb2_ldb_queue *tmp_ldb_queue;
+	struct dlb2_ldb_port *tmp_ldb_port;
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_list_head *list;
+	int ret, i;
+	RTE_SET_USED(tmp_dir_port);
+	RTE_SET_USED(tmp_ldb_queue);
+	RTE_SET_USED(tmp_ldb_port);
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list = &domain->used_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb2_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		dlb2_list_del(&domain->used_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list = &domain->avail_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		ldb_queue->owned = false;
+
+		dlb2_list_del(&domain->avail_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		list = &domain->used_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB2_QUEUE_UNMAPPED;
+
+			dlb2_list_del(&domain->used_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list = &domain->avail_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			ldb_port->owned = false;
+
+			dlb2_list_del(&domain->avail_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list = &domain->used_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		dlb2_list_del(&domain->used_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list = &domain->avail_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				    domain->hist_list_entry_base,
+				    domain->total_hist_list_entries);
+	if (ret) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			    __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	dlb2_list_del(&rsrcs->used_domains, &domain->func_list);
+	dlb2_list_add(&rsrcs->avail_domains, &domain->func_list);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain,
+					    struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_ldb_port *port;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (!dlb2_list_empty(&domain->used_ldb_ports[i]))
+			break;
+	}
+
+	if (i == DLB2_NUM_COS_DOMAINS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: No configured LDB ports\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb2_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+	 * prior to this in dlb2_domain_drain_mapped_queues().
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings != 0 ||
+		    dlb2_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ *	hardware resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain  == NULL || !domain->configured)
+		return -EINVAL;
+
+	/* Disable VPPs */
+	if (vdev_req) {
+		dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+		dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+	}
+
+	/* Disable CQ interrupts */
+	dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+	dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+	dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb2_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_mapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	dlb2_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb2_domain_disable_dir_cqs(hw, domain);
+
+	/* Disable PPs */
+	dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+	ret = dlb2_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Reset the QID and port state. */
+	dlb2_domain_reset_registers(hw, domain);
+
+	/* Hardware reset complete. Reset the domain's software state */
+	ret = dlb2_domain_reset_software_state(hw, domain);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 210b3186..285ad2c 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -598,3 +598,18 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
 
 	return 0;
 }
+
+int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+			    struct dlb2_create_sched_domain_args *args,
+			    struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_sched_domain(hw, args, resp, NOT_VF_REQ,
+					   PF_ID_ZERO);
+}
+
+int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 7d64309..9310f72 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -108,15 +108,59 @@ dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
 	return 0;
 }
 
+static int
+dlb2_pf_sched_domain_create(struct dlb2_hw_dev *handle,
+			    struct dlb2_create_sched_domain_args *arg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (dlb2_dev->domain_reset_failed) {
+		response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = dlb2_pf_create_sched_domain(&dlb2_dev->hw, arg, &response);
+	if (ret)
+		goto done;
+
+done:
+
+	arg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static void
+dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret;
+
+	dlb2_dev = (struct dlb2_dev *)dlb2->qm_instance.pf_dev;
+	ret = dlb2_pf_reset_domain(&dlb2_dev->hw, dlb2->qm_instance.domain_id);
+	if (ret)
+		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
 	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
 	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_domain_reset = dlb2_pf_domain_reset;
 	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
 	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v3 11/23] event/dlb2: add queue and port default conf

[Timothy McDaniel]

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 6798e66..ef1c000 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -661,12 +661,42 @@ dlb2_eventdev_configure(const struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
+				    uint8_t port_id,
+				    struct rte_event_port_conf *port_conf)
+{
+	RTE_SET_USED(port_id);
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	port_conf->new_event_threshold = dlb2->new_event_limit;
+	port_conf->dequeue_depth = 32;
+	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
+	port_conf->event_port_cfg = 0;
+}
+
+static void
+dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
+				     uint8_t queue_id,
+				     struct rte_event_queue_conf *queue_conf)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(queue_id);
+
+	queue_conf->nb_atomic_flows = 1024;
+	queue_conf->nb_atomic_order_sequences = 64;
+	queue_conf->event_queue_cfg = 0;
+	queue_conf->priority = 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v3 12/23] event/dlb2: add queue setup

[Timothy McDaniel]

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  73 +++--
 drivers/event/dlb2/dlb2.c                  | 312 +++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |  12 +
 drivers/event/dlb2/dlb2_iface.h            |  12 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 464 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  82 +++++
 7 files changed, 926 insertions(+), 39 deletions(-)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 5f6c486..e57e009 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -47,45 +47,40 @@ setup argument and the per-port ``new_event_threshold`` argument apply as
 defined in the eventdev header file. The limit is applied to all enqueues,
 regardless of whether it will consume a directed or load-balanced credit.
 
-Load-balanced and Directed Ports
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
-does not have the same concept, but it has a similar one: ports and queues that
-are singly-linked (i.e. linked to a single queue or port, respectively).
-
-The ``rte_event_dev_info_get()`` function reports the number of available
-event ports and queues (among other things). For the DLB2 PMD, max_event_ports
-and max_event_queues report the number of available load-balanced ports and
-queues, and max_single_link_event_port_queue_pairs reports the number of
-available directed ports and queues.
-
-When a scheduling domain is created in ``rte_event_dev_configure()``, the user
-specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
-control the total number of ports (load-balanced and directed) and the number
-of directed ports. Hence, the number of requested load-balanced ports is
-``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
-specifies the total number of queues (load-balanced and directed). The number
-of directed queues comes from ``nb_single_link_event_port_queues``, since
-directed ports and queues come in pairs.
-
-When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
-whether it should be configured as a directed (the flag is set) or a
-load-balanced (the flag is unset) port. Similarly, the
-``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
-whether it is a directed or load-balanced queue.
-
-Load-balanced ports can only be linked to load-balanced queues, and directed
-ports can only be linked to directed queues. Furthermore, directed ports can
-only be linked to a single directed queue (and vice versa), and that link
-cannot change after the eventdev is started.
-
-The eventdev API does not have a directed scheduling type. To support directed
-traffic, the dlb PMD detects when an event is being sent to a directed queue
-and overrides its scheduling type. Note that the originally selected scheduling
-type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
-will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
-port.
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
 
 Flow ID
 ~~~~~~~
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index ef1c000..8c1e06d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -688,6 +688,317 @@ dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
 	queue_conf->priority = 0;
 }
 
+static int32_t
+dlb2_get_sn_allocation(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_allocation_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_set_sn_allocation(struct dlb2_eventdev *dlb2, int group, int num)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_set_sn_allocation_args cfg;
+	int ret;
+
+	cfg.num = num;
+	cfg.group = group;
+
+	ret = dlb2_iface_set_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: set_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int32_t
+dlb2_get_sn_occupancy(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_occupancy_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_occupancy(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_occupancy ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+/* Query the current sequence number allocations and, if they conflict with the
+ * requested LDB queue configuration, attempt to re-allocate sequence numbers.
+ * This is best-effort; if it fails, the PMD will attempt to configure the
+ * load-balanced queue and return an error.
+ */
+static void
+dlb2_program_sn_allocation(struct dlb2_eventdev *dlb2,
+			   const struct rte_event_queue_conf *queue_conf)
+{
+	int grp_occupancy[DLB2_NUM_SN_GROUPS];
+	int grp_alloc[DLB2_NUM_SN_GROUPS];
+	int i, sequence_numbers;
+
+	sequence_numbers = (int)queue_conf->nb_atomic_order_sequences;
+
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		int total_slots;
+
+		grp_alloc[i] = dlb2_get_sn_allocation(dlb2, i);
+		if (grp_alloc[i] < 0)
+			return;
+
+		total_slots = DLB2_MAX_LDB_SN_ALLOC / grp_alloc[i];
+
+		grp_occupancy[i] = dlb2_get_sn_occupancy(dlb2, i);
+		if (grp_occupancy[i] < 0)
+			return;
+
+		/* DLB has at least one available slot for the requested
+		 * sequence numbers, so no further configuration required.
+		 */
+		if (grp_alloc[i] == sequence_numbers &&
+		    grp_occupancy[i] < total_slots)
+			return;
+	}
+
+	/* None of the sequence number groups are configured for the requested
+	 * sequence numbers, so we have to reconfigure one of them. This is
+	 * only possible if a group is not in use.
+	 */
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		if (grp_occupancy[i] == 0)
+			break;
+	}
+
+	if (i == DLB2_NUM_SN_GROUPS) {
+		printf("[%s()] No groups with %d sequence_numbers are available or have free slots\n",
+		       __func__, sequence_numbers);
+		return;
+	}
+
+	/* Attempt to configure slot i with the requested number of sequence
+	 * numbers. Ignore the return value -- if this fails, the error will be
+	 * caught during subsequent queue configuration.
+	 */
+	dlb2_set_sn_allocation(dlb2, i, sequence_numbers);
+}
+
+static int32_t
+dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 const struct rte_event_queue_conf *evq_conf)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_queue *queue = &ev_queue->qm_queue;
+	struct dlb2_create_ldb_queue_args cfg;
+	int32_t ret;
+	uint32_t qm_qid;
+	int sched_type = -1;
+
+	if (evq_conf == NULL)
+		return -EINVAL;
+
+	if (evq_conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_ALL_TYPES) {
+		if (evq_conf->nb_atomic_order_sequences != 0)
+			sched_type = RTE_SCHED_TYPE_ORDERED;
+		else
+			sched_type = RTE_SCHED_TYPE_PARALLEL;
+	} else
+		sched_type = evq_conf->schedule_type;
+
+	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
+	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
+	cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences;
+
+	if (sched_type != RTE_SCHED_TYPE_ORDERED) {
+		cfg.num_sequence_numbers = 0;
+		cfg.num_qid_inflights = 2048;
+	}
+
+	/* App should set this to the number of hardware flows they want, not
+	 * the overall number of flows they're going to use. E.g. if app is
+	 * using 64 flows and sets compression to 64, best-case they'll get
+	 * 64 unique hashed flows in hardware.
+	 */
+	switch (evq_conf->nb_atomic_flows) {
+	/* Valid DLB2 compression levels */
+	case 64:
+	case 128:
+	case 256:
+	case 512:
+	case (1 * 1024): /* 1K */
+	case (2 * 1024): /* 2K */
+	case (4 * 1024): /* 4K */
+	case (64 * 1024): /* 64K */
+		cfg.lock_id_comp_level = evq_conf->nb_atomic_flows;
+		break;
+	default:
+		/* Invalid compression level */
+		cfg.lock_id_comp_level = 0; /* no compression */
+	}
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create LB event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	qm_qid = cfg.response.id;
+
+	/* Save off queue config for debug, resource lookups, and reconfig */
+	queue->num_qid_inflights = cfg.num_qid_inflights;
+	queue->num_atm_inflights = cfg.num_atomic_inflights;
+
+	queue->sched_type = sched_type;
+	queue->config_state = DLB2_CONFIGURED;
+
+	DLB2_LOG_DBG("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n",
+		     qm_qid,
+		     cfg.num_atomic_inflights,
+		     cfg.num_sequence_numbers,
+		     cfg.num_qid_inflights);
+
+	return qm_qid;
+}
+
+static int
+dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int32_t qm_qid;
+
+	if (queue_conf->nb_atomic_order_sequences)
+		dlb2_program_sn_allocation(dlb2, queue_conf);
+
+	qm_qid = dlb2_hw_create_ldb_queue(dlb2, ev_queue, queue_conf);
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the load-balanced queue\n");
+
+		return qm_qid;
+	}
+
+	dlb2->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int dlb2_num_dir_queues_setup(struct dlb2_eventdev *dlb2)
+{
+	int i, num = 0;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].setup_done &&
+		    dlb2->ev_queues[i].qm_queue.is_directed)
+			num++;
+	}
+
+	return num;
+}
+
+static void
+dlb2_queue_link_teardown(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue)
+{
+	struct dlb2_eventdev_port *ev_port;
+	int i, j;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (!ev_port->link[j].valid ||
+			    ev_port->link[j].queue_id != ev_queue->id)
+				continue;
+
+			ev_port->link[j].valid = false;
+			ev_port->num_links--;
+		}
+	}
+
+	ev_queue->num_links = 0;
+}
+
+static int
+dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
+			  uint8_t ev_qid,
+			  const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_queue *ev_queue;
+	int ret;
+
+	if (queue_conf == NULL)
+		return -EINVAL;
+
+	if (ev_qid >= dlb2->num_queues)
+		return -EINVAL;
+
+	ev_queue = &dlb2->ev_queues[ev_qid];
+
+	ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg &
+		RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+	ev_queue->id = ev_qid;
+	ev_queue->conf = *queue_conf;
+
+	if (!ev_queue->qm_queue.is_directed) {
+		ret = dlb2_eventdev_ldb_queue_setup(dev, ev_queue, queue_conf);
+	} else {
+		/* The directed queue isn't setup until link time, at which
+		 * point we know its directed port ID. Directed queue setup
+		 * will only fail if this queue is already setup or there are
+		 * no directed queues left to configure.
+		 */
+		ret = 0;
+
+		ev_queue->qm_queue.config_state = DLB2_NOT_CONFIGURED;
+
+		if (ev_queue->setup_done ||
+		    dlb2_num_dir_queues_setup(dlb2) == dlb2->num_dir_queues)
+			ret = -EINVAL;
+	}
+
+	/* Tear down pre-existing port->queue links */
+	if (!ret && dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_queue_link_teardown(dlb2, ev_queue);
+
+	if (!ret)
+		ev_queue->setup_done = true;
+
+	return ret;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -696,6 +1007,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a829b9b..4c07574 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -31,3 +31,15 @@ int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				struct dlb2_create_sched_domain_args *args);
 
 void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_ldb_queue_args *cfg);
+
+int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_sn_allocation_args *args);
+
+int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_set_sn_allocation_args *args);
+
+int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_occupancy_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 6663dab..4c88fe0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -30,4 +30,16 @@ extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				 struct dlb2_create_sched_domain_args *args);
 
 extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+extern int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_queue_args *cfg);
+
+extern int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_allocation_args *args);
+
+extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_set_sn_allocation_args *args);
+
+extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				  struct dlb2_get_sn_occupancy_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 63a68a6..e6ea0d7 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3509,3 +3509,467 @@ unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
 
 	return num;
 }
+
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_ldb_queue *queue,
+				     struct dlb2_create_ldb_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+	union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+	union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+	union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+	union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+	union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+	union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+	union dlb2_sys_ldb_qid_its r7 = { {0} };
+	union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+	union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+	union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+	union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+	union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+	union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+	struct dlb2_sn_group *sn_group;
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r3.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	r4.field.limit = args->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+	r5.field.limit = queue->aqed_limit;
+
+	if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+		r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+		    r5.val);
+
+	switch (args->lock_id_comp_level) {
+	case 64:
+		r6.field.compress_code = 1;
+		break;
+	case 128:
+		r6.field.compress_code = 2;
+		break;
+	case 256:
+		r6.field.compress_code = 3;
+		break;
+	case 512:
+		r6.field.compress_code = 4;
+		break;
+	case 1024:
+		r6.field.compress_code = 5;
+		break;
+	case 2048:
+		r6.field.compress_code = 6;
+		break;
+	case 4096:
+		r6.field.compress_code = 7;
+		break;
+	case 0:
+	case 65536:
+		r6.field.compress_code = 0;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+		    r6.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r7.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+		    r7.val);
+
+	r8.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+		    r8.val);
+
+	r9.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+		    r9.val);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	r10.field.qid_fid_limit = 512;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+		    r10.val);
+
+	/* Configure SNs */
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	r11.field.mode = sn_group->mode;
+	r11.field.slot = queue->sn_slot;
+	r11.field.grp  = sn_group->id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+	r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+	r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		r0.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+		r1.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+		r2.field.vqid = queue->id.virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+			    r2.val);
+	}
+
+	r13.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+				  struct dlb2_ldb_queue *queue,
+				  struct dlb2_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb2_sn_group_full(group)) {
+			slot = dlb2_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no sequence number slots available\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+				struct dlb2_hw_domain *domain,
+				struct dlb2_ldb_queue *queue,
+				struct dlb2_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_ldb_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (dlb2_list_empty(&domain->avail_ldb_queues)) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb2_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_ldb_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		    args->num_sequence_numbers);
+	DLB2_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		    args->num_qid_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		    args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+
+	dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			    __func__, __LINE__);
+		return ret;
+	}
+
+	dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_queues, &queue->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_queues, &queue->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+						unsigned int group_id,
+						unsigned long val)
+{
+	DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+	DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+	DLB2_HW_DBG(hw, "\tValue:    %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val)
+{
+	u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+	union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+	struct dlb2_sn_group *group;
+	int mode;
+
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	group = &hw->rsrcs.sn_groups[group_id];
+
+	/*
+	 * Once the first load-balanced queue using an SN group is configured,
+	 * the group cannot be changed.
+	 */
+	if (group->slot_use_bitmap != 0)
+		return -EPERM;
+
+	for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+		if (val == valid_allocations[mode])
+			break;
+
+	if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+		return -EINVAL;
+
+	group->mode = mode;
+	group->sequence_numbers_per_queue = val;
+
+	r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+	r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+	DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+	dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 285ad2c..14bb04b 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -613,3 +613,13 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
 {
 	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_ldb_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 9310f72..a6cd178 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -150,6 +150,84 @@ dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
 		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
 }
 
+static int
+dlb2_pf_ldb_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_ldb_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_ldb_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw_dev *handle,
+			 struct dlb2_get_sn_occupancy_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_number_occupancy(&dlb2_dev->hw,
+						       args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_numbers(&dlb2_dev->hw, args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_set_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_set_group_sequence_numbers(&dlb2_dev->hw, args->group,
+					      args->num);
+
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -161,6 +239,10 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
+	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
+	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
+	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v3 13/23] event/dlb2: add port setup

[Timothy McDaniel]

Configure the load balanded (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  75 +++
 drivers/event/dlb2/dlb2.c                  | 498 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   8 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 922 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  28 +
 drivers/event/dlb2/pf/dlb2_pf.c            | 180 ++++++
 7 files changed, 1719 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index e57e009..8027af7 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -82,6 +82,81 @@ The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
 the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
 load-balanced queues can use the full 16-bit flow ID range.
 
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 8c1e06d..9e8dcce 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -999,6 +999,503 @@ dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
 	return ret;
 }
 
+static int
+dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_cq_pop_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_cq_pop_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL)	{
+		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->consume_qe = qe;
+
+	qe->qe_valid = 0;
+	qe->qe_frag = 0;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
+	 * and so on.
+	 */
+	qe->tokens = 0;	/* set at run time */
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_enqueue_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_enqueue_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL) {
+		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->int_arm_qe = qe;
+
+	/* V2 - INT ARM is CQ_TOKEN + FRAG */
+	qe->qe_valid = 0;
+	qe->qe_frag = 1;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
+{
+	int ret, sz;
+
+	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct dlb2_enqueue_qe);
+
+	qm_port->qe4 = rte_zmalloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
+
+	if (qm_port->qe4 == NULL) {
+		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	ret = dlb2_init_consume_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	return 0;
+
+error_exit:
+
+	dlb2_free_qe_mem(qm_port);
+
+	return ret;
+}
+
+static int
+dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_ldb_port_args cfg = {0};
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_CQ_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	cfg.cq_history_list_size = DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	if (handle->cos_id == DLB2_COS_DEFAULT)
+		cfg.cos_id = 0;
+	else
+		cfg.cos_id = handle->cos_id;
+
+	cfg.cos_strict = 0;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* If there are no directed ports, the kernel driver will ignore this
+	 * port's directed credit settings. Don't use enqueue_depth if it would
+	 * require more directed credits than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2; /* back ptr */
+	/*
+	 * Allocate and init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_ldb_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(qm_port->cfg.ldb));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     qm_port->ldb_credits,
+		     qm_port->dir_credits);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
+
+	return ret;
+}
+
+static void
+dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port)
+{
+	struct dlb2_eventdev_queue *ev_queue;
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (!ev_port->link[i].valid)
+			continue;
+
+		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
+
+		ev_port->link[i].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+}
+
+static int
+dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_port_args cfg = {0};
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (dlb2 == NULL || handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* Directed queues are configured at link time. */
+	cfg.queue_id = -1;
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* Don't use enqueue_depth if it would require more directed credits
+	 * than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2;  /* back ptr */
+
+	/*
+	 * Init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE to be aligned
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_dir_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = cfg.cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(qm_port->cfg.dir));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     dir_credit_high_watermark,
+		     ldb_credit_high_watermark);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_setup(struct rte_eventdev *dev,
+			 uint8_t ev_port_id,
+			 const struct rte_event_port_conf *port_conf)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_eventdev_port *ev_port;
+	int ret;
+
+	if (dev == NULL || port_conf == NULL) {
+		DLB2_LOG_ERR("Null parameter\n");
+		return -EINVAL;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
+		return -EINVAL;
+
+	if (port_conf->dequeue_depth >
+		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
+	    port_conf->enqueue_depth >
+		evdev_dlb2_default_info.max_event_port_enqueue_depth)
+		return -EINVAL;
+
+	ev_port = &dlb2->ev_ports[ev_port_id];
+	/* configured? */
+	if (ev_port->setup_done) {
+		DLB2_LOG_ERR("evport %d is already configured\n", ev_port_id);
+		return -EINVAL;
+	}
+
+	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
+		RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (!ev_port->qm_port.is_directed) {
+		ret = dlb2_hw_create_ldb_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the lB port ve portId=%d\n",
+				     ev_port_id);
+
+			return ret;
+		}
+	} else {
+		ret = dlb2_hw_create_dir_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the DIR port\n");
+			return ret;
+		}
+	}
+
+	/* Save off port config for reconfig */
+	ev_port->conf = *port_conf;
+
+	ev_port->id = ev_port_id;
+	ev_port->enq_configured = true;
+	ev_port->setup_done = true;
+	ev_port->inflight_max = port_conf->new_event_threshold;
+	ev_port->implicit_release = !(port_conf->event_port_cfg &
+		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
+	ev_port->outstanding_releases = 0;
+	ev_port->inflight_credits = 0;
+	ev_port->credit_update_quanta = RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
+	ev_port->dlb2 = dlb2; /* reverse link */
+
+	/* Tear down pre-existing port->queue links */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
+
+	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1009,6 +1506,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
+		.port_setup       = dlb2_eventdev_port_setup,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 4c07574..01818c7 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -43,3 +43,11 @@ int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				   struct dlb2_get_sn_occupancy_args *args);
+
+int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_dir_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4c88fe0..a6b923a 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -42,4 +42,12 @@ extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				  struct dlb2_get_sn_occupancy_args *args);
+
+extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_ldb_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_dir_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index e6ea0d7..a633d3e 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3973,3 +3973,925 @@ int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_ldb_port *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_pp2vas r0 = { {0} };
+	union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+		union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_ldb_port *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_ldb_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+	union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+	union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+	union dlb2_chp_hist_list_lim r5 = { {0} };
+	union dlb2_chp_hist_list_base r6 = { {0} };
+	union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+	union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+	union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+	union dlb2_sys_ldb_cq_at r10 = { {0} };
+	union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+	union dlb2_chp_ldb_cq2vas r12 = { {0} };
+	union dlb2_lsp_cq2priov r13 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r14.field.token_count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    r14.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	r5.field.limit = port->hist_list_entry_limit - 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+	r6.field.base = port->hist_list_entry_base;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	r7.field.limit = args->cq_history_list_size;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+	r8.field.push_ptr = r6.field.base;
+	r8.field.generation = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    r8.val);
+
+	r9.field.pop_ptr = r6.field.base;
+	r9.field.generation = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+	/* Disable the port's QID mappings */
+	r13.field.v = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+	return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_ldb_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb2_ldb_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+	if (ret < 0)
+		return ret;
+
+	dlb2_ldb_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_ldb_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		    args->cq_history_list_size);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+	DLB2_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		    args->cos_strict);
+}
+
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_ldb_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+		resp->status = DLB2_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		if (dlb2_list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	} else {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			if (!dlb2_list_empty(&domain->avail_ldb_ports[i]))
+				break;
+		}
+
+		if (i == DLB2_NUM_COS_DOMAINS) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ *	its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	int ret, cos_id, i;
+
+	dlb2_log_create_ldb_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->cos_strict) {
+		cos_id = args->cos_id;
+
+		port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+					  typeof(*port));
+	} else {
+		int idx;
+
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+			port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+						  typeof(*port));
+			if (port)
+				break;
+		}
+
+		cos_id = idx;
+	}
+
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (port->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_ldb_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_ports[cos_id], &port->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_ports[cos_id], &port->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_dir_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+}
+
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+			    bool vdev_req,
+			    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+
+	return NULL;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_dir_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the queue is already configured, validate
+	 * the queue ID, its domain, and whether the queue is configured.
+	 */
+	if (args->queue_id != -1) {
+		struct dlb2_dir_pq_pair *queue;
+
+		queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+						    vdev_req,
+						    domain);
+
+		if (queue == NULL || queue->domain_id.phys_id !=
+				domain->id.phys_id ||
+				!queue->queue_configured) {
+			resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the port's queue is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->queue_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_dir_pq_pair *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_dir_pp2vas r0 = { {0} };
+	union dlb2_sys_dir_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+		union dlb2_sys_dir_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_dir_pq_pair *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_dir_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+	union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+	union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+	union dlb2_sys_dir_cq_fmt r9 = { {0} };
+	union dlb2_sys_dir_cq_at r10 = { {0} };
+	union dlb2_sys_dir_cq_pasid r11 = { {0} };
+	union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r13.field.count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    r13.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.disable_wb_opt = 0;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	r9.field.keep_pf_ppid = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+	return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_dir_pq_pair *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_dir_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb2_dir_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+
+	if (ret < 0)
+		return ret;
+
+	dlb2_dir_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ *	and queue. The port/queue pair have the same ID and name.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *port;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (args->queue_id != -1)
+		port = dlb2_get_domain_used_dir_pq(args->queue_id,
+						   vdev_req,
+						   domain);
+	else
+		port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					  typeof(*port));
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_dir_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs, &port->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs, &port->domain_list);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 14bb04b..2a089f6 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -623,3 +623,31 @@ dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_ldb_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
+
+int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_dir_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a6cd178..a9a7fad 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -228,6 +228,184 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static void *
+dlb2_alloc_coherent_aligned(const struct rte_memzone **mz, uintptr_t *phys,
+			    size_t size, int align)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t core_id = rte_lcore_id();
+	unsigned int socket_id;
+
+	snprintf(mz_name, sizeof(mz_name) - 1, "event_dlb2_pf_%lx",
+		 (unsigned long)rte_get_timer_cycles());
+	if (core_id == (unsigned int)LCORE_ID_ANY)
+		core_id = rte_get_master_lcore();
+	socket_id = rte_lcore_to_socket_id(core_id);
+	*mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
+					 RTE_MEMZONE_IOVA_CONTIG, align);
+	if (*mz == NULL) {
+		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu bytes - %s\n",
+			     size, rte_strerror(rte_errno));
+		*phys = 0;
+		return NULL;
+	}
+	*phys = (*mz)->iova;
+	return (*mz)->addr;
+}
+
+static int
+dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_ldb_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret, cq_alloc_depth;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = false;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	cq_alloc_depth = RTE_MAX(cfg->cq_depth, DLB2_MIN_HARDWARE_CQ_DEPTH);
+	alloc_sz = cq_alloc_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].cq_base = (void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+	return ret;
+}
+
+static int
+dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_dir_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = true;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	alloc_sz = cfg->cq_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
+		(void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -240,6 +418,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v3 14/23] event/dlb2: add port link

[Timothy McDaniel]

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 302 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 633 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
 6 files changed, 1007 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 9e8dcce..c6593c7 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1496,6 +1496,307 @@ dlb2_eventdev_port_setup(struct rte_eventdev *dev,
 	return 0;
 }
 
+static int16_t
+dlb2_hw_map_ldb_qid_to_port(struct dlb2_hw_dev *handle,
+			    uint32_t qm_port_id,
+			    uint16_t qm_qid,
+			    uint8_t priority)
+{
+	struct dlb2_map_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	/* Build message */
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+	cfg.priority = EV_TO_DLB2_PRIO(priority);
+
+	ret = dlb2_iface_map_qid(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: map qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		DLB2_LOG_ERR("dlb2: grp=%d, qm_port=%d, qm_qid=%d prio=%d\n",
+			     handle->domain_id, cfg.port_id,
+			     cfg.qid,
+			     cfg.priority);
+	} else {
+		DLB2_LOG_DBG("dlb2: mapped queue %d to qm_port %d\n",
+			     qm_qid, qm_port_id);
+	}
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port,
+			  struct dlb2_eventdev_queue *ev_queue,
+			  uint8_t priority)
+{
+	int first_avail = -1;
+	int ret, i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid) {
+			if (ev_port->link[i].queue_id == ev_queue->id &&
+			    ev_port->link[i].priority == priority) {
+				if (ev_port->link[i].mapped)
+					return 0; /* already mapped */
+				first_avail = i;
+			}
+		} else if (first_avail == -1)
+			first_avail = i;
+	}
+	if (first_avail == -1) {
+		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
+			     ev_port->qm_port.id);
+		return -EINVAL;
+	}
+
+	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
+					  ev_port->qm_port.id,
+					  ev_queue->qm_queue.id,
+					  priority);
+
+	if (!ret)
+		ev_port->link[first_avail].mapped = true;
+
+	return ret;
+}
+
+static int32_t
+dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 int32_t qm_port_id)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_queue_args cfg;
+	int32_t ret;
+
+	/* The directed port is always configured before its queue */
+	cfg.port_id = qm_port_id;
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_dir_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	int32_t qm_qid;
+
+	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port->qm_port.id);
+
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the DIR queue\n");
+		return qm_qid;
+	}
+
+	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int
+dlb2_do_port_link(struct rte_eventdev *dev,
+		  struct dlb2_eventdev_queue *ev_queue,
+		  struct dlb2_eventdev_port *ev_port,
+		  uint8_t prio)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int err;
+
+	/* Don't link until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	if (ev_queue->qm_queue.is_directed)
+		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue, ev_port);
+	else
+		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue, prio);
+
+	if (err) {
+		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
+			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
+			     ev_queue->id, ev_port->id);
+
+		rte_errno = err;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
+			uint8_t queue_id,
+			bool link_exists,
+			int index)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	bool port_is_dir, queue_is_dir;
+
+	if (queue_id > dlb2->num_queues) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	ev_queue = &dlb2->ev_queues[queue_id];
+
+	if (!ev_queue->setup_done &&
+	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	port_is_dir = ev_port->qm_port.is_directed;
+	queue_is_dir = ev_queue->qm_queue.is_directed;
+
+	if (port_is_dir != queue_is_dir) {
+		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
+			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
+			     port_is_dir ? "DIR" : "LDB", ev_port->id);
+
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if there is space for the requested link */
+	if (!link_exists && index == -1) {
+		DLB2_LOG_ERR("no space for new link\n");
+		rte_errno = -ENOSPC;
+		return -1;
+	}
+
+	/* Check if the directed port is already linked */
+	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if the directed queue is already linked */
+	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
+			     ev_queue->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
+			const uint8_t queues[], const uint8_t priorities[],
+			uint16_t nb_links)
+
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i, j;
+
+	RTE_SET_USED(dev);
+
+	if (ev_port == NULL) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!ev_port->setup_done &&
+	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
+	 * queues pointer.
+	 */
+	if (nb_links == 0) {
+		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
+		     nb_links,
+		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
+		     ev_port->id);
+
+	for (i = 0; i < nb_links; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		uint8_t queue_id, prio;
+		bool found = false;
+		int index = -1;
+
+		queue_id = queues[i];
+		prio = priorities[i];
+
+		/* Check if the link already exists. */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].valid) {
+				if (ev_port->link[j].queue_id == queue_id) {
+					found = true;
+					index = j;
+					break;
+				}
+			} else if (index == -1) {
+				index = j;
+			}
+
+		/* could not link */
+		if (index == -1)
+			break;
+
+		/* Check if already linked at the requested priority */
+		if (found && ev_port->link[j].priority == prio)
+			continue;
+
+		if (dlb2_validate_port_link(ev_port, queue_id, found, index))
+			break; /* return index of offending queue */
+
+		ev_queue = &dlb2->ev_queues[queue_id];
+
+		if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+			break; /* return index of offending queue */
+
+		ev_queue->num_links++;
+
+		ev_port->link[index].queue_id = queue_id;
+		ev_port->link[index].priority = prio;
+		ev_port->link[index].valid = true;
+		/* Entry already exists?  If so, then must be prio change */
+		if (!found)
+			ev_port->num_links++;
+	}
+	return i;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1507,6 +1808,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_link        = dlb2_eventdev_port_link,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 01818c7..7d1ba73 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -51,3 +51,9 @@ int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 				  struct dlb2_create_dir_port_args *cfg,
 				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_dir_queue_args *cfg);
+
+int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+			  struct dlb2_map_qid_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index a6b923a..4eddd14 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -50,4 +50,10 @@ extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 struct dlb2_create_dir_port_args *cfg,
 					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+					struct dlb2_create_dir_queue_args *cfg);
+
+extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+				 struct dlb2_map_qid_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index a633d3e..14ad262 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -4895,3 +4895,636 @@ int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_dir_pq_pair *queue,
+				     struct dlb2_create_dir_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_dir_vasqid_v r0 = { {0} };
+	union dlb2_sys_dir_qid_its r1 = { {0} };
+	union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+	union dlb2_sys_dir_qid_v r5 = { {0} };
+
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r0.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r1.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+		    r1.val);
+
+	r2.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+		    r2.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		r3.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+		r4.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+	}
+
+	r5.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+	queue->queue_configured = true;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_dir_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_dir_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = dlb2_get_domain_used_dir_pq(args->port_id,
+						   vdev_req,
+						   domain);
+
+		if (port == NULL || port->domain_id.phys_id !=
+				domain->id.phys_id || !port->port_configured) {
+			resp->status = DLB2_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the queue's port is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->port_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->port_id != -1)
+		queue = dlb2_get_domain_used_dir_pq(args->port_id,
+						    vdev_req,
+						    domain);
+	else
+		queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					   typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &queue->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs,
+			      &queue->domain_list);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot,
+					      struct dlb2_map_qid_args *args)
+{
+	union dlb2_lsp_cq2priov r0;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+	r0.field.v |= 1 << slot;
+	r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+	dlb2_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+					      struct dlb2_ldb_queue *queue,
+					      struct dlb2_cmd_response *resp)
+{
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB2_QUEUE_UNMAPPED;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+			  bool vdev_req,
+			  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter)
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+			      bool vdev_req,
+			      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+
+		DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+	}
+
+	return NULL;
+}
+
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+				    u32 domain_id,
+				    struct dlb2_map_qid_args *args,
+				    struct dlb2_cmd_response *resp,
+				    bool vdev_req,
+				    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	struct dlb2_ldb_queue *queue;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB2_QID_PRIORITIES) {
+		resp->status = DLB2_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_map_qid_args *args,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	DLB2_HW_DBG(hw, "\tPriority:  %d\n",
+		    args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	int ret, i, id;
+	u8 prio;
+
+	dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_map_qid_args(hw,
+				       domain_id,
+				       args,
+				       resp,
+				       vdev_req,
+				       vdev_id);
+	if (ret)
+		return ret;
+
+	prio = args->priority;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	if (port->num_pending_removals)
+		dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].pending_priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+		if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb2_qid_map_state st;
+
+			if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+				DLB2_HW_ERR(hw,
+					    "[%s():%d] Internal error: port slot tracking failed\n",
+					    __func__, __LINE__);
+				return -EFAULT;
+			}
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb2_port_slot_state_transition(hw, port, queue,
+							      i, st);
+			if (ret)
+				return ret;
+
+			DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 2a089f6..a010e25 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -651,3 +651,13 @@ dlb2_pf_create_dir_port(struct dlb2_hw *hw,
 				       NOT_VF_REQ,
 				       PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_dir_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a9a7fad..0f54b8d 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -406,6 +406,54 @@ dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_dir_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_dir_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_dir_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
+		struct dlb2_map_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_map_qid(&dlb2_dev->hw,
+			      handle->domain_id,
+			      cfg,
+			      &response,
+			      false,
+			      0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -419,7 +467,9 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
 	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
+	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v3 15/23] event/dlb2: add port unlink and port unlinks in progress

[Timothy McDaniel]

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 163 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 283 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  51 ++++++
 5 files changed, 509 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index c6593c7..4ef7b6d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1797,6 +1797,166 @@ dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
 	return i;
 }
 
+static int16_t
+dlb2_hw_unmap_ldb_qid_from_port(struct dlb2_hw_dev *handle,
+				uint32_t qm_port_id,
+				uint16_t qm_qid)
+{
+	struct dlb2_unmap_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+
+	ret = dlb2_iface_unmap_qid(handle, &cfg);
+	if (ret < 0)
+		DLB2_LOG_ERR("dlb2: unmap qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_detach_ldb(struct dlb2_eventdev *dlb2,
+			    struct dlb2_eventdev_port *ev_port,
+			    struct dlb2_eventdev_queue *ev_queue)
+{
+	int ret, i;
+
+	/* Don't unlink until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid &&
+		    ev_port->link[i].queue_id == ev_queue->id)
+			break; /* found */
+	}
+
+	/* This is expected with eventdev API!
+	 * It blindly attemmpts to unmap all queues.
+	 */
+	if (i == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_LOG_DBG("dlb2: ignoring LB QID %d not mapped for qm_port %d.\n",
+			     ev_queue->qm_queue.id,
+			     ev_port->qm_port.id);
+		return 0;
+	}
+
+	ret = dlb2_hw_unmap_ldb_qid_from_port(&dlb2->qm_instance,
+					      ev_port->qm_port.id,
+					      ev_queue->qm_queue.id);
+	if (!ret)
+		ev_port->link[i].mapped = false;
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
+			  uint8_t queues[], uint16_t nb_unlinks)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (queues == NULL || nb_unlinks == 0) {
+		DLB2_LOG_DBG("dlb2: queues is NULL or nb_unlinks is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	/* FIXME: How to handle unlink on a directed port? */
+	if (ev_port->qm_port.is_directed) {
+		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
+			     ev_port->id);
+		rte_errno = 0;
+		return nb_unlinks; /* as if success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	for (i = 0; i < nb_unlinks; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		int ret, j;
+
+		if (queues[i] >= dlb2->num_queues) {
+			DLB2_LOG_ERR("dlb2: invalid queue id %d\n", queues[i]);
+			rte_errno = -EINVAL;
+			return i; /* return index of offending queue */
+		}
+
+		ev_queue = &dlb2->ev_queues[queues[i]];
+
+		/* Does a link exist? */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].queue_id == queues[i] &&
+			    ev_port->link[j].valid)
+				break;
+
+		if (j == DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+			continue;
+
+		ret = dlb2_event_queue_detach_ldb(dlb2, ev_port, ev_queue);
+		if (ret) {
+			DLB2_LOG_ERR("unlink err=%d for port %d queue %d\n",
+				     ret, ev_port->id, queues[i]);
+			rte_errno = -ENOENT;
+			return i; /* return index of offending queue */
+		}
+
+		ev_port->link[j].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+
+	return nb_unlinks;
+}
+
+static int
+dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
+				       void *event_port)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	struct dlb2_pending_port_unmaps_args cfg;
+	int ret;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	cfg.port_id = ev_port->qm_port.id;
+	dlb2 = ev_port->dlb2;
+	handle = &dlb2->qm_instance;
+	ret = dlb2_iface_pending_port_unmaps(handle, &cfg);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: num_unlinks_in_progress ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1809,6 +1969,9 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
 		.port_link        = dlb2_eventdev_port_link,
+		.port_unlink      = dlb2_eventdev_port_unlink,
+		.port_unlinks_in_progress =
+				    dlb2_eventdev_port_unlinks_in_progress,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 7d1ba73..bd241f3 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -57,3 +57,9 @@ int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 			  struct dlb2_map_qid_args *cfg);
+
+int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+			    struct dlb2_unmap_qid_args *cfg);
+
+int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4eddd14..e892036 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -56,4 +56,10 @@ extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 				 struct dlb2_map_qid_args *cfg);
+
+extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+				   struct dlb2_unmap_qid_args *cfg);
+
+extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 14ad262..4a3d96d 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5528,3 +5528,286 @@ int dlb2_hw_map_qid(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_unmap_qid_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+		DLB2_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			    hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+				      u32 domain_id,
+				      struct dlb2_unmap_qid_args *args,
+				      struct dlb2_cmd_response *resp,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	struct dlb2_ldb_port *port;
+	int slot;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			    __func__, args->qid);
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		return 0;
+
+	resp->status = DLB2_ST_INVALID_QID;
+	return -EINVAL;
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	bool unmap_complete;
+	int i, ret, id;
+
+	dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_unmap_qid_args(hw,
+					 domain_id,
+					 args,
+					 resp,
+					 vdev_req,
+					 vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+		st = DLB2_QUEUE_UNMAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		st = DLB2_QUEUE_UNMAP_IN_PROG;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB2_QUEUE_MAPPED;
+	if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: no available CQ slots\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asynchronous procedure. It requires
+	 * stopping the DLB2 from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb2_ldb_port_cq_disable(hw, port);
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete && !os_worker_active(hw))
+		os_schedule_work(hw);
+
+unmap_qid_done:
+	resp->status = 0;
+
+	return 0;
+}
+
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+				  struct dlb2_pending_port_unmaps_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+
+	dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 0f54b8d..1ea5fa5 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -454,6 +454,55 @@ dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw_dev *handle,
+		  struct dlb2_unmap_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_unmap_qid(&dlb2_dev->hw,
+				handle->domain_id,
+				cfg,
+				&response,
+				false,
+				0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
+			    struct dlb2_pending_port_unmaps_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_pending_port_unmaps(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -470,6 +519,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
+	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v3 16/23] event/dlb2: add eventdev start

[Timothy McDaniel]

Add support for the eventdev start entry point.
We delay initializing some resources until
eventdev start, since the number of linked queues can be
used to determine if we are dealing with a ldb or dir resource.
If this is a device restart, then the previous configuration
will be reapplied.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 133 +++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   3 +
 drivers/event/dlb2/dlb2_iface.h            |   3 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 123 ++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +++
 drivers/event/dlb2/pf/dlb2_pf.c            |  25 ++++++
 6 files changed, 297 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 4ef7b6d..2d233bf 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1957,6 +1957,138 @@ dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
 	return cfg.response.id;
 }
 
+static int
+dlb2_eventdev_reapply_configuration(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret, i;
+
+	/* If an event queue or port was previously configured, but hasn't been
+	 * reconfigured, reapply its original configuration.
+	 */
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+
+		ev_queue = &dlb2->ev_queues[i];
+
+		if (ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_queue_setup(dev, i, &ev_queue->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure queue %d", i);
+			return ret;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+
+		if (ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_port_setup(dev, i, &ev_port->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure ev_port %d",
+				     i);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_apply_port_links(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int i;
+
+	/* Perform requested port->queue links */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+		int j;
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			struct dlb2_eventdev_queue *ev_queue;
+			uint8_t prio, queue_id;
+
+			if (!ev_port->link[j].valid)
+				continue;
+
+			prio = ev_port->link[j].priority;
+			queue_id = ev_port->link[j].queue_id;
+
+			if (dlb2_validate_port_link(ev_port, queue_id, true, j))
+				return -EINVAL;
+
+			ev_queue = &dlb2->ev_queues[queue_id];
+
+			if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_start(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_start_domain_args cfg;
+	int ret, i;
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_ERR("bad state %d for dev_start\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return -EINVAL;
+	}
+	dlb2->run_state = DLB2_RUN_STATE_STARTING;
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	/* If the device was configured more than once, some event ports and/or
+	 * queues may need to be reconfigured.
+	 */
+	ret = dlb2_eventdev_reapply_configuration(dev);
+	if (ret)
+		return ret;
+
+	/* The DLB PMD delays port links until the device is started. */
+	ret = dlb2_eventdev_apply_port_links(dev);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		if (!dlb2->ev_ports[i].setup_done) {
+			DLB2_LOG_ERR("dlb2: port %d not setup", i);
+			return -ESTALE;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0) {
+			DLB2_LOG_ERR("dlb2: queue %d is not linked", i);
+			return -ENOLINK;
+		}
+	}
+
+	ret = dlb2_iface_sched_domain_start(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: sched_domain_start ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STARTED;
+	DLB2_LOG_DBG("dlb2: sched_domain_start completed OK\n");
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1964,6 +2096,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.dev_start        = dlb2_eventdev_start,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index bd241f3..a86191d 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -63,3 +63,6 @@ int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				     struct dlb2_start_domain_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index e892036..7ead374 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -62,4 +62,7 @@ extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				struct dlb2_start_domain_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 4a3d96d..c14a2f3 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5811,3 +5811,126 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 struct dlb2_cmd_response *resp,
+					 bool vdev_req,
+					 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_start_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 start domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_hw_start_domain() - Lock the domain configuration
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @arg: User-provided arguments (unused, here for ioctl callback template).
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int
+dlb2_hw_start_domain(struct dlb2_hw *hw,
+		     u32 domain_id,
+		     __attribute((unused)) struct dlb2_start_domain_args *arg,
+		     struct dlb2_cmd_response *resp,
+		     bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_queue;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+	RTE_SET_USED(arg);
+	RTE_SET_USED(iter);
+
+	dlb2_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb2_verify_start_domain_args(hw,
+					    domain_id,
+					    resp,
+					    vdev_req,
+					    vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB2 will drop all
+	 * incoming traffic to those queues.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, ldb_queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r0.val);
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+	}
+
+	dlb2_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index a010e25..aee8336 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -661,3 +661,13 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_start_domain(struct dlb2_hw *hw,
+		     u32 id,
+		     struct dlb2_start_domain_args *args,
+		     struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_start_domain(hw, id, args, resp, NOT_VF_REQ,
+				    PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 1ea5fa5..3bcf3b9 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -503,6 +503,30 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
 
 	return ret;
 }
+
+static int
+dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
+			   struct dlb2_start_domain_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_start_domain(&dlb2_dev->hw,
+				   handle->domain_id,
+				   cfg,
+				   &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -520,6 +544,7 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v3 17/23] event/dlb2: add enqueue and its burst variants

[Timothy McDaniel]

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst | 118 +++++++++
 drivers/event/dlb2/dlb2.c     | 578 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 696 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 8027af7..b9607b7 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -163,6 +163,124 @@ Flow ID
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Hardware Credits
+~~~~~~~~~~~~~~~~
+
+DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
+event storage, with each unit of storage represented by a credit. A port spends
+a credit to enqueue an event, and hardware refills the ports with credits as the
+events are scheduled to ports. Refills come from credit pools, and each port is
+a member of a load-balanced credit pool and a directed credit pool. The
+load-balanced credits are used to enqueue to load-balanced queues, and directed
+credits are used for directed queues.
+
+A DLB2 eventdev contains one load-balanced and one directed credit pool. These
+pools' sizes are controlled by the nb_events_limit field in struct
+rte_event_dev_config. The load-balanced pool is sized to contain
+nb_events_limit credits, and the directed pool is sized to contain
+nb_events_limit/4 credits. The directed pool size can be overridden with the
+num_dir_credits vdev argument, like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,num_dir_credits=<value>
+
+This can be used if the default allocation is too low or too high for the
+specific application needs. The PMD also supports a vdev arg that limits the
+max_num_events reported by rte_event_dev_info_get():
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,max_num_events=<value>
+
+By default, max_num_events is reported as the total available load-balanced
+credits. If multiple DLB2-based applications are being used, it may be desirable
+to control how many load-balanced credits each application uses, particularly
+when application(s) are written to configure nb_events_limit equal to the
+reported max_num_events.
+
+Each port is a member of both credit pools. A port's credit allocation is
+defined by its low watermark, high watermark, and refill quanta. These three
+parameters are calculated by the dlb PMD like so:
+
+- The load-balanced high watermark is set to the port's enqueue_depth.
+  The directed high watermark is set to the minimum of the enqueue_depth and
+  the directed pool size divided by the total number of ports.
+- The refill quanta is set to half the high watermark.
+- The low watermark is set to the minimum of 16 and the refill quanta.
+
+When the eventdev is started, each port is pre-allocated a high watermark's
+worth of credits. For example, if an eventdev contains four ports with enqueue
+depths of 32 and a load-balanced credit pool size of 4096, each port will start
+with 32 load-balanced credits, and there will be 3968 credits available to
+replenish the ports. Thus, a single port is not capable of enqueueing up to the
+nb_events_limit (without any events being dequeued), since the other ports are
+retaining their initial credit allocation; in short, all ports must enqueue in
+order to reach the limit.
+
+If a port attempts to enqueue and has no credits available, the enqueue
+operation will fail and the application must retry the enqueue. Credits are
+replenished asynchronously by the DLB2 hardware.
+
+Software Credits
+~~~~~~~~~~~~~~~~
+
+The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
+credit scheme on top of the hardware credit scheme in order to comply with
+the per-port backpressure described in the eventdev API.
+
+The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
+credit when it enqueues to a queue, and credits are later replenished after the
+events are dequeued and released.
+
+In the software credit scheme, a credit is consumed when a new (.op =
+RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
+replenished when the event is released from the system (either explicitly with
+RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
+
+In this model, an event is "in the system" from its first enqueue into eventdev
+until it is last dequeued. If the event goes through multiple event queues, it
+is still considered "in the system" while a worker thread is processing it.
+
+A port will fail to enqueue if the number of events in the system exceeds its
+``new_event_threshold`` (specified at port setup time). A port will also fail
+to enqueue if it lacks enough hardware credits to enqueue; load-balanced
+credits are used to enqueue to a load-balanced queue, and directed credits are
+used to enqueue to a directed queue.
+
+The out-of-credit situations are typically transient, and an eventdev
+application using the DLB2 ought to retry its enqueues if they fail.
+If enqueue fails, DLB2 PMD sets rte_errno as follows:
+
+- -ENOSPC: Credit exhaustion (either hardware or software)
+- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
+
+Depending on the pipeline the application has constructed, it's possible to
+enter a credit deadlock scenario wherein the worker thread lacks the credit
+to enqueue an event, and it must dequeue an event before it can recover the
+credit. If the worker thread retries its enqueue indefinitely, it will not
+make forward progress. Such deadlock is possible if the application has event
+"loops", in which an event in dequeued from queue A and later enqueued back to
+queue A.
+
+Due to this, workers should stop retrying after a time, release the events it
+is attempting to enqueue, and dequeue more events. It is important that the
+worker release the events and don't simply set them aside to retry the enqueue
+again later, because the port has limited history list size (by default, twice
+the port's dequeue_depth).
+
+Priority
+~~~~~~~~
+
+The DLB2 supports event priority and per-port queue service priority, as
+described in the eventdev header file. The DLB2 does not support 'global' event
+queue priority established at queue creation time.
+
+DLB2 supports 8 event and queue service priority levels. For both priority
+types, the PMD uses the upper three bits of the priority field to determine the
+DLB2 priority, discarding the 5 least significant bits. The 5 least significant
+event priority bits are not preserved when an event is enqueued.
+
 Reconfiguration
 ~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2d233bf..90a5c9d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2089,6 +2089,578 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
 	return 0;
 }
 
+static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
+	{
+		/* Load-balanced cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
+	},
+	{
+		/* Directed cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
+	},
+};
+
+static inline uint32_t
+dlb2_port_credits_get(struct dlb2_port *qm_port,
+		      enum dlb2_hw_queue_types type)
+{
+	uint32_t credits = *qm_port->credit_pool[type];
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	if (unlikely(credits < batch_size))
+		batch_size = credits;
+
+	if (likely(credits &&
+		   __atomic_compare_exchange_n(
+			qm_port->credit_pool[type],
+			&credits, credits - batch_size, false,
+			__ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
+		return batch_size;
+	else
+		return 0;
+}
+
+static inline void
+dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port)
+{
+	uint16_t quanta = ev_port->credit_update_quanta;
+
+	if (ev_port->inflight_credits >= quanta * 2) {
+		/* Replenish credits, saving one quanta for enqueues */
+		uint16_t val = ev_port->inflight_credits - quanta;
+
+		__atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits -= val;
+	}
+}
+
+static inline int
+dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
+						__ATOMIC_SEQ_CST);
+	const int num = 1;
+
+	if (unlikely(ev_port->inflight_max < sw_inflights)) {
+		DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
+		rte_errno = -ENOSPC;
+		return 1;
+	}
+
+	if (ev_port->inflight_credits < num) {
+		/* check if event enqueue brings ev_port over max threshold */
+		uint32_t credit_update_quanta = ev_port->credit_update_quanta;
+
+		if (sw_inflights + credit_update_quanta >
+				dlb2->new_event_limit) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_new_event_limit,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+
+		__atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
+				   __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits += (credit_update_quanta);
+
+		if (ev_port->inflight_credits < num) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_inflight_credits,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_ldb_credits == 0)) {
+		qm_port->cached_ldb_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_LDB_QUEUE);
+		if (unlikely(qm_port->cached_ldb_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
+			1);
+			DLB2_LOG_DBG("ldb credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_dir_credits == 0)) {
+		qm_port->cached_dir_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_DIR_QUEUE);
+		if (unlikely(qm_port->cached_dir_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
+			1);
+			DLB2_LOG_DBG("dir credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static __rte_always_inline void
+dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
+	      struct process_local_port_data *port_data)
+{
+	dlb2_movdir64b(port_data->pp_addr, qe4);
+}
+
+static inline int
+dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_cq_pop_qe *qe;
+
+	RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
+
+	qe = qm_port->consume_qe;
+
+	qe->tokens = num - 1;
+
+	/* No store fence needed since no pointer is being sent, and CQ token
+	 * pops can be safely reordered with other HCWs.
+	 */
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	dlb2_movntdq_single(port_data->pp_addr, qe);
+
+	DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
+
+	qm_port->owed_tokens = 0;
+
+	return 0;
+}
+
+static inline void
+dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
+		   bool do_sfence,
+		   struct process_local_port_data *port_data)
+{
+	/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
+	 * application writes complete before enqueueing the QE.
+	 */
+	if (do_sfence)
+		rte_wmb();
+
+	dlb2_pp_write(qm_port->qe4, port_data);
+}
+
+static inline void
+dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
+{
+	struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
+	int num = qm_port->owed_tokens;
+
+	qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
+	qe[idx].tokens = num - 1;
+
+	qm_port->owed_tokens = 0;
+}
+
+static inline void
+dlb2_event_build_hcws(struct dlb2_port *qm_port,
+		      const struct rte_event ev[],
+		      int num,
+		      uint8_t *sched_type,
+		      uint8_t *queue_id)
+{
+	struct dlb2_enqueue_qe *qe;
+	uint16_t sched_word[4];
+	__m128i sse_qe[2];
+	int i;
+
+	qe = qm_port->qe4;
+
+	sse_qe[0] = _mm_setzero_si128();
+	sse_qe[1] = _mm_setzero_si128();
+
+	switch (num) {
+	case 4:
+		/* Construct the metadata portion of two HCWs in one 128b SSE
+		 * register. HCW metadata is constructed in the SSE registers
+		 * like so:
+		 * sse_qe[0][63:0]:   qe[0]'s metadata
+		 * sse_qe[0][127:64]: qe[1]'s metadata
+		 * sse_qe[1][63:0]:   qe[2]'s metadata
+		 * sse_qe[1][127:64]: qe[3]'s metadata
+		 */
+
+		/* Convert the event operation into a command byte and store it
+		 * in the metadata:
+		 * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
+		 * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
+		 * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
+		 * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
+		 */
+#define DLB2_QE_CMD_BYTE 7
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[0].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[1].op],
+				DLB2_QE_CMD_BYTE + 8);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[2].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[3].op],
+				DLB2_QE_CMD_BYTE + 8);
+
+		/* Store priority, scheduling type, and queue ID in the sched
+		 * word array because these values are re-used when the
+		 * destination is a directed queue.
+		 */
+		sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
+				sched_type[0] << 8 |
+				queue_id[0];
+		sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
+				sched_type[1] << 8 |
+				queue_id[1];
+		sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
+				sched_type[2] << 8 |
+				queue_id[2];
+		sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
+				sched_type[3] << 8 |
+				queue_id[3];
+
+		/* Store the event priority, scheduling type, and queue ID in
+		 * the metadata:
+		 * sse_qe[0][31:16] = sched_word[0]
+		 * sse_qe[0][95:80] = sched_word[1]
+		 * sse_qe[1][31:16] = sched_word[2]
+		 * sse_qe[1][95:80] = sched_word[3]
+		 */
+#define DLB2_QE_QID_SCHED_WORD 1
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[0],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[1],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[2],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[3],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+
+		/* If the destination is a load-balanced queue, store the lock
+		 * ID. If it is a directed queue, DLB places this field in
+		 * bytes 10-11 of the received QE, so we format it accordingly:
+		 * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
+		 * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
+		 * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
+		 * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
+		 */
+#define DLB2_QE_LOCK_ID_WORD 2
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[0] == DLB2_SCHED_DIRECTED) ?
+					sched_word[0] : ev[0].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[1] == DLB2_SCHED_DIRECTED) ?
+					sched_word[1] : ev[1].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[2] == DLB2_SCHED_DIRECTED) ?
+					sched_word[2] : ev[2].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[3] == DLB2_SCHED_DIRECTED) ?
+					sched_word[3] : ev[3].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+
+		/* Store the event type and sub event type in the metadata:
+		 * sse_qe[0][15:0]  = flow_id[0]
+		 * sse_qe[0][79:64] = flow_id[1]
+		 * sse_qe[1][15:0]  = flow_id[2]
+		 * sse_qe[1][79:64] = flow_id[3]
+		 */
+#define DLB2_QE_EV_TYPE_WORD 0
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[0].sub_event_type << 8 |
+						ev[0].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[1].sub_event_type << 8 |
+						ev[1].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[2].sub_event_type << 8 |
+						ev[2].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[3].sub_event_type << 8 |
+						ev[3].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+
+		/* Store the metadata to memory (use the double-precision
+		 * _mm_storeh_pd because there is no integer function for
+		 * storing the upper 64b):
+		 * qe[0] metadata = sse_qe[0][63:0]
+		 * qe[1] metadata = sse_qe[0][127:64]
+		 * qe[2] metadata = sse_qe[1][63:0]
+		 * qe[3] metadata = sse_qe[1][127:64]
+		 */
+		_mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
+		_mm_storeh_pd((double *)&qe[1].u.opaque_data,
+			      (__m128d)sse_qe[0]);
+		_mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
+		_mm_storeh_pd((double *)&qe[3].u.opaque_data,
+			      (__m128d)sse_qe[1]);
+
+		qe[0].data = ev[0].u64;
+		qe[1].data = ev[1].u64;
+		qe[2].data = ev[2].u64;
+		qe[3].data = ev[3].u64;
+
+		break;
+	case 3:
+	case 2:
+	case 1:
+		/* At least one QE will be valid, so only zero out three */
+		qe[1].cmd_byte = 0;
+		qe[2].cmd_byte = 0;
+		qe[3].cmd_byte = 0;
+
+		for (i = 0; i < num; i++) {
+			qe[i].cmd_byte =
+				cmd_byte_map[qm_port->is_directed][ev[i].op];
+			qe[i].sched_type = sched_type[i];
+			qe[i].data = ev[i].u64;
+			qe[i].qid = queue_id[i];
+			qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
+			qe[i].lock_id = ev[i].flow_id;
+			if (sched_type[i] == DLB2_SCHED_DIRECTED) {
+				struct dlb2_msg_info *info =
+					(struct dlb2_msg_info *)&qe[i].lock_id;
+
+				info->qid = queue_id[i];
+				info->sched_type = DLB2_SCHED_DIRECTED;
+				info->priority = qe[i].priority;
+			}
+			qe[i].u.event_type.major = ev[i].event_type;
+			qe[i].u.event_type.sub = ev[i].sub_event_type;
+		}
+		break;
+	}
+}
+
+static inline int
+dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
+			struct dlb2_port *qm_port,
+			const struct rte_event ev[],
+			uint8_t *sched_type,
+			uint8_t *queue_id)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	uint16_t *cached_credits = NULL;
+	struct dlb2_queue *qm_queue;
+
+	ev_queue = &dlb2->ev_queues[ev->queue_id];
+	qm_queue = &ev_queue->qm_queue;
+	*queue_id = qm_queue->id;
+
+	/* Ignore sched_type and hardware credits on release events */
+	if (ev->op == RTE_EVENT_OP_RELEASE)
+		goto op_check;
+
+	if (!qm_queue->is_directed) {
+		/* Load balanced destination queue */
+
+		if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_ldb_credits;
+
+		switch (ev->sched_type) {
+		case RTE_SCHED_TYPE_ORDERED:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
+			if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
+				DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
+					     *queue_id);
+				rte_errno = -EINVAL;
+				return 1;
+			}
+			*sched_type = DLB2_SCHED_ORDERED;
+			break;
+		case RTE_SCHED_TYPE_ATOMIC:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
+			*sched_type = DLB2_SCHED_ATOMIC;
+			break;
+		case RTE_SCHED_TYPE_PARALLEL:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
+			if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
+				*sched_type = DLB2_SCHED_ORDERED;
+			else
+				*sched_type = DLB2_SCHED_UNORDERED;
+			break;
+		default:
+			DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
+			DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
+			rte_errno = -EINVAL;
+			return 1;
+		}
+	} else {
+		/* Directed destination queue */
+
+		if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_dir_credits;
+
+		DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
+
+		*sched_type = DLB2_SCHED_DIRECTED;
+	}
+
+op_check:
+	switch (ev->op) {
+	case RTE_EVENT_OP_NEW:
+		/* Check that a sw credit is available */
+		if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		ev_port->inflight_credits--;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_FORWARD:
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_RELEASE:
+		ev_port->inflight_credits++;
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+
+		/* Replenish s/w credits if enough are cached */
+		dlb2_replenish_sw_credits(dlb2, ev_port);
+		break;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+	if (ev->op != RTE_EVENT_OP_RELEASE) {
+		DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
+		DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
+	}
+#endif
+
+	return 0;
+}
+
+static inline uint16_t
+dlb2_event_enqueue_burst(void *event_port,
+			 const struct rte_event events[],
+			 uint16_t num)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct process_local_port_data *port_data;
+	int i, cnt;
+
+	RTE_ASSERT(ev_port->enq_configured);
+	RTE_ASSERT(events != NULL);
+
+	cnt = 0;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
+		int j = 0;
+
+		for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
+			const struct rte_event *ev = &events[i + j];
+
+			if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
+						    &sched_types[j],
+						    &queue_ids[j]))
+				break;
+		}
+
+		if (j == 0)
+			break;
+
+		dlb2_event_build_hcws(qm_port, &events[i], j,
+				      sched_types, queue_ids);
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+
+		if (j < DLB2_NUM_QES_PER_CACHE_LINE)
+			break;
+	}
+
+	return cnt;
+}
+
+static inline uint16_t
+dlb2_event_enqueue(void *event_port,
+		   const struct rte_event events[])
+{
+	return dlb2_event_enqueue_burst(event_port, events, 1);
+}
+
+static uint16_t
+dlb2_event_enqueue_new_burst(void *event_port,
+			     const struct rte_event events[],
+			     uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
+static uint16_t
+dlb2_event_enqueue_forward_burst(void *event_port,
+				 const struct rte_event events[],
+				 uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2112,7 +2684,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
 	};
 
+	/* Expose PMD's eventdev interface */
+
 	dev->dev_ops = &dlb2_eventdev_entry_ops;
+	dev->enqueue = dlb2_event_enqueue;
+	dev->enqueue_burst = dlb2_event_enqueue_burst;
+	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
+	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v3 18/23] event/dlb2: add dequeue and its burst variants

[Timothy McDaniel]

Add support for dequeue, dequeue_burst, ...

DLB2 does not currently support interrupts, but instead use
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB2 supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst |  56 ++--
 drivers/event/dlb2/dlb2.c     | 761 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 782 insertions(+), 35 deletions(-)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index b9607b7..770227f 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -82,41 +82,6 @@ The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
 the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
 load-balanced queues can use the full 16-bit flow ID range.
 
-Load-Balanced Queues
-~~~~~~~~~~~~~~~~~~~
-
-A load-balanced queue can support atomic and ordered scheduling, or atomic and
-unordered scheduling, but not atomic and unordered and ordered scheduling. A
-queue's scheduling types are controlled by the event queue configuration.
-
-If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
-``nb_atomic_order_sequences`` determines the supported scheduling types.
-With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
-and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
-supported by scheduling those events as ordered events.  Note that when the
-event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
-``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
-unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
-
-If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
-dictates the queue's scheduling type.
-
-The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
-queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
-group is configured to contain either 1 queue with 1024 reorder entries, 2
-queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
-
-When a load-balanced queue is created, the PMD will configure a new sequence
-number group on-demand if num_sequence_numbers does not match a pre-existing
-group with available reorder buffer entries. If all sequence number groups are
-in use, no new group will be created and queue configuration will fail. (Note
-that when the PMD is used with a virtual DLB2 device, it cannot change the
-sequence number configuration.)
-
-The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
-the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
-load-balanced queues can use the full 16-bit flow ID range.
-
 Load-balanced and Directed Ports
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -314,6 +279,27 @@ The PMD does not support the following configuration sequences:
 This sequence is not supported because the event device must be reconfigured
 before its ports or queues can be.
 
+Deferred Scheduling
+~~~~~~~~~~~~~~~~~~~
+
+The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
+dequeued event before returning from rte_event_dequeue_burst(). This frees the
+corresponding entries in the CQ, which enables the DLB2 to schedule more events
+to it.
+
+To support applications seeking finer-grained scheduling control -- for example
+deferring scheduling to get the best possible priority scheduling and
+load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
+the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
+call. This mode only applies to load-balanced event ports with dequeue depth of
+1.
+
+To enable deferred scheduling, use the defer_sched vdev argument like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,defer_sched=on
+
 Atomic Inflights Allocation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 90a5c9d..bd96055 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2661,9 +2661,761 @@ dlb2_event_enqueue_forward_burst(void *event_port,
 	return dlb2_event_enqueue_burst(event_port, events, num);
 }
 
+static inline void
+dlb2_port_credits_inc(struct dlb2_port *qm_port, int num)
+{
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	/* increment port credits, and return to pool if exceeds threshold */
+	if (!qm_port->is_directed) {
+		qm_port->cached_ldb_credits += num;
+		if (qm_port->cached_ldb_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_LDB_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_ldb_credits -= batch_size;
+		}
+	} else {
+		qm_port->cached_dir_credits += num;
+		if (qm_port->cached_dir_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_DIR_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_dir_credits -= batch_size;
+		}
+	}
+}
+
+static inline bool
+dlb2_cq_is_empty(struct dlb2_port *qm_port)
+{
+	volatile struct dlb2_dequeue_qe *qe_ptr;
+	struct dlb2_dequeue_qe qe;
+
+	qe_ptr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	qe = qe_ptr[qm_port->cq_idx];
+
+	return (qe.cq_gen != qm_port->gen_bit);
+}
+
+static inline int
+dlb2_dequeue_wait(struct dlb2_eventdev *dlb2,
+		  struct dlb2_eventdev_port *ev_port,
+		  struct dlb2_port *qm_port,
+		  uint64_t timeout,
+		  uint64_t start_ticks)
+{
+	struct process_local_port_data *port_data;
+	uint64_t elapsed_ticks;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	elapsed_ticks = rte_get_timer_cycles() - start_ticks;
+
+	/* Wait/poll time expired */
+	if (elapsed_ticks >= timeout) {
+		return 1;
+	} else if (dlb2->umwait_allowed) {
+		volatile struct dlb2_dequeue_qe *cq_base;
+
+		cq_base = port_data->cq_base;
+
+		/* Block on cache line write to CQ. Note: it's
+		 * safe to access the per-process cq_base
+		 * address here, since the PMD has already
+		 * attempted at least one CQ dequeue.
+		 */
+		dlb2_umonitor(&cq_base[qm_port->cq_idx]);
+
+		/* Avoid race condition. Check if still empty */
+		if (dlb2_cq_is_empty(qm_port)) {
+			dlb2_umwait(RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE,
+				    timeout + start_ticks);
+			DLB2_INC_STAT(
+				ev_port->stats.traffic.rx_umonitor_umwait, 1);
+		}
+	} else {
+		uint64_t poll_interval = RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL;
+		uint64_t curr_ticks = rte_get_timer_cycles();
+		uint64_t init_ticks = curr_ticks;
+
+		while ((curr_ticks - start_ticks < timeout) &&
+		       (curr_ticks - init_ticks < poll_interval))
+			curr_ticks = rte_get_timer_cycles();
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
+			 struct dlb2_port *qm_port,
+			 struct rte_event *events,
+			 struct dlb2_dequeue_qe *qes,
+			 int cnt)
+{
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	int i, num, evq_id;
+
+	for (i = 0, num = 0; i < cnt; i++) {
+		struct dlb2_dequeue_qe *qe = &qes[i];
+		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
+			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+			[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+		};
+
+		/* Fill in event information.
+		 * Note that flow_id must be embedded in the data by
+		 * the app, such as the mbuf RSS hash field if the data
+		 * buffer is a mbuf.
+		 */
+		if (unlikely(qe->error)) {
+			DLB2_LOG_ERR("QE error bit ON\n");
+			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
+			dlb2_consume_qe_immediate(qm_port, 1);
+			continue; /* Ignore */
+		}
+
+		events[num].u64 = qe->data;
+		events[num].flow_id = qe->flow_id;
+		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
+		events[num].event_type = qe->u.event_type.major;
+		events[num].sub_event_type = qe->u.event_type.sub;
+		events[num].sched_type = sched_type_map[qe->sched_type];
+		events[num].impl_opaque = qe->qid_depth;
+
+		/* qid not preserved for directed queues */
+		if (qm_port->is_directed)
+			evq_id = ev_port->link[0].queue_id;
+		else
+			evq_id = qid_mappings[qe->qid];
+
+		events[num].queue_id = evq_id;
+		DLB2_INC_STAT(
+			ev_port->stats.queue[evq_id].qid_depth[qe->qid_depth],
+			1);
+		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type], 1);
+		num++;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num);
+
+	return num;
+}
+
+static inline int
+dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
+			      struct dlb2_port *qm_port,
+			      struct rte_event *events,
+			      struct dlb2_dequeue_qe *qes)
+{
+	int sched_type_map[] = {
+		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+	};
+	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	__m128i sse_evt[2];
+
+	/* In the unlikely case that any of the QE error bits are set, process
+	 * them one at a time.
+	 */
+	if (unlikely(qes[0].error || qes[1].error ||
+		     qes[2].error || qes[3].error))
+		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
+						 qes, num_events);
+
+	events[0].u64 = qes[0].data;
+	events[1].u64 = qes[1].data;
+	events[2].u64 = qes[2].data;
+	events[3].u64 = qes[3].data;
+
+	/* Construct the metadata portion of two struct rte_events
+	 * in one 128b SSE register. Event metadata is constructed in the SSE
+	 * registers like so:
+	 * sse_evt[0][63:0]:   event[0]'s metadata
+	 * sse_evt[0][127:64]: event[1]'s metadata
+	 * sse_evt[1][63:0]:   event[2]'s metadata
+	 * sse_evt[1][127:64]: event[3]'s metadata
+	 */
+	sse_evt[0] = _mm_setzero_si128();
+	sse_evt[1] = _mm_setzero_si128();
+
+	/* Convert the hardware queue ID to an event queue ID and store it in
+	 * the metadata:
+	 * sse_evt[0][47:40]   = qid_mappings[qes[0].qid]
+	 * sse_evt[0][111:104] = qid_mappings[qes[1].qid]
+	 * sse_evt[1][47:40]   = qid_mappings[qes[2].qid]
+	 * sse_evt[1][111:104] = qid_mappings[qes[3].qid]
+	 */
+#define DLB_EVENT_QUEUE_ID_BYTE 5
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[0].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[1].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[2].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[3].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+
+	/* Convert the hardware priority to an event priority and store it in
+	 * the metadata, while also returning the queue depth status
+	 * value captured by the hardware, storing it in impl_opaque, which can
+	 * be read by the application but not modified
+	 * sse_evt[0][55:48]   = DLB2_TO_EV_PRIO(qes[0].priority)
+	 * sse_evt[0][63:56]   = qes[0].qid_depth
+	 * sse_evt[0][119:112] = DLB2_TO_EV_PRIO(qes[1].priority)
+	 * sse_evt[0][127:120] = qes[1].qid_depth
+	 * sse_evt[1][55:48]   = DLB2_TO_EV_PRIO(qes[2].priority)
+	 * sse_evt[1][63:56]   = qes[2].qid_depth
+	 * sse_evt[1][119:112] = DLB2_TO_EV_PRIO(qes[3].priority)
+	 * sse_evt[1][127:120] = qes[3].qid_depth
+	 */
+#define DLB_EVENT_PRIO_IMPL_OPAQUE_WORD 3
+#define DLB_BYTE_SHIFT 8
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[0].priority) |
+			(qes[0].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[1].priority) |
+			(qes[1].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[2].priority) |
+			(qes[2].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[3].priority) |
+			(qes[3].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+
+	/* Write the event type, sub event type, and flow_id to the event
+	 * metadata.
+	 * sse_evt[0][31:0]   = qes[0].flow_id |
+	 *			qes[0].u.event_type.major << 28 |
+	 *			qes[0].u.event_type.sub << 20;
+	 * sse_evt[0][95:64]  = qes[1].flow_id |
+	 *			qes[1].u.event_type.major << 28 |
+	 *			qes[1].u.event_type.sub << 20;
+	 * sse_evt[1][31:0]   = qes[2].flow_id |
+	 *			qes[2].u.event_type.major << 28 |
+	 *			qes[2].u.event_type.sub << 20;
+	 * sse_evt[1][95:64]  = qes[3].flow_id |
+	 *			qes[3].u.event_type.major << 28 |
+	 *			qes[3].u.event_type.sub << 20;
+	 */
+#define DLB_EVENT_EV_TYPE_DW 0
+#define DLB_EVENT_EV_TYPE_SHIFT 28
+#define DLB_EVENT_SUB_EV_TYPE_SHIFT 20
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[0].flow_id |
+			qes[0].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[0].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[1].flow_id |
+			qes[1].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[1].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[2].flow_id |
+			qes[2].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[2].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[3].flow_id |
+			qes[3].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT  |
+			qes[3].u.event_type.sub << DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+
+	/* Write the sched type to the event metadata. 'op' and 'rsvd' are not
+	 * set:
+	 * sse_evt[0][39:32]  = sched_type_map[qes[0].sched_type] << 6
+	 * sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6
+	 * sse_evt[1][39:32]  = sched_type_map[qes[2].sched_type] << 6
+	 * sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6
+	 */
+#define DLB_EVENT_SCHED_TYPE_BYTE 4
+#define DLB_EVENT_SCHED_TYPE_SHIFT 6
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[0].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[1].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[2].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[3].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+
+	/* Store the metadata to the event (use the double-precision
+	 * _mm_storeh_pd because there is no integer function for storing the
+	 * upper 64b):
+	 * events[0].event = sse_evt[0][63:0]
+	 * events[1].event = sse_evt[0][127:64]
+	 * events[2].event = sse_evt[1][63:0]
+	 * events[3].event = sse_evt[1][127:64]
+	 */
+	_mm_storel_epi64((__m128i *)&events[0].event, sse_evt[0]);
+	_mm_storeh_pd((double *)&events[1].event, (__m128d) sse_evt[0]);
+	_mm_storel_epi64((__m128i *)&events[2].event, sse_evt[1]);
+	_mm_storeh_pd((double *)&events[3].event, (__m128d) sse_evt[1]);
+
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[0].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[1].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[2].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[3].sched_type], 1);
+
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[0].queue_id].
+			qid_depth[qes[0].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[1].queue_id].
+			qid_depth[qes[1].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[2].queue_id].
+			qid_depth[qes[2].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[3].queue_id].
+			qid_depth[qes[3].qid_depth],
+		1);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num_events);
+
+	return num_events;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe_sparse(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe)
+{
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	uint8_t xor_mask[2] = {0x0F, 0x00};
+	const uint8_t and_mask = 0x0F;
+	__m128i *qes = (__m128i *)qe;
+	uint8_t gen_bits, gen_bit;
+	uintptr_t addr[4];
+	uint16_t idx;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+
+	idx = qm_port->cq_idx;
+
+	/* Load the next 4 QEs */
+	addr[0] = (uintptr_t)&cq_addr[idx];
+	addr[1] = (uintptr_t)&cq_addr[(idx +  4) & qm_port->cq_depth_mask];
+	addr[2] = (uintptr_t)&cq_addr[(idx +  8) & qm_port->cq_depth_mask];
+	addr[3] = (uintptr_t)&cq_addr[(idx + 12) & qm_port->cq_depth_mask];
+
+	/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */
+	rte_prefetch0(&cq_addr[(idx + 16) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 20) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 24) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 28) & qm_port->cq_depth_mask]);
+
+	/* Correct the xor_mask for wrap-around QEs */
+	gen_bit = qm_port->gen_bit;
+	xor_mask[gen_bit] ^= !!((idx +  4) > qm_port->cq_depth_mask) << 1;
+	xor_mask[gen_bit] ^= !!((idx +  8) > qm_port->cq_depth_mask) << 2;
+	xor_mask[gen_bit] ^= !!((idx + 12) > qm_port->cq_depth_mask) << 3;
+
+	/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is
+	 * valid, then QEs[0:N-1] are too.
+	 */
+	qes[3] = _mm_load_si128((__m128i *)(void *)addr[3]);
+	rte_compiler_barrier();
+	qes[2] = _mm_load_si128((__m128i *)(void *)addr[2]);
+	rte_compiler_barrier();
+	qes[1] = _mm_load_si128((__m128i *)(void *)addr[1]);
+	rte_compiler_barrier();
+	qes[0] = _mm_load_si128((__m128i *)(void *)addr[0]);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[gen_bit];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask;
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline void
+dlb2_inc_cq_idx(struct dlb2_port *qm_port, int cnt)
+{
+	uint16_t idx = qm_port->cq_idx_unmasked + cnt;
+
+	qm_port->cq_idx_unmasked = idx;
+	qm_port->cq_idx = idx & qm_port->cq_depth_mask;
+	qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) & 0x1;
+}
+
+static int
+dlb2_event_release(struct dlb2_eventdev *dlb2,
+		   uint8_t port_id,
+		   int n)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_eventdev_port *ev_port;
+	struct dlb2_port *qm_port;
+	int i, cnt;
+
+	if (port_id > dlb2->num_ports) {
+		DLB2_LOG_ERR("Invalid port id %d in dlb2-event_release\n",
+			     port_id);
+		rte_errno = -EINVAL;
+		return rte_errno;
+	}
+
+	ev_port = &dlb2->ev_ports[port_id];
+	qm_port = &ev_port->qm_port;
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	cnt = 0;
+
+	if (qm_port->is_directed) {
+		cnt = n;
+		goto sw_credit_update;
+	}
+
+	for (i = 0; i < n; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		int j;
+
+		/* Zero-out QEs */
+		qm_port->qe4[0].cmd_byte = 0;
+		qm_port->qe4[1].cmd_byte = 0;
+		qm_port->qe4[2].cmd_byte = 0;
+		qm_port->qe4[3].cmd_byte = 0;
+
+		for (j = 0; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++)
+			qm_port->qe4[j].cmd_byte = DLB2_COMP_CMD_BYTE;
+
+		qm_port->issued_releases += j;
+
+		if (j == 0)
+			break;
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+	}
+
+sw_credit_update:
+	/* each release returns one credit */
+	if (!ev_port->outstanding_releases) {
+		DLB2_LOG_ERR("Unrecoverable application error. Outstanding releases underflowed.\n");
+		rte_errno = -ENOTRECOVERABLE;
+		return rte_errno;
+	}
+
+	ev_port->outstanding_releases -= cnt;
+	ev_port->inflight_credits += cnt;
+
+	/* Replenish s/w credits if enough releases are performed */
+	dlb2_replenish_sw_credits(dlb2, ev_port);
+	return 0;
+}
+
+static inline int16_t
+dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
+		       struct dlb2_eventdev_port *ev_port,
+		       struct rte_event *events,
+		       uint16_t max_num,
+		       uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe_sparse(qm_port, qes);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail << 2);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							      qm_port,
+							      &events[num],
+							      &qes[0]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							 qm_port,
+							 &events[num],
+							 &qes[0],
+							 num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe,
+	     uint8_t *offset)
+{
+	uint8_t xor_mask[2][4] = { {0x0F, 0x0E, 0x0C, 0x08},
+				   {0x00, 0x01, 0x03, 0x07} };
+	uint8_t and_mask[4] = {0x0F, 0x0E, 0x0C, 0x08};
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	__m128i *qes = (__m128i *)qe;
+	uint64_t *cache_line_base;
+	uint8_t gen_bits;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	cq_addr = &cq_addr[qm_port->cq_idx];
+
+	cache_line_base = (void *)(((uintptr_t)cq_addr) & ~0x3F);
+	*offset = ((uintptr_t)cq_addr & 0x30) >> 4;
+
+	/* Load the next CQ cache line from memory. Pack these reads as tight
+	 * as possible to reduce the chance that DLB invalidates the line while
+	 * the CPU is reading it. Read the cache line backwards to ensure that
+	 * if QE[N] (N > 0) is valid, then QEs[0:N-1] are too.
+	 *
+	 * (Valid QEs start at &qe[offset])
+	 */
+	qes[3] = _mm_load_si128((__m128i *)&cache_line_base[6]);
+	qes[2] = _mm_load_si128((__m128i *)&cache_line_base[4]);
+	qes[1] = _mm_load_si128((__m128i *)&cache_line_base[2]);
+	qes[0] = _mm_load_si128((__m128i *)&cache_line_base[0]);
+
+	/* Evict the cache line ASAP */
+	dlb2_cldemote(cache_line_base);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[qm_port->gen_bit][*offset];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask[*offset];
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline int16_t
+dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
+		struct dlb2_eventdev_port *ev_port,
+		struct rte_event *events,
+		uint16_t max_num,
+		uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t offset;
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe(qm_port, qes, &offset);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							     qm_port,
+							     &events[num],
+							     &qes[offset]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							qm_port,
+							&events[num],
+							&qes[offset],
+							num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static uint16_t
+dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
+			 uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue(void *event_port, struct rte_event *ev, uint64_t wait)
+{
+	return dlb2_event_dequeue_burst(event_port, ev, 1, wait);
+}
+
+static uint16_t
+dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
+				uint16_t num, uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
+			  uint64_t wait)
+{
+	return dlb2_event_dequeue_burst_sparse(event_port, ev, 1, wait);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
+	struct dlb2_eventdev *dlb2;
+
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
@@ -2691,6 +3443,15 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	dev->enqueue_burst = dlb2_event_enqueue_burst;
 	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
 	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
+
+	dlb2 = dev->data->dev_private;
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE) {
+		dev->dequeue = dlb2_event_dequeue_sparse;
+		dev->dequeue_burst = dlb2_event_dequeue_burst_sparse;
+	} else {
+		dev->dequeue = dlb2_event_dequeue;
+		dev->dequeue_burst = dlb2_event_dequeue_burst;
+	}
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v3 19/23] event/dlb2: add eventdev stop and close

[Timothy McDaniel]

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 257 +++++++++++++++++++++++++++--
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c |  91 ++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  52 ++++++
 5 files changed, 396 insertions(+), 16 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index bd96055..b02a70c 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,21 +70,6 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
-/*
- * DUMMY - added so that xstats path will compile/link.
- * Will be replaced by real version in a subsequent
- * patch.
- */
-uint32_t
-dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
-		     struct dlb2_eventdev_queue *queue)
-{
-	RTE_SET_USED(dlb2);
-	RTE_SET_USED(queue);
-
-	return 0;
-}
-
 static void
 dlb2_free_qe_mem(struct dlb2_port *qm_port)
 {
@@ -1877,7 +1862,6 @@ dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
 		return 0; /* Ignore and return success */
 	}
 
-	/* FIXME: How to handle unlink on a directed port? */
 	if (ev_port->qm_port.is_directed) {
 		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
 			     ev_port->id);
@@ -3412,6 +3396,245 @@ dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
 }
 
 static void
+dlb2_flush_port(struct rte_eventdev *dev, int port_id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	eventdev_stop_flush_t flush;
+	struct rte_event ev;
+	uint8_t dev_id;
+	void *arg;
+	int i;
+
+	flush = dev->dev_ops->dev_stop_flush;
+	dev_id = dev->data->dev_id;
+	arg = dev->data->dev_stop_flush_arg;
+
+	while (rte_event_dequeue_burst(dev_id, port_id, &ev, 1, 0)) {
+		if (flush)
+			flush(dev_id, ev, arg);
+
+		if (dlb2->ev_ports[port_id].qm_port.is_directed)
+			continue;
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+	}
+
+	/* Enqueue any additional outstanding releases */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = dlb2->ev_ports[port_id].outstanding_releases; i > 0; i--)
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+}
+
+static uint32_t
+dlb2_get_ldb_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_ldb_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_ldb_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_ldb_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static uint32_t
+dlb2_get_dir_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_dir_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_dir_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_dir_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	if (queue->qm_queue.is_directed)
+		return dlb2_get_dir_queue_depth(dlb2, queue);
+	else
+		return dlb2_get_ldb_queue_depth(dlb2, queue);
+}
+
+static bool
+dlb2_queue_is_empty(struct dlb2_eventdev *dlb2,
+		    struct dlb2_eventdev_queue *queue)
+{
+	return dlb2_get_queue_depth(dlb2, queue) == 0;
+}
+
+static bool
+dlb2_linked_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0)
+			continue;
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static bool
+dlb2_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static void
+dlb2_drain(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_port *ev_port = NULL;
+	uint8_t dev_id;
+	int i;
+
+	dev_id = dev->data->dev_id;
+
+	while (!dlb2_linked_queues_empty(dlb2)) {
+		/* Flush all the ev_ports, which will drain all their connected
+		 * queues.
+		 */
+		for (i = 0; i < dlb2->num_ports; i++)
+			dlb2_flush_port(dev, i);
+	}
+
+	/* The queues are empty, but there may be events left in the ports. */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_flush_port(dev, i);
+
+	/* If the domain's queues are empty, we're done. */
+	if (dlb2_queues_empty(dlb2))
+		return;
+
+	/* Else, there must be at least one unlinked load-balanced queue.
+	 * Select a load-balanced port with which to drain the unlinked
+	 * queue(s).
+	 */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		if (!ev_port->qm_port.is_directed)
+			break;
+	}
+
+	if (i == dlb2->num_ports) {
+		DLB2_LOG_ERR("internal error: no LDB ev_ports\n");
+		return;
+	}
+
+	rte_errno = 0;
+	rte_event_port_unlink(dev_id, ev_port->id, NULL, 0);
+
+	if (rte_errno) {
+		DLB2_LOG_ERR("internal error: failed to unlink ev_port %d\n",
+			     ev_port->id);
+		return;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		uint8_t qid, prio;
+		int ret;
+
+		if (dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			continue;
+
+		qid = i;
+		prio = 0;
+
+		/* Link the ev_port to the queue */
+		ret = rte_event_port_link(dev_id, ev_port->id, &qid, &prio, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to link ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+
+		/* Flush the queue */
+		while (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			dlb2_flush_port(dev, ev_port->id);
+
+		/* Drain any extant events in the ev_port. */
+		dlb2_flush_port(dev, ev_port->id);
+
+		/* Unlink the ev_port from the queue */
+		ret = rte_event_port_unlink(dev_id, ev_port->id, &qid, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to unlink ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+	}
+}
+
+static void
+dlb2_eventdev_stop(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_DBG("Internal error: already stopped\n");
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	} else if (dlb2->run_state != DLB2_RUN_STATE_STARTED) {
+		DLB2_LOG_ERR("Internal error: bad state %d for dev_stop\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPING;
+
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	dlb2_drain(dev);
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPED;
+}
+
+static int
+dlb2_eventdev_close(struct rte_eventdev *dev)
+{
+	dlb2_hw_reset_sched_domain(dev, false);
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3421,6 +3644,8 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.dev_start        = dlb2_eventdev_start,
+		.dev_stop         = dlb2_eventdev_stop,
+		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a86191d..5471dd8 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -66,3 +66,9 @@ int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				     struct dlb2_start_domain_args *cfg);
+
+int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 7ead374..b508eb0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -65,4 +65,10 @@ extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				struct dlb2_start_domain_args *cfg);
+
+extern int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+extern int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index c14a2f3..ae5ef2f 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5934,3 +5934,94 @@ dlb2_hw_start_domain(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_dir_queue_depth(hw, queue);
+
+	return 0;
+}
+
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+
+	dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 3bcf3b9..d7db9da 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -527,6 +527,56 @@ dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_ldb_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_ldb_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_dir_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret = 0;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_dir_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -544,6 +594,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth;
+	dlb2_iface_get_dir_queue_depth = dlb2_pf_get_dir_queue_depth;
 	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v3 20/23] event/dlb2: add PMD's token pop public interface

[Timothy McDaniel]

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/api/doxy-api-index.md                         |  1 +
 drivers/event/dlb2/dlb2.c                         | 53 +++++++++++++++--
 drivers/event/dlb2/dlb2_priv.h                    |  3 +
 drivers/event/dlb2/meson.build                    |  5 +-
 drivers/event/dlb2/rte_pmd_dlb2.c                 | 39 ++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2.h                 | 72 +++++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2_event_version.map |  6 ++
 7 files changed, 172 insertions(+), 7 deletions(-)
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index b855a8f..6bc72e9 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -51,6 +51,7 @@ The public API headers are grouped by topics:
   [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
   [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
+  [dlb2]	       (@ref rte_pmd_dlb2.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index b02a70c..b25b213 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1196,7 +1196,7 @@ dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
 	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
 
 	qm_port->dequeue_depth = dequeue_depth;
-
+	qm_port->token_pop_thresh = dequeue_depth;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -1364,6 +1364,8 @@ dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
 
 	qm_port->dequeue_depth = dequeue_depth;
 
+	/* Directed ports are auto-pop, by default. */
+	qm_port->token_pop_mode = AUTO_POP;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -2611,6 +2613,14 @@ dlb2_event_enqueue_burst(void *event_port,
 		dlb2_event_build_hcws(qm_port, &events[i], j,
 				      sched_types, queue_ids);
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
@@ -2619,6 +2629,11 @@ dlb2_event_enqueue_burst(void *event_port,
 			break;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
 	return cnt;
 }
 
@@ -3101,11 +3116,25 @@ dlb2_event_release(struct dlb2_eventdev *dlb2,
 		if (j == 0)
 			break;
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
 sw_credit_update:
 	/* each release returns one credit */
 	if (!ev_port->outstanding_releases) {
@@ -3189,8 +3218,8 @@ dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3316,8 +3345,8 @@ dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3332,6 +3361,7 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 			 uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3347,6 +3377,9 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3366,6 +3399,7 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 				uint16_t num, uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3381,6 +3415,9 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3685,7 +3722,7 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    struct dlb2_devargs *dlb2_args)
 {
 	struct dlb2_eventdev *dlb2;
-	int err;
+	int err, i;
 
 	dlb2 = dev->data->dev_private;
 
@@ -3735,6 +3772,10 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Initialize each port's token pop mode */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
+		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 61567a6..b73cf3f 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -12,6 +12,7 @@
 #include <rte_config.h>
 #include "dlb2_user.h"
 #include "dlb2_log.h"
+#include "rte_pmd_dlb2.h"
 
 #ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
 #define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
@@ -290,6 +291,7 @@ struct dlb2_port {
 	bool gen_bit;
 	uint16_t dir_credits;
 	uint32_t dequeue_depth;
+	enum dlb2_token_pop_mode token_pop_mode;
 	union dlb2_port_config cfg;
 	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
 	uint16_t cached_ldb_credits;
@@ -298,6 +300,7 @@ struct dlb2_port {
 	bool int_armed;
 	uint16_t owed_tokens;
 	int16_t issued_releases;
+	int16_t token_pop_thresh;
 	int cq_depth;
 	uint16_t cq_idx;
 	uint16_t cq_idx_unmasked;
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index dc16c96..b734ff6 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,3 +1,4 @@
+
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
@@ -12,7 +13,9 @@ sources = files('dlb2.c',
 		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
-		'pf/base/dlb2_resource.c'
+		'pf/base/dlb2_resource.c',
+		'rte_pmd_dlb2.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
+install_headers('rte_pmd_dlb2.h')
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
new file mode 100644
index 0000000..b09b585
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "rte_pmd_dlb2.h"
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode)
+{
+	struct dlb2_eventdev *dlb2;
+	struct rte_eventdev *dev;
+
+	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+	dev = &rte_eventdevs[dev_id];
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (mode >= NUM_TOKEN_POP_MODES)
+		return -EINVAL;
+
+	/* The event device must be configured, but not yet started */
+	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
+		return -EINVAL;
+
+	/* The token pop mode must be set before configuring the port */
+	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
+		return -EINVAL;
+
+	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
new file mode 100644
index 0000000..74399db
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+/*!
+ *  @file      rte_pmd_dlb2.h
+ *
+ *  @brief     DLB PMD-specific functions
+ */
+
+#ifndef _RTE_PMD_DLB2_H_
+#define _RTE_PMD_DLB2_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Selects the token pop mode for a DLB2 port.
+ */
+enum dlb2_token_pop_mode {
+	/* Pop the CQ tokens immediately after dequeueing. */
+	AUTO_POP,
+	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
+	 * Supported on load-balanced ports only.
+	 */
+	DELAYED_POP,
+	/* Pop the CQ tokens during next dequeue operation. */
+	DEFERRED_POP,
+
+	/* NUM_TOKEN_POP_MODES must be last */
+	NUM_TOKEN_POP_MODES
+};
+
+/*!
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Configure the token pop mode for a DLB2 port. By default, all ports use
+ * AUTO_POP. This function must be called before calling rte_event_port_setup()
+ * for the port, but after calling rte_event_dev_configure().
+ *
+ * @param dev_id
+ *    The identifier of the event device.
+ * @param port_id
+ *    The identifier of the event port.
+ * @param mode
+ *    The token pop mode.
+ *
+ * @return
+ * - 0: Success
+ * - EINVAL: Invalid dev_id, port_id, or mode
+ * - EINVAL: The DLB2 is not configured, is already running, or the port is
+ *   already setup
+ */
+
+__rte_experimental
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_DLB2_H_ */
diff --git a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
index 4a76d1d..b1e4dff 100644
--- a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
+++ b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
@@ -1,3 +1,9 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmd_dlb2_set_token_pop_mode;
+};
-- 
2.6.4

[dpdk-dev] [PATCH v3 21/23] event/dlb2: add PMD self-tests

[Timothy McDaniel]

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 app/test/test_eventdev.c           |    7 +
 drivers/event/dlb2/dlb2.c          |    1 +
 drivers/event/dlb2/dlb2_selftest.c | 1570 ++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build     |    3 +-
 4 files changed, 1580 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c

diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
index 62019c1..bcfaa53 100644
--- a/app/test/test_eventdev.c
+++ b/app/test/test_eventdev.c
@@ -1030,6 +1030,12 @@ test_eventdev_selftest_dpaa2(void)
 	return test_eventdev_selftest_impl("event_dpaa2", "");
 }
 
+static int
+test_eventdev_selftest_dlb2(void)
+{
+	return test_eventdev_selftest_impl("dlb2_event", "");
+}
+
 REGISTER_TEST_COMMAND(eventdev_common_autotest, test_eventdev_common);
 REGISTER_TEST_COMMAND(eventdev_selftest_sw, test_eventdev_selftest_sw);
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
@@ -1037,3 +1043,4 @@ REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
 		test_eventdev_selftest_octeontx2);
 REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2, test_eventdev_selftest_dpaa2);
+REGISTER_TEST_COMMAND(eventdev_selftest_dlb2, test_eventdev_selftest_dlb2);
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index b25b213..113fedf 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3696,6 +3696,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
 		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+		.dev_selftest     = test_dlb2_eventdev,
 	};
 
 	/* Expose PMD's eventdev interface */
diff --git a/drivers/event/dlb2/dlb2_selftest.c b/drivers/event/dlb2/dlb2_selftest.c
new file mode 100644
index 0000000..418a53b
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_selftest.c
@@ -0,0 +1,1570 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <unistd.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_eventdev.h>
+#include <rte_pause.h>
+
+#include "dlb2_priv.h"
+#include "rte_pmd_dlb2.h"
+
+#define MAX_PORTS 32
+#define MAX_QIDS 32
+#define DEFAULT_NUM_SEQ_NUMS 64
+
+static struct rte_mempool *eventdev_func_mempool;
+static int evdev;
+
+struct test {
+	struct rte_mempool *mbuf_pool;
+	int nb_qids;
+};
+
+/* initialization and config */
+static inline int
+init(struct test *t, int nb_queues, int nb_ports)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	memset(t, 0, sizeof(*t));
+
+	t->mbuf_pool = eventdev_func_mempool;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = nb_queues;
+	config.nb_event_ports = nb_ports;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0)
+		printf("%d: Error configuring device\n", __LINE__);
+
+	return ret;
+}
+
+static inline int
+create_ports(int num_ports)
+{
+	int i;
+
+	if (num_ports > MAX_PORTS)
+		return -1;
+
+	for (i = 0; i < num_ports; i++) {
+		struct rte_event_port_conf conf;
+
+		if (rte_event_port_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default port conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		if (rte_event_port_setup(evdev, i, &conf) < 0) {
+			printf("%d: Error setting up port %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+create_lb_qids(struct test *t, int num_qids, uint32_t flags)
+{
+	int i;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		struct rte_event_queue_conf conf;
+
+		if (rte_event_queue_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default queue conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		conf.schedule_type = flags;
+
+		if (conf.schedule_type == RTE_SCHED_TYPE_PARALLEL)
+			conf.nb_atomic_order_sequences = 0;
+		else
+			conf.nb_atomic_order_sequences = DEFAULT_NUM_SEQ_NUMS;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+static inline int
+create_atomic_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ATOMIC);
+}
+
+static inline int
+create_ordered_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ORDERED);
+}
+
+static inline int
+create_unordered_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_PARALLEL);
+}
+
+static inline int
+create_directed_qids(struct test *t, int num_qids, const uint8_t ports[])
+{
+	static struct rte_event_queue_conf conf;
+	int i;
+
+	conf.priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
+	conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		uint8_t queue_id;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, ports[i - t->nb_qids],
+					&queue_id, NULL, 1) != 1) {
+			printf("%d: error creating link for qid %d\n",
+			       __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+/* destruction */
+static inline void
+cleanup(void)
+{
+	int ret = 0;
+
+	rte_event_dev_stop(evdev);
+	ret = rte_event_dev_close(evdev);
+
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+};
+
+static inline int
+enqueue_timeout(uint8_t port_id, struct rte_event *ev, uint64_t tmo_us)
+{
+	const uint64_t start = rte_get_timer_cycles();
+	const uint64_t ticks = (tmo_us * rte_get_timer_hz()) / 1E6;
+
+	while ((rte_get_timer_cycles() - start) < ticks) {
+		if (rte_event_enqueue_burst(evdev, port_id, ev, 1) == 1)
+			return 0;
+
+		if (rte_errno != -ENOSPC) {
+			printf("enqueue_burst returned rte_errno %d\n",
+			       rte_errno);
+			return -1;
+		}
+	}
+	printf("%s time out\n", __func__);
+	return -1;
+}
+
+static void
+flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused)
+{
+	rte_pktmbuf_free(event.mbuf);
+}
+
+static int
+test_stop_flush(struct test *t) /* test to check we can properly flush events */
+{
+	struct rte_event ev;
+	uint32_t dequeue_depth;
+	unsigned int i, count;
+	uint8_t queue_id;
+
+	ev.op = RTE_EVENT_OP_NEW;
+
+	if (init(t, 2, 1) < 0 ||
+	    create_ports(1) < 0 ||
+	    create_atomic_qids(t, 2) < 0) {
+		printf("%d: Error initializing device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_port_link(evdev, 0, NULL, NULL, 0) != 2) {
+		printf("%d: Error linking queues to the port\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: Error with start call\n", __LINE__);
+		goto err;
+	}
+
+	/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked
+	 * queue.
+	 */
+	queue_id = 1;
+
+	if (rte_event_port_unlink(evdev, 0, &queue_id, 1) != 1) {
+		printf("%d: Error unlinking queue 1 from port\n", __LINE__);
+		goto err;
+	}
+
+	count = rte_mempool_avail_count(t->mbuf_pool);
+
+	if (rte_event_port_attr_get(evdev,
+				    0,
+				    RTE_EVENT_PORT_ATTR_DEQ_DEPTH,
+				    &dequeue_depth)) {
+		printf("%d: Error retrieveing dequeue depth\n", __LINE__);
+		goto err;
+	}
+
+	/* Send QEs to queue 0 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 0;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Send QEs to queue 1 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 1;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Now the DLB is scheduling events from the port to the IQ, and at
+	 * least one event should be remaining in each queue.
+	 */
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, flush, NULL)) {
+		printf("%d: Error installing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+
+	if (count != rte_mempool_avail_count(t->mbuf_pool)) {
+		printf("%d: Error executing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, NULL, NULL)) {
+		printf("%d: Error uninstalling the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	return 0;
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_single_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Create a directed port */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) == 0) {
+		printf("%d: port 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Create a load-balanced port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a directed queue */
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed queue */
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) == 0) {
+		printf("%d: queue 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a load-balanced queue */
+	queue_conf.event_queue_cfg = 0;
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to link directed and load-balanced resources */
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 0 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 1 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Link ports to queues */
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+#define NUM_LDB_PORTS 64
+#define NUM_LDB_QUEUES 32
+
+static int
+test_info_get(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	config.nb_event_ports = info.max_event_ports;
+	config.nb_event_queues = NUM_LDB_QUEUES + info.max_event_ports / 2;
+	config.nb_single_link_event_port_queues = info.max_event_ports / 2;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		goto err;
+	}
+
+	/* The DLB2 PMD only reports load-balanced ports and queues in its
+	 * info_get function. Confirm that these values don't include the
+	 * directed port or queue counts.
+	 */
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret) {
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+		return -1;
+	}
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+static int
+test_reconfiguration_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 2 LDB queues */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+			printf("%d: P0->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P1->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+			printf("%d: P1->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Configure device with 2 DIR ports and 2 DIR queues */
+	config.nb_single_link_event_port_queues = 2;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q1 */
+	queue_id = 1;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: P0->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Link P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: P1->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_queues = 5;
+	config.nb_event_ports = 5;
+	config.nb_single_link_event_port_queues = 1;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < config.nb_event_queues - 1; i++) {
+		port_conf.event_port_cfg = 0;
+		queue_conf.event_queue_cfg = 0;
+
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: P%d->Q%d link expected to succeed\n",
+			       __LINE__, i, i);
+			goto err;
+		}
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+		printf("%d: port %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+		printf("%d: queue %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	queue_id = i;
+
+	if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+		printf("%d: P%d->Q%d link expected to succeed\n",
+		       __LINE__, i, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_ports += 1;
+
+	/* Reconfigure device with 1 more load-balanced port */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Configure the new port */
+	if (rte_event_port_setup(evdev, config.nb_event_ports - 1,
+				 &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_load_balanced_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_directed_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 DIR port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (ev.queue_id != 0) {
+		printf("%d: invalid dequeued event queue ID (%d)\n",
+		       __LINE__, ev.queue_id);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_deferred_sched(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	const int num_events = 128;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 1 queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 1, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.dequeue_depth = 1;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.schedule_type = RTE_SCHED_TYPE_PARALLEL;
+	queue_conf.nb_atomic_order_sequences = 0;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 128 NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_PARALLEL;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue one event from port 0 */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue (and release) all other events from port 1. Deferred
+	 * scheduling ensures no other events are scheduled to port 0 without a
+	 * subsequent rte_event_dequeue_burst() call.
+	 */
+	for (i = 0; i < num_events - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 1, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		if (rte_event_enqueue_burst(evdev, 1, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_delayed_pop(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	int ret, i, num_events;
+	struct rte_event ev;
+	uint8_t queue_id;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DELAYED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	num_events = 2 * port_conf.dequeue_depth;
+
+	/* Enqueue 2 * dequeue_depth NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue dequeue_depth events but only release dequeue_depth - 2.
+	 * Delayed pop won't perform the pop and no more events will be
+	 * scheduled.
+	 */
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = 0; i < port_conf.dequeue_depth - 2; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	ret = rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout);
+	if (ret != 0) {
+		printf("%d: event dequeue expected to fail (ret = %d)\n",
+		       __LINE__, ret);
+		goto err;
+	}
+
+	/* Release one more event. This will trigger the token pop, and
+	 * dequeue_depth - 1 more events will be scheduled to the device.
+	 */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: RELEASE enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 0) {
+		printf("%d: event dequeue expected to fail\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+do_selftest(void)
+{
+	struct test t;
+	int ret;
+
+	/* Only create mbuf pool once, reuse for each test run */
+	if (!eventdev_func_mempool) {
+		eventdev_func_mempool =
+			rte_pktmbuf_pool_create("EVENTDEV_DLB2_ST_POOL",
+						(1 << 12), /* 4k buffers */
+						32 /*MBUF_CACHE_SIZE*/,
+						0,
+						512, /* use very small mbufs */
+						rte_socket_id());
+		if (!eventdev_func_mempool) {
+			printf("ERROR creating mempool\n");
+			goto test_fail;
+		}
+	}
+	t.mbuf_pool = eventdev_func_mempool;
+
+	printf("*** Running Stop Flush test...\n");
+	ret = test_stop_flush(&t);
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Single Link test...\n");
+	ret = test_single_link();
+	if (ret != 0) {
+		printf("ERROR - Single Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Info Get test...\n");
+	ret = test_info_get();
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Reconfiguration Link test...\n");
+	ret = test_reconfiguration_link();
+	if (ret != 0) {
+		printf("ERROR - Reconfiguration Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Load-Balanced Traffic test...\n");
+	ret = test_load_balanced_traffic();
+	if (ret != 0) {
+		printf("ERROR - Load-Balanced Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Directed Traffic test...\n");
+	ret = test_directed_traffic();
+	if (ret != 0) {
+		printf("ERROR - Directed Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Deferred Scheduling test...\n");
+	ret = test_deferred_sched();
+	if (ret != 0) {
+		printf("ERROR - Deferred Scheduling test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Delayed Pop test...\n");
+	ret = test_delayed_pop();
+	if (ret != 0) {
+		printf("ERROR - Delayed Pop test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	return 0;
+
+test_fail:
+	return -1;
+}
+
+int
+test_dlb2_eventdev(void)
+{
+	const char *dlb2_eventdev_name = "dlb2_event";
+	uint8_t num_evdevs = rte_event_dev_count();
+	int i, ret = 0;
+	int found = 0, skipped = 0, passed = 0, failed = 0;
+	struct rte_event_dev_info info;
+
+	for (i = 0; found + skipped < num_evdevs && i < RTE_EVENT_MAX_DEVS;
+	     i++) {
+		ret = rte_event_dev_info_get(i, &info);
+		if (ret < 0)
+			continue;
+
+		/* skip non-dlb2 event devices */
+		if (strncmp(info.driver_name, dlb2_eventdev_name,
+			    sizeof(*info.driver_name)) != 0) {
+			skipped++;
+			continue;
+		}
+
+		evdev = rte_event_dev_get_dev_id(info.driver_name);
+		if (evdev < 0) {
+			printf("Could not get dev_id for eventdev with name %s, i=%d\n",
+			       info.driver_name, i);
+			skipped++;
+			continue;
+		}
+		found++;
+		printf("Running selftest on eventdev %s\n", info.driver_name);
+		ret = do_selftest();
+		if (ret == 0) {
+			passed++;
+			printf("Selftest passed for eventdev %s\n",
+			       info.driver_name);
+		} else {
+			failed++;
+			printf("Selftest failed for eventdev %s, err=%d\n",
+			       info.driver_name, ret);
+		}
+	}
+
+	printf("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n",
+	       found, skipped, passed, failed);
+	return ret;
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index b734ff6..340bc57 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -14,7 +14,8 @@ sources = files('dlb2.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c',
-		'rte_pmd_dlb2.c'
+		'rte_pmd_dlb2.c',
+		'dlb2_selftest.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v3 22/23] event/dlb2: add queue and port release

[Timothy McDaniel]

DLB does not support reconfiguring individual queues
or ports on the fly. The entire device must be reconfigured.
Previously allocated port QE and memzone memory
is freed in this patch.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 113fedf..4b4719f 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3672,6 +3672,28 @@ dlb2_eventdev_close(struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(id);
+
+	/* This function intentionally left blank. */
+}
+
+static void
+dlb2_eventdev_port_release(void *port)
+{
+	struct dlb2_eventdev_port *ev_port = port;
+	struct dlb2_port *qm_port;
+
+	if (ev_port) {
+		qm_port = &ev_port->qm_port;
+		if (qm_port->config_state == DLB2_CONFIGURED)
+			dlb2_free_qe_mem(qm_port);
+	}
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3685,8 +3707,10 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
+		.queue_release    = dlb2_eventdev_queue_release,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_release     = dlb2_eventdev_port_release,
 		.port_link        = dlb2_eventdev_port_link,
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
-- 
2.6.4

[dpdk-dev] [PATCH v3 23/23] event/dlb2: add timeout ticks entry point

[Timothy McDaniel]

Adds the timeout ticks conversion function.

Adds announcement of availabililty for the new driver
for Intel Dynamic Load Balancer 2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/rel_notes/release_20_11.rst |  5 +++++
 drivers/event/dlb2/dlb2.c              | 13 +++++++++++++
 2 files changed, 18 insertions(+)

diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
index df227a1..f80df21 100644
--- a/doc/guides/rel_notes/release_20_11.rst
+++ b/doc/guides/rel_notes/release_20_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
+
+  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
+  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
+
 
 Removed Items
 -------------
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 4b4719f..dea61fc 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3693,6 +3693,18 @@ dlb2_eventdev_port_release(void *port)
 	}
 }
 
+static int
+dlb2_eventdev_timeout_ticks(struct rte_eventdev *dev, uint64_t ns,
+			    uint64_t *timeout_ticks)
+{
+	RTE_SET_USED(dev);
+	uint64_t cycles_per_ns = rte_get_timer_hz() / 1E9;
+
+	*timeout_ticks = ns * cycles_per_ns;
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -3715,6 +3727,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
 				    dlb2_eventdev_port_unlinks_in_progress,
+		.timeout_ticks    = dlb2_eventdev_timeout_ticks,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v4 00/23] Add DLB2 PMD

[Timothy McDaniel]

The following patch series adds support for a new eventdev PMD. The DLB2
PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
hardware.
The DLB2 is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB2 hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB2 ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a
directed queue. Additionally, even though LDB ports may link multiple
queues,
the number of queues that may be linked is limited by hardware.

While reviewing the code, please be aware that this PMD has full
control over the DLB2 hardware. Intel will be extending the DLB2 PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.
Note that the DLB2 hardware is a successor of the DLB hardware, and
as such is structured similarly, both in terms of code layout and
implementation.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Changes since V3:
================
- updated MAINTAINERS file to alphabetically insert DLB
- don't create RTE_ symbols in pmd
- converted to use version.map scheme
- converted to use .._master_lcore instead of .._main_lcore
- this patch set is based on dpdk-next-eventdev

Changes since V2:
================
- fixed meson conditional build. Moved test into driver’s meson.build
  file instead of event/meson.build
- documentation is populated as associated code is introduced
- add log_register in add dynamic logging patch
- rename RTE_xxx symbol(s) as DLB2_xxx
- replaced function ptr enqueue_four with direct call to movdir64b
- remove unused port_pages
- broke up probe patch into 3 smaller patches for easier review
- changed param order of movdir64b/movntdq to match intrinsics
- added self to MAINTAINERS files
- squashed announcement of availability into last patch in series
- correct spelling errors and delete repeated words
- DPDK_21.0 -> DPDK 21 in map file
- add experimental banner to public structs and APIs

Changes since V1:
=================
- implement changes requested in code reviews by Gage Eads and Mike
  Chen
- fix a memzone leak
- convert to use eal rte-cpuflags patch from Liang Ma

Known Issues:
- the documentation contained in dlb2.rst is not complete, and
  should be updated to include command line parameters (class of service,
        etc ...).

Depends-on: patch-82202 ("eventdev: increase MAX QUEUES PER DEV to 255")
Depends-on: patch-79539 ("eal: add new x86 cpuid support for WAITPKG")

Timothy McDaniel (23):
  event/dlb2: add documentation and meson build infrastructure
  event/dlb2: add dynamic logging
  event/dlb2: add private data structures and constants
  event/dlb2: add definitions shared with LKM or shared code
  event/dlb2: add inline functions
  event/dlb2: add eventdev probe
  event/dlb2: add flexible interface
  event/dlb2: add probe-time hardware init
  event/dlb2: add xstats
  event/dlb2: add infos get and configure
  event/dlb2: add queue and port default conf
  event/dlb2: add queue setup
  event/dlb2: add port setup
  event/dlb2: add port link
  event/dlb2: add port unlink and port unlinks in progress
  event/dlb2: add eventdev start
  event/dlb2: add enqueue and its burst variants
  event/dlb2: add dequeue and its burst variants
  event/dlb2: add eventdev stop and close
  event/dlb2: add PMD's token pop public interface
  event/dlb2: add PMD self-tests
  event/dlb2: add queue and port release
  event/dlb2: add timeout ticks entry point

 MAINTAINERS                                    |    6 +-
 app/test/test_eventdev.c                       |    7 +
 config/rte_config.h                            |    7 +
 doc/api/doxy-api-index.md                      |    1 +
 doc/guides/eventdevs/dlb2.rst                  |  330 ++
 doc/guides/eventdevs/index.rst                 |    1 +
 doc/guides/rel_notes/release_20_11.rst         |    5 +
 drivers/event/dlb2/dlb2.c                      | 3951 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c                |   74 +
 drivers/event/dlb2/dlb2_iface.h                |   74 +
 drivers/event/dlb2/dlb2_inline_fns.h           |   61 +
 drivers/event/dlb2/dlb2_log.h                  |   25 +
 drivers/event/dlb2/dlb2_priv.h                 |  578 +++
 drivers/event/dlb2/dlb2_selftest.c             | 1570 ++++++
 drivers/event/dlb2/dlb2_user.h                 |  679 +++
 drivers/event/dlb2/dlb2_xstats.c               | 1240 +++++
 drivers/event/dlb2/meson.build                 |   22 +
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 +++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 +
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c     | 6027 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  673 +++
 drivers/event/dlb2/pf/dlb2_main.h              |   97 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  728 +++
 drivers/event/dlb2/rte_pmd_dlb2.c              |   39 +
 drivers/event/dlb2/rte_pmd_dlb2.h              |   72 +
 drivers/event/dlb2/version.map                 |    9 +
 drivers/event/meson.build                      |    3 +-
 33 files changed, 22512 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
 create mode 100644 drivers/event/dlb2/dlb2_log.h
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c
 create mode 100644 drivers/event/dlb2/dlb2_user.h
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
 create mode 100644 drivers/event/dlb2/version.map

-- 
2.6.4

[dpdk-dev] [PATCH v4 01/23] event/dlb2: add documentation and meson build infrastructure

[Timothy McDaniel]

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux X86 platforms at this time.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 MAINTAINERS                    |  6 +++++-
 config/rte_config.h            |  7 +++++++
 doc/guides/eventdevs/dlb2.rst  | 40 ++++++++++++++++++++++++++++++++++++++++
 doc/guides/eventdevs/index.rst |  1 +
 drivers/event/dlb2/meson.build | 13 +++++++++++++
 drivers/event/dlb2/version.map |  3 +++
 drivers/event/meson.build      |  3 ++-
 7 files changed, 71 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb2.rst
 create mode 100644 drivers/event/dlb2/meson.build
 create mode 100644 drivers/event/dlb2/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index a3d1927..727451b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1162,6 +1162,11 @@ Cavium OCTEON TX timvf
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 F: drivers/event/octeontx/timvf_*
 
+Intel DLB2
+M: Timothy McDaniel <timothy.mcdaniel@intel.com>
+F: drivers/event/dlb2/
+F: doc/guides/eventdevs/dlb2.rst
+
 Marvell OCTEON TX2
 M: Pavan Nikhilesh <pbhagavatula@marvell.com>
 M: Jerin Jacob <jerinj@marvell.com>
@@ -1198,7 +1203,6 @@ M: Peter Mccarthy <peter.mccarthy@intel.com>
 F: drivers/event/opdl/
 F: doc/guides/eventdevs/opdl.rst
 
-
 Rawdev Drivers
 --------------
 
diff --git a/config/rte_config.h b/config/rte_config.h
index b78c6aa..64aa333 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -135,4 +135,11 @@
 /* QEDE PMD defines */
 #define RTE_LIBRTE_QEDE_FW ""
 
+/* DLB2 defines */
+#define RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL 1000
+#define RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE  0
+#undef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
+#define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
+
 #endif /* _RTE_CONFIG_H_ */
diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
new file mode 100644
index 0000000..e3091c9
--- /dev/null
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -0,0 +1,40 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+Driver for the Intel® Dynamic Load Balancer (DLB2)
+==================================================
+
+The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.
+
+Prerequisites
+-------------
+
+Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup
+the basic DPDK environment.
+
+Configuration
+-------------
+
+The DLB2 PF PMD is a user-space PMD that uses VFIO to gain direct
+device access. To use this operation mode, the PCIe PF device must be bound
+to a DPDK-compatible VFIO driver, such as vfio-pci.
+
+Eventdev API Notes
+------------------
+
+The DLB2 provides the functions of a DPDK event device; specifically, it
+supports atomic, ordered, and parallel scheduling events from queues to ports.
+However, the DLB2 hardware is not a perfect match to the eventdev API. Some DLB2
+features are abstracted by the PMD such as directed ports.
+
+In general the dlb PMD is designed for ease-of-use and does not require a
+detailed understanding of the hardware, but these details are important when
+writing high-performance code. This section describes the places where the
+eventdev API and DLB2 misalign.
+
+Flow ID
+~~~~~~~
+
+The flow ID field is preserved in the event when it is scheduled in the
+DLB2.
+
diff --git a/doc/guides/eventdevs/index.rst b/doc/guides/eventdevs/index.rst
index bb66a5e..738788d 100644
--- a/doc/guides/eventdevs/index.rst
+++ b/doc/guides/eventdevs/index.rst
@@ -11,6 +11,7 @@ application through the eventdev API.
     :maxdepth: 2
     :numbered:
 
+    dlb2
     dpaa
     dpaa2
     dsw
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
new file mode 100644
index 0000000..5324043
--- /dev/null
+++ b/drivers/event/dlb2/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2020 Intel Corporation
+
+if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
+        build = false
+        reason = 'only supported on ARCH_X86_64 Linux'
+        subdir_done()
+endif
+
+sources = files(
+)
+
+deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
new file mode 100644
index 0000000..4a76d1d
--- /dev/null
+++ b/drivers/event/dlb2/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
diff --git a/drivers/event/meson.build b/drivers/event/meson.build
index a7dac99..c433c10 100644
--- a/drivers/event/meson.build
+++ b/drivers/event/meson.build
@@ -5,7 +5,8 @@ if is_windows
 	subdir_done()
 endif
 
-drivers = ['dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw', 'dsw']
+drivers = ['dlb2', 'dpaa', 'dpaa2', 'octeontx2', 'opdl', 'skeleton', 'sw',
+	   'dsw']
 if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and
 	dpdk_conf.has('RTE_ARCH_ARM64'))
 	drivers += 'octeontx'
-- 
2.6.4

[dpdk-dev] [PATCH v4 02/23] event/dlb2: add dynamic logging

[Timothy McDaniel]

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c      |  7 +++++++
 drivers/event/dlb2/dlb2_log.h  | 25 +++++++++++++++++++++++++
 drivers/event/dlb2/meson.build |  3 +--
 3 files changed, 33 insertions(+), 2 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_log.h

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
new file mode 100644
index 0000000..2082dd6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2.c
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <rte_log.h>
+
+RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
new file mode 100644
index 0000000..dc1481e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_log.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_EVDEV_LOG_H_
+#define _DLB2_EVDEV_LOG_H_
+
+extern int eventdev_dlb2_log_level;
+
+/* Dynamic logging */
+#define DLB2_LOG_IMPL(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
+		__func__, ##args)
+
+#define DLB2_LOG_INFO(fmt, args...) \
+	DLB2_LOG_IMPL(INFO, fmt, ## args)
+
+#define DLB2_LOG_ERR(fmt, args...) \
+	DLB2_LOG_IMPL(ERR, fmt, ## args)
+
+/* remove debug logs at compile time unless actually debugging */
+#define DLB2_LOG_DBG(fmt, args...) \
+	RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
+
+#endif /* _DLB2_EVDEV_LOG_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 5324043..435c1ee 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,7 +7,6 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files(
-)
+sources = files('dlb2.c')
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v4 03/23] event/dlb2: add private data structures and constants

[Timothy McDaniel]

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_priv.h | 575 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 575 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_priv.h

diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
new file mode 100644
index 0000000..61567a6
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -0,0 +1,575 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_PRIV_H_
+#define _DLB2_PRIV_H_
+
+#include <emmintrin.h>
+#include <stdbool.h>
+
+#include <rte_eventdev.h>
+#include <rte_config.h>
+#include "dlb2_user.h"
+#include "dlb2_log.h"
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
+#else
+#define DLB2_INC_STAT(_stat, _incr_val)
+#endif
+
+#define EVDEV_DLB2_NAME_PMD dlb2_event
+
+/*  command line arg strings */
+#define NUMA_NODE_ARG "numa_node"
+#define DLB2_MAX_NUM_EVENTS "max_num_events"
+#define DLB2_NUM_DIR_CREDITS "num_dir_credits"
+#define DEV_ID_ARG "dev_id"
+#define DLB2_DEFER_SCHED_ARG "defer_sched"
+#define DLB2_QID_DEPTH_THRESH_ARG "qid_depth_thresh"
+#define DLB2_COS_ARG "cos"
+
+/* Begin HW related defines and structs */
+
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_VFS 16
+#define DLB2_MAX_NUM_LDB_QUEUES 32
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS 64
+#define DLB2_MAX_NUM_DIR_QUEUES 64
+#define DLB2_MAX_NUM_FLOWS (64 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDITS (8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS (2 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_DIR_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQOS_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_MAX_DEVICE_PATH 32
+#define DLB2_MIN_DEQUEUE_TIMEOUT_NS 1
+/* Note: "- 1" here to support the timeout range check in eventdev_autotest */
+#define DLB2_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1)
+#define DLB2_SW_CREDIT_BATCH_SZ 32
+#define DLB2_NUM_SN_GROUPS 2
+#define DLB2_MAX_LDB_SN_ALLOC 1024
+#define DLB2_MAX_QUEUE_DEPTH_THRESHOLD 8191
+
+/* 2048 total hist list entries and 64 total ldb ports, which
+ * makes for 2048/64 == 32 hist list entries per port. However, CQ
+ * depth must be a power of 2 and must also be >= HIST LIST entries.
+ * As a result we just limit the maximum dequeue depth to 32.
+ */
+#define DLB2_MIN_CQ_DEPTH 1
+#define DLB2_MAX_CQ_DEPTH 32
+#define DLB2_MIN_HARDWARE_CQ_DEPTH 8
+#define DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \
+	DLB2_MAX_CQ_DEPTH
+
+/*
+ * Static per queue/port provisioning values
+ */
+#define DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 64
+
+#define CQ_BASE(is_dir) ((is_dir) ? DLB2_DIR_CQ_BASE : DLB2_LDB_CQ_BASE)
+#define CQ_SIZE(is_dir) ((is_dir) ? DLB2_DIR_CQ_MAX_SIZE : \
+				    DLB2_LDB_CQ_MAX_SIZE)
+#define PP_BASE(is_dir) ((is_dir) ? DLB2_DIR_PP_BASE : DLB2_LDB_PP_BASE)
+
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+
+#define DLB2_NUM_QES_PER_CACHE_LINE 4
+
+#define DLB2_MAX_ENQUEUE_DEPTH 64
+#define DLB2_MIN_ENQUEUE_DEPTH 4
+
+#define DLB2_NAME_SIZE 64
+
+#define DLB2_1K 1024
+#define DLB2_2K (2 * DLB2_1K)
+#define DLB2_4K (4 * DLB2_1K)
+#define DLB2_16K (16 * DLB2_1K)
+#define DLB2_32K (32 * DLB2_1K)
+#define DLB2_1MB (DLB2_1K * DLB2_1K)
+#define DLB2_16MB (16 * DLB2_1MB)
+
+/* Use the upper 3 bits of the event priority to select the DLB2 priority */
+#define EV_TO_DLB2_PRIO(x) ((x) >> 5)
+#define DLB2_TO_EV_PRIO(x) ((x) << 5)
+
+enum dlb2_hw_port_types {
+	DLB2_LDB_PORT,
+	DLB2_DIR_PORT,
+	DLB2_NUM_PORT_TYPES /* Must be last */
+};
+
+enum dlb2_hw_queue_types {
+	DLB2_LDB_QUEUE,
+	DLB2_DIR_QUEUE,
+	DLB2_NUM_QUEUE_TYPES /* Must be last */
+};
+
+#define PORT_TYPE(p) ((p)->is_directed ? DLB2_DIR_PORT : DLB2_LDB_PORT)
+
+/* Do not change - must match hardware! */
+enum dlb2_hw_sched_type {
+	DLB2_SCHED_ATOMIC = 0,
+	DLB2_SCHED_UNORDERED,
+	DLB2_SCHED_ORDERED,
+	DLB2_SCHED_DIRECTED,
+	/* DLB2_NUM_HW_SCHED_TYPES must be last */
+	DLB2_NUM_HW_SCHED_TYPES
+};
+
+struct dlb2_hw_rsrcs {
+	int32_t nb_events_limit;
+	uint32_t num_queues;		/* Total queues (lb + dir) */
+	uint32_t num_ldb_queues;	/* Number of available ldb queues */
+	uint32_t num_ldb_ports;         /* Number of load balanced ports */
+	uint32_t num_dir_ports;         /* Number of directed ports */
+	uint32_t num_ldb_credits;       /* Number of load balanced credits */
+	uint32_t num_dir_credits;       /* Number of directed credits */
+	uint32_t reorder_window_size;   /* Size of reorder window */
+};
+
+struct dlb2_hw_resource_info {
+	/**> Max resources that can be provided */
+	struct dlb2_hw_rsrcs hw_rsrc_max;
+	int num_sched_domains;
+	uint32_t socket_id;
+};
+
+enum dlb2_enqueue_type {
+	/**>
+	 * New : Used to inject a new packet into the QM.
+	 */
+	DLB2_ENQ_NEW,
+	/**>
+	 * Forward : Enqueues a packet, and
+	 *  - if atomic: release any lock it holds in the QM
+	 *  - if ordered: release the packet for egress re-ordering
+	 */
+	DLB2_ENQ_FWD,
+	/**>
+	 * Enqueue Drop : Release an inflight packet. Must be called with
+	 * event == NULL. Used to drop a packet.
+	 *
+	 * Note that all packets dequeued from a load-balanced port must be
+	 * released, either with DLB2_ENQ_DROP or DLB2_ENQ_FWD.
+	 */
+	DLB2_ENQ_DROP,
+
+	/* marker for array sizing etc. */
+	_DLB2_NB_ENQ_TYPES
+};
+
+/* hw-specific format - do not change */
+
+struct dlb2_event_type {
+	uint8_t major:4;
+	uint8_t unused:4;
+	uint8_t sub;
+};
+
+union dlb2_opaque_data {
+	uint16_t opaque_data;
+	struct dlb2_event_type event_type;
+};
+
+struct dlb2_msg_info {
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+};
+
+#define DLB2_NEW_CMD_BYTE 0x08
+#define DLB2_FWD_CMD_BYTE 0x0A
+#define DLB2_COMP_CMD_BYTE 0x02
+#define DLB2_POP_CMD_BYTE 0x01
+#define DLB2_NOOP_CMD_BYTE 0x00
+
+/* hw-specific format - do not change */
+struct dlb2_enqueue_qe {
+	uint64_t data;
+	/* Word 3 */
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	/* Word 4 */
+	uint16_t lock_id;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_cq_pop_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t tokens:10;
+	uint16_t rsvd2:6;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_dequeue_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t flow_id:16; /* was pp_id in v1 */
+	uint8_t debug;
+	uint8_t cq_gen:1;
+	uint8_t qid_depth:2; /* 2 bits in v2 */
+	uint8_t rsvd1:2;
+	uint8_t error:1;
+	uint8_t rsvd2:2;
+};
+
+union dlb2_port_config {
+	struct dlb2_create_ldb_port_args ldb;
+	struct dlb2_create_dir_port_args dir;
+};
+
+enum dlb2_port_state {
+	PORT_CLOSED,
+	PORT_STARTED,
+	PORT_STOPPED
+};
+
+enum dlb2_configuration_state {
+	/* The resource has not been configured */
+	DLB2_NOT_CONFIGURED,
+	/* The resource was configured, but the device was stopped */
+	DLB2_PREV_CONFIGURED,
+	/* The resource is currently configured */
+	DLB2_CONFIGURED
+};
+
+struct dlb2_port {
+	uint32_t id;
+	bool is_directed;
+	bool gen_bit;
+	uint16_t dir_credits;
+	uint32_t dequeue_depth;
+	union dlb2_port_config cfg;
+	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
+	uint16_t cached_ldb_credits;
+	uint16_t ldb_credits;
+	uint16_t cached_dir_credits;
+	bool int_armed;
+	uint16_t owed_tokens;
+	int16_t issued_releases;
+	int cq_depth;
+	uint16_t cq_idx;
+	uint16_t cq_idx_unmasked;
+	uint16_t cq_depth_mask;
+	uint16_t gen_bit_shift;
+	enum dlb2_port_state state;
+	enum dlb2_configuration_state config_state;
+	int num_mapped_qids;
+	uint8_t *qid_mappings;
+	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
+	struct dlb2_enqueue_qe *int_arm_qe;
+	struct dlb2_cq_pop_qe *consume_qe;
+	struct dlb2_eventdev *dlb2; /* back ptr */
+	struct dlb2_eventdev_port *ev_port; /* back ptr */
+};
+
+/* Per-process per-port mmio and memory pointers */
+struct process_local_port_data {
+	uint64_t *pp_addr;
+	struct dlb2_dequeue_qe *cq_base;
+	const struct rte_memzone *mz;
+	bool mmaped;
+};
+
+struct dlb2_eventdev;
+
+struct dlb2_config {
+	int configured;
+	int reserved;
+	uint32_t num_ldb_credits;
+	uint32_t num_dir_credits;
+	struct dlb2_create_sched_domain_args resources;
+};
+
+enum dlb2_cos {
+	DLB2_COS_DEFAULT = -1,
+	DLB2_COS_0 = 0,
+	DLB2_COS_1,
+	DLB2_COS_2,
+	DLB2_COS_3
+};
+
+struct dlb2_hw_dev {
+	struct dlb2_config cfg;
+	struct dlb2_hw_resource_info info;
+	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
+	uint32_t domain_id;
+	enum dlb2_cos cos_id;
+	rte_spinlock_t resource_lock; /* for MP support */
+} __rte_cache_aligned;
+
+/* End HW related defines and structs */
+
+/* Begin DLB2 PMD Eventdev related defines and structs */
+
+#define DLB2_MAX_NUM_QUEUES \
+	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
+
+#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS + DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
+
+/** Structure to hold the queue to port link establishment attributes */
+
+struct dlb2_event_queue_link {
+	uint8_t queue_id;
+	uint8_t priority;
+	bool mapped;
+	bool valid;
+};
+
+struct dlb2_traffic_stats {
+	uint64_t rx_ok;
+	uint64_t rx_drop;
+	uint64_t rx_interrupt_wait;
+	uint64_t rx_umonitor_umwait;
+	uint64_t tx_ok;
+	uint64_t total_polls;
+	uint64_t zero_polls;
+	uint64_t tx_nospc_ldb_hw_credits;
+	uint64_t tx_nospc_dir_hw_credits;
+	uint64_t tx_nospc_inflight_max;
+	uint64_t tx_nospc_new_event_limit;
+	uint64_t tx_nospc_inflight_credits;
+};
+
+/* DLB2 HW sets the 2bit qid_depth in rx QEs based on the programmable depth
+ * threshold. The global default value in config/common_base (or rte_config.h)
+ * can be overridden on a per-qid basis using a vdev command line parameter.
+ * 3: depth > threshold
+ * 2: threshold >= depth > 3/4 threshold
+ * 1: 3/4 threshold >= depth > 1/2 threshold
+ * 0: depth <= 1/2 threshold.
+ */
+#define DLB2_QID_DEPTH_LE50 0
+#define DLB2_QID_DEPTH_GT50_LE75 1
+#define DLB2_QID_DEPTH_GT75_LE100 2
+#define DLB2_QID_DEPTH_GT100 3
+#define DLB2_NUM_QID_DEPTH_STAT_VALS 4 /* 2 bits */
+
+struct dlb2_queue_stats {
+	uint64_t enq_ok;
+	uint64_t qid_depth[DLB2_NUM_QID_DEPTH_STAT_VALS];
+};
+
+struct dlb2_port_stats {
+	struct dlb2_traffic_stats traffic;
+	uint64_t tx_op_cnt[4]; /* indexed by rte_event.op */
+	uint64_t tx_implicit_rel;
+	uint64_t tx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t tx_invalid;
+	uint64_t rx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t rx_sched_invalid;
+	struct dlb2_queue_stats queue[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_eventdev_port {
+	struct dlb2_port qm_port; /* hw specific data structure */
+	struct rte_event_port_conf conf; /* user-supplied configuration */
+	uint16_t inflight_credits; /* num credits this port has right now */
+	uint16_t credit_update_quanta;
+	struct dlb2_eventdev *dlb2; /* backlink optimization */
+	struct dlb2_port_stats stats __rte_cache_aligned;
+	struct dlb2_event_queue_link link[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	int num_links;
+	uint32_t id; /* port id */
+	/* num releases yet to be completed on this port.
+	 * Only applies to load-balanced ports.
+	 */
+	uint16_t outstanding_releases;
+	uint16_t inflight_max; /* app requested max inflights for this port */
+	/* setup_done is set when the event port is setup */
+	bool setup_done;
+	/* enq_configured is set when the qm port is created */
+	bool enq_configured;
+	uint8_t implicit_release; /* release events before dequeueing */
+}  __rte_cache_aligned;
+
+struct dlb2_queue {
+	uint32_t num_qid_inflights; /* User config */
+	uint32_t num_atm_inflights; /* User config */
+	enum dlb2_configuration_state config_state;
+	int sched_type; /* LB queue only */
+	uint32_t id;
+	bool is_directed;
+};
+
+struct dlb2_eventdev_queue {
+	struct dlb2_queue qm_queue;
+	struct rte_event_queue_conf conf; /* User config */
+	int depth_threshold; /* use default if 0 */
+	uint32_t id;
+	bool setup_done;
+	uint8_t num_links;
+};
+
+enum dlb2_run_state {
+	DLB2_RUN_STATE_STOPPED = 0,
+	DLB2_RUN_STATE_STOPPING,
+	DLB2_RUN_STATE_STARTING,
+	DLB2_RUN_STATE_STARTED
+};
+
+struct dlb2_eventdev {
+	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
+	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each queue */
+	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
+	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each port */
+	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
+	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
+	struct dlb2_get_num_resources_args hw_rsrc_query_results;
+	uint32_t xstats_count_mode_queue;
+	struct dlb2_hw_dev qm_instance; /* strictly hw related */
+	uint64_t global_dequeue_wait_ticks;
+	struct dlb2_xstats_entry *xstats;
+	struct rte_eventdev *event_dev; /* backlink to dev */
+	uint32_t xstats_count_mode_dev;
+	uint32_t xstats_count_mode_port;
+	uint32_t xstats_count;
+	uint32_t inflights; /* use __atomic builtins */
+	uint32_t new_event_limit;
+	int max_num_events_override;
+	int num_dir_credits_override;
+	volatile enum dlb2_run_state run_state;
+	uint16_t num_dir_queues; /* total num of evdev dir queues requested */
+	uint16_t num_dir_credits;
+	uint16_t num_ldb_credits;
+	uint16_t num_queues; /* total queues */
+	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested */
+	uint16_t num_ports; /* total num of evdev ports requested */
+	uint16_t num_ldb_ports; /* total num of ldb ports requested */
+	uint16_t num_dir_ports; /* total num of dir ports requested */
+	bool umwait_allowed;
+	bool global_dequeue_wait; /* Not using per dequeue wait if true */
+	bool defer_sched;
+	enum dlb2_cq_poll_modes poll_mode;
+	uint8_t revision;
+	bool configured;
+	uint16_t max_ldb_credits;
+	uint16_t max_dir_credits;
+
+	/* force hw credit pool counters into exclusive cache lines */
+
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t ldb_credit_pool __rte_cache_aligned;
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t dir_credit_pool __rte_cache_aligned;
+};
+
+/* used for collecting and passing around the dev args */
+struct dlb2_qid_depth_thresholds {
+	int val[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_devargs {
+	int socket_id;
+	int max_num_events;
+	int num_dir_credits_override;
+	int dev_id;
+	int defer_sched;
+	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
+	enum dlb2_cos cos_id;
+};
+
+/* End Eventdev related defines and structs */
+
+/* Forwards for non-inlined functions */
+
+void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
+
+int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
+
+void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
+
+int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n);
+
+int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstat_names,
+		unsigned int *ids, unsigned int size);
+
+uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+					  const char *name, unsigned int *id);
+
+int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode,
+		int16_t queue_port_id,
+		const uint32_t ids[],
+		uint32_t nb_ids);
+
+int test_dlb2_eventdev(void);
+
+int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+				const char *name,
+				struct dlb2_devargs *dlb2_args);
+
+int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+				  const char *name);
+
+uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *queue);
+
+int dlb2_parse_params(const char *params,
+		      const char *name,
+		      struct dlb2_devargs *dlb2_args);
+
+/* Extern globals */
+extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
+
+#endif	/* _DLB2_PRIV_H_ */
-- 
2.6.4

Re: [dpdk-dev] [PATCH v4 03/23] event/dlb2: add private data structures and constants

[Stephen Hemminger]

On Thu, 29 Oct 2020 10:24:57 -0500
Timothy McDaniel <timothy.mcdaniel@intel.com> wrote:

> +
> +	/* marker for array sizing etc. */
> +	_DLB2_NB_ENQ_TYPES

Be careful with this type of array sizing value.
It becomes a breaking point for any API/ABI changes.

Re: [dpdk-dev] [PATCH v4 03/23] event/dlb2: add private data structures and constants

[McDaniel, Timothy]

> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Thursday, October 29, 2020 10:29 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dev@dpdk.org; Carrillo, Erik G <Erik.G.Carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com; thomas@monjalon.net
> Subject: Re: [dpdk-dev] [PATCH v4 03/23] event/dlb2: add private data
> structures and constants
> 
> On Thu, 29 Oct 2020 10:24:57 -0500
> Timothy McDaniel <timothy.mcdaniel@intel.com> wrote:
> 
> > +
> > +	/* marker for array sizing etc. */
> > +	_DLB2_NB_ENQ_TYPES
> 
> Be careful with this type of array sizing value.
> It becomes a breaking point for any API/ABI changes.

Thanks for the comment Stephen.   We do not expect these enums to change in the future.

Re: [dpdk-dev] [PATCH v4 03/23] event/dlb2: add private data structures and constants

[Stephen Hemminger]

On Thu, 29 Oct 2020 10:24:57 -0500
Timothy McDaniel <timothy.mcdaniel@intel.com> wrote:

> +struct dlb2_port {
> +	uint32_t id;
> +	bool is_directed;
> +	bool gen_bit;
> +	uint16_t dir_credits;
> +	uint32_t dequeue_depth;
> +	union dlb2_port_config cfg;
> +	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
> +	uint16_t cached_ldb_credits;
> +	uint16_t ldb_credits;
> +	uint16_t cached_dir_credits;
> +	bool int_armed;
> +	uint16_t owed_tokens;
> +	int16_t issued_releases;
> +	int cq_depth;
> +	uint16_t cq_idx;
> +	uint16_t cq_idx_unmasked;
> +	uint16_t cq_depth_mask;
> +	uint16_t gen_bit_shift;
> +	enum dlb2_port_state state;
> +	enum dlb2_configuration_state config_state;
> +	int num_mapped_qids;
> +	uint8_t *qid_mappings;
> +	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
> +	struct dlb2_enqueue_qe *int_arm_qe;
> +	struct dlb2_cq_pop_qe *consume_qe;
> +	struct dlb2_eventdev *dlb2; /* back ptr */
> +	struct dlb2_eventdev_port *ev_port; /* back ptr */
> +};

Is there any case where these int values could be negative?
Would you be better off using unsigned types?

Re: [dpdk-dev] [PATCH v4 03/23] event/dlb2: add private data structures and constants

[McDaniel, Timothy]

> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Thursday, October 29, 2020 10:31 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dev@dpdk.org; Carrillo, Erik G <Erik.G.Carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com; thomas@monjalon.net
> Subject: Re: [dpdk-dev] [PATCH v4 03/23] event/dlb2: add private data
> structures and constants
> 
> On Thu, 29 Oct 2020 10:24:57 -0500
> Timothy McDaniel <timothy.mcdaniel@intel.com> wrote:
> 
> > +struct dlb2_port {
> > +	uint32_t id;
> > +	bool is_directed;
> > +	bool gen_bit;
> > +	uint16_t dir_credits;
> > +	uint32_t dequeue_depth;
> > +	union dlb2_port_config cfg;
> > +	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic
> builtins */
> > +	uint16_t cached_ldb_credits;
> > +	uint16_t ldb_credits;
> > +	uint16_t cached_dir_credits;
> > +	bool int_armed;
> > +	uint16_t owed_tokens;
> > +	int16_t issued_releases;
> > +	int cq_depth;
> > +	uint16_t cq_idx;
> > +	uint16_t cq_idx_unmasked;
> > +	uint16_t cq_depth_mask;
> > +	uint16_t gen_bit_shift;
> > +	enum dlb2_port_state state;
> > +	enum dlb2_configuration_state config_state;
> > +	int num_mapped_qids;
> > +	uint8_t *qid_mappings;
> > +	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
> > +	struct dlb2_enqueue_qe *int_arm_qe;
> > +	struct dlb2_cq_pop_qe *consume_qe;
> > +	struct dlb2_eventdev *dlb2; /* back ptr */
> > +	struct dlb2_eventdev_port *ev_port; /* back ptr */
> > +};
> 
> Is there any case where these int values could be negative?
> Would you be better off using unsigned types?

These structure members should never go negative.  I believe that I could change them to
unsigned if required, but I don't think it really matters for these.

[dpdk-dev] [PATCH v4 04/23] event/dlb2: add definitions shared with LKM or shared code

[Timothy McDaniel]

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_user.h | 679 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 679 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_user.h

diff --git a/drivers/event/dlb2/dlb2_user.h b/drivers/event/dlb2/dlb2_user.h
new file mode 100644
index 0000000..f4bda78
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_user.h
@@ -0,0 +1,679 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#define DLB2_MAX_NAME_LEN 64
+
+#include <linux/types.h>
+
+enum dlb2_error {
+	DLB2_ST_SUCCESS = 0,
+	DLB2_ST_NAME_EXISTS,
+	DLB2_ST_DOMAIN_UNAVAILABLE,
+	DLB2_ST_LDB_PORTS_UNAVAILABLE,
+	DLB2_ST_DIR_PORTS_UNAVAILABLE,
+	DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB2_ST_INVALID_DOMAIN_ID,
+	DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB2_ST_INVALID_LDB_QUEUE_ID,
+	DLB2_ST_INVALID_CQ_DEPTH,
+	DLB2_ST_INVALID_CQ_VIRT_ADDR,
+	DLB2_ST_INVALID_PORT_ID,
+	DLB2_ST_INVALID_QID,
+	DLB2_ST_INVALID_PRIORITY,
+	DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+	DLB2_ST_INVALID_DIR_QUEUE_ID,
+	DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB2_ST_DOMAIN_NOT_CONFIGURED,
+	DLB2_ST_INTERNAL_ERROR,
+	DLB2_ST_DOMAIN_IN_USE,
+	DLB2_ST_DOMAIN_NOT_FOUND,
+	DLB2_ST_QUEUE_NOT_FOUND,
+	DLB2_ST_DOMAIN_STARTED,
+	DLB2_ST_DOMAIN_NOT_STARTED,
+	DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB2_ST_DOMAIN_RESET_FAILED,
+	DLB2_ST_MBOX_ERROR,
+	DLB2_ST_INVALID_HIST_LIST_DEPTH,
+	DLB2_ST_NO_MEMORY,
+	DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB2_ST_INVALID_COS_ID,
+};
+
+static const char dlb2_error_strings[][128] = {
+	"DLB2_ST_SUCCESS",
+	"DLB2_ST_NAME_EXISTS",
+	"DLB2_ST_DOMAIN_UNAVAILABLE",
+	"DLB2_ST_LDB_PORTS_UNAVAILABLE",
+	"DLB2_ST_DIR_PORTS_UNAVAILABLE",
+	"DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+	"DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+	"DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+	"DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+	"DLB2_ST_INVALID_DOMAIN_ID",
+	"DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+	"DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+	"DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+	"DLB2_ST_INVALID_LDB_QUEUE_ID",
+	"DLB2_ST_INVALID_CQ_DEPTH",
+	"DLB2_ST_INVALID_CQ_VIRT_ADDR",
+	"DLB2_ST_INVALID_PORT_ID",
+	"DLB2_ST_INVALID_QID",
+	"DLB2_ST_INVALID_PRIORITY",
+	"DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+	"DLB2_ST_INVALID_DIR_QUEUE_ID",
+	"DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+	"DLB2_ST_DOMAIN_NOT_CONFIGURED",
+	"DLB2_ST_INTERNAL_ERROR",
+	"DLB2_ST_DOMAIN_IN_USE",
+	"DLB2_ST_DOMAIN_NOT_FOUND",
+	"DLB2_ST_QUEUE_NOT_FOUND",
+	"DLB2_ST_DOMAIN_STARTED",
+	"DLB2_ST_DOMAIN_NOT_STARTED",
+	"DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+	"DLB2_ST_DOMAIN_RESET_FAILED",
+	"DLB2_ST_MBOX_ERROR",
+	"DLB2_ST_INVALID_HIST_LIST_DEPTH",
+	"DLB2_ST_NO_MEMORY",
+	"DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+	"DLB2_ST_INVALID_COS_ID",
+};
+
+struct dlb2_cmd_response {
+	__u32 status; /* Interpret using enum dlb2_error */
+	__u32 id;
+};
+
+/*******************/
+/* 'dlb2' commands */
+/*******************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+	DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	This ioctl interface is the same in all driver versions and is always
+ *	the first ioctl.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+	/* Output parameters */
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ *	load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ *		percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ *	load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ *   ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+enum dlb2_cq_poll_modes {
+	DLB2_CQ_POLL_MODE_STD,
+	DLB2_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB2_CQ_POLL_MODE must be last */
+	NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/********************************/
+/* 'scheduling domain' commands */
+/********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_start_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
+#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
+
+
+#endif /* __DLB2_USER_H */
-- 
2.6.4

[dpdk-dev] [PATCH v4 05/23] event/dlb2: add inline functions

[Timothy McDaniel]

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2_inline_fns.h | 61 ++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h

diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
new file mode 100644
index 0000000..ccbc4f4
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_inline_fns.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_INLINE_FNS_H_
+#define _DLB2_INLINE_FNS_H_
+
+/* Inline functions required in more than one source file. */
+
+static inline struct dlb2_eventdev *
+dlb2_pmd_priv(const struct rte_eventdev *eventdev)
+{
+	return eventdev->data->dev_private;
+}
+
+static inline void
+dlb2_umonitor(volatile void *addr)
+{
+	asm volatile(".byte 0xf3, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (addr));
+}
+
+static inline void
+dlb2_umwait(int state, uint64_t timeout)
+{
+	uint32_t eax = timeout & UINT32_MAX;
+	uint32_t edx = timeout >> 32;
+
+	asm volatile(".byte 0xf2, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (state),  "a" (eax), "d" (edx));
+}
+
+static inline void
+dlb2_movntdq_single(void *pp_addr, void *qe4)
+{
+	long long *_qe  = (long long *)qe4;
+	__v2di src_data0 = (__v2di){_qe[0], _qe[1]};
+
+	__builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
+}
+
+static inline void
+dlb2_cldemote(void *addr)
+{
+	/* Load addr into RSI, then demote the cache line of the address
+	 * contained in that register.
+	 */
+	asm volatile(".byte 0x0f, 0x1c, 0x06" :: "S" (addr));
+}
+
+static inline void
+dlb2_movdir64b(void *pp_addr, void *qe4)
+{
+	asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
+		     :
+		     : "a" (pp_addr), "d" (qe4));
+}
+
+#endif /* _DLB2_INLINE_FNS_H_ */
-- 
2.6.4

[dpdk-dev] [PATCH v4 06/23] event/dlb2: add eventdev probe

[Timothy McDaniel]

Add the eventdev portion of probe, and parse command line
options, but do not initialize hardware.

Changes since v2 patch-set probe:

Primary and secondary probe-time init has been removed, and
will be introduced in subsequent patches contained in
the v3 patch-set.

Hardware init has been moved to a subsequent patch in order to
minimize the patch size. The files dlb2/pf/dlb2_pf.c and
dlb2/pf/dlb_main.c include stubs of hardware init functions,
and these will be replaced with the real versions when the
hardware layer is committed.

Initialization of the flexible interface layer has been moved to
a subsequent patch in order to minimize patch size.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                      |  343 ++++
 drivers/event/dlb2/meson.build                 |    5 +-
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  230 +++
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  440 +++++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  621 ++++++
 drivers/event/dlb2/pf/dlb2_main.h              |   98 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  196 ++
 13 files changed, 7497 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2082dd6..1b11979 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2,6 +2,349 @@
  * Copyright(c) 2016-2020 Intel Corporation
  */
 
+#include <assert.h>
+#include <errno.h>
+#include <nmmintrin.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_cycles.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_io.h>
+#include <rte_kvargs.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_ring.h>
+#include <rte_string_fns.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+/*
+ * Resources exposed to eventdev. Some values overridden at runtime using
+ * values returned by the DLB kernel driver.
+ */
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
+#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
+#endif
+
+struct process_local_port_data
+dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
+
+#define DLB2_BASE_10 10
+
+static int
+dlb2_string_to_int(int *result, const char *str)
+{
+	long ret;
+	char *endptr;
+
+	if (str == NULL || result == NULL)
+		return -EINVAL;
+
+	errno = 0;
+	ret = strtol(str, &endptr, DLB2_BASE_10);
+	if (errno)
+		return -errno;
+
+	/* long int and int may be different width for some architectures */
+	if (ret < INT_MIN || ret > INT_MAX || endptr == str)
+		return -EINVAL;
+
+	*result = ret;
+	return 0;
+}
+
+static int
+set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
+{
+	int *socket_id = opaque;
+	int ret;
+
+	ret = dlb2_string_to_int(socket_id, value);
+	if (ret < 0)
+		return ret;
+
+	if (*socket_id > RTE_MAX_NUMA_NODES)
+		return -EINVAL;
+	return 0;
+}
+
+static int
+set_max_num_events(const char *key __rte_unused,
+		   const char *value,
+		   void *opaque)
+{
+	int *max_num_events = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(max_num_events, value);
+	if (ret < 0)
+		return ret;
+
+	if (*max_num_events < 0 || *max_num_events >
+			DLB2_MAX_NUM_LDB_CREDITS) {
+		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
+			     DLB2_MAX_NUM_LDB_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_num_dir_credits(const char *key __rte_unused,
+		    const char *value,
+		    void *opaque)
+{
+	int *num_dir_credits = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(num_dir_credits, value);
+	if (ret < 0)
+		return ret;
+
+	if (*num_dir_credits < 0 ||
+	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
+		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
+			     DLB2_MAX_NUM_DIR_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_dev_id(const char *key __rte_unused,
+	   const char *value,
+	   void *opaque)
+{
+	int *dev_id = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(dev_id, value);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+set_cos(const char *key __rte_unused,
+	const char *value,
+	void *opaque)
+{
+	enum dlb2_cos *cos_id = opaque;
+	int x = 0;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(&x, value);
+	if (ret < 0)
+		return ret;
+
+	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
+		DLB2_LOG_ERR(
+			"COS %d out of range, must be DLB2_COS_DEFAULT or 0-3\n",
+			x);
+		return -EINVAL;
+	}
+
+	*cos_id = x;
+
+	return 0;
+}
+
+
+static int
+set_qid_depth_thresh(const char *key __rte_unused,
+		     const char *value,
+		     void *opaque)
+{
+	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
+	int first, last, thresh, i;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	/* command line override may take one of the following 3 forms:
+	 * qid_depth_thresh=all:<threshold_value> ... all queues
+	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
+	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
+	 */
+	if (sscanf(value, "all:%d", &thresh) == 1) {
+		first = 0;
+		last = DLB2_MAX_NUM_QUEUES - 1;
+	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
+		/* we have everything we need */
+	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
+		last = first;
+	} else {
+		DLB2_LOG_ERR("Error parsing qid depth devarg. Should be all:val, qid-qid:val, or qid:val\n");
+		return -EINVAL;
+	}
+
+	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, invalid qid value\n");
+		return -EINVAL;
+	}
+
+	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
+		DLB2_LOG_ERR("Error parsing qid depth devarg, threshold > %d\n",
+			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
+		return -EINVAL;
+	}
+
+	for (i = first; i <= last; i++)
+		qid_thresh->val[i] = thresh; /* indexed by qid */
+
+	return 0;
+}
+
+int
+dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+			    const char *name,
+			    struct dlb2_devargs *dlb2_args)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+	RTE_SET_USED(dlb2_args);
+
+	return 0;
+}
+
+int
+dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+			      const char *name)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+int
+dlb2_parse_params(const char *params,
+		  const char *name,
+		  struct dlb2_devargs *dlb2_args)
+{
+	int ret = 0;
+	static const char * const args[] = { NUMA_NODE_ARG,
+					     DLB2_MAX_NUM_EVENTS,
+					     DLB2_NUM_DIR_CREDITS,
+					     DEV_ID_ARG,
+					     DLB2_QID_DEPTH_THRESH_ARG,
+					     DLB2_COS_ARG,
+					     NULL };
+
+	if (params != NULL && params[0] != '\0') {
+		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
+
+		if (kvlist == NULL) {
+			RTE_LOG(INFO, PMD,
+				"Ignoring unsupported parameters when creating device '%s'\n",
+				name);
+		} else {
+			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
+						     set_numa_node,
+						     &dlb2_args->socket_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing numa node parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
+						 set_max_num_events,
+						 &dlb2_args->max_num_events);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist,
+					DLB2_NUM_DIR_CREDITS,
+					set_num_dir_credits,
+					&dlb2_args->num_dir_credits_override);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
+						 set_dev_id,
+						 &dlb2_args->dev_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(
+					kvlist,
+					DLB2_QID_DEPTH_THRESH_ARG,
+					set_qid_depth_thresh,
+					&dlb2_args->qid_depth_thresholds);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
+						 set_cos,
+						 &dlb2_args->cos_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing cos parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
 
+			rte_kvargs_free(kvlist);
+		}
+	}
+	return ret;
+}
 RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 435c1ee..1eab9b9 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -7,6 +7,9 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
         subdir_done()
 endif
 
-sources = files('dlb2.c')
+sources = files('dlb2.c',
+		'pf/dlb2_main.c',
+		'pf/dlb2_pf.c'
+)
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
new file mode 100644
index 0000000..1d99f1e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_osdep_list.h"
+#include "dlb2_osdep_types.h"
+
+#define DLB2_MAX_NUM_VDEVS			16
+#define DLB2_MAX_NUM_DOMAINS			32
+#define DLB2_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS			64
+#define DLB2_MAX_NUM_DIR_PORTS			64
+#define DLB2_MAX_NUM_LDB_CREDITS		(8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS		(2 * 1024)
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB2_MAX_NUM_AQED_ENTRIES		2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB2_QID_PRIORITIES			8
+#define DLB2_NUM_ARB_WEIGHTS			8
+#define DLB2_MAX_WEIGHT				255
+#define DLB2_NUM_COS_DOMAINS			4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#ifdef FPGA
+#define DLB2_HZ					2000000
+#else
+#define DLB2_HZ					800000000
+#endif
+
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
+
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+	DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+	DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID		3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID		4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW		5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ	1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
+#define DLB2_VF_BASE_CQ_VECTOR_ID	     0
+#define DLB2_VF_LAST_CQ_VECTOR_ID	     30
+#define DLB2_VF_MBOX_VECTOR_ID		     31
+#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
+
+#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
+					     DLB2_MAX_NUM_DIR_PORTS + 1)
+
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE   0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
+				DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE   0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
+				DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE       0x2100000
+#define DLB2_LDB_PP_STRIDE     0x1000
+#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
+				DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE       0x2000000
+#define DLB2_DIR_PP_STRIDE     0x1000
+#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
+				DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
+struct dlb2_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb2_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb2_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
+struct dlb2_ldb_queue {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB2_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB2_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB2_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB2_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+	enum dlb2_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb2_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+	u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+	u32 bound[6] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & (1 << i))) {
+			group->slot_use_bitmap |= 1 << i;
+			return i;
+		}
+	}
+
+	return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+	int i, cnt = 0;
+
+	for (i = 0; i < 32; i++)
+		cnt += !!(group->slot_use_bitmap & (1 << i));
+
+	return cnt;
+}
+
+struct dlb2_hw_domain {
+	struct dlb2_function_resources *parent_func;
+	struct dlb2_list_entry func_list;
+	struct dlb2_list_head used_ldb_queues;
+	struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head used_dir_pq_pairs;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb2_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb2_bitmap;
+
+struct dlb2_function_resources {
+	struct dlb2_list_head avail_domains;
+	struct dlb2_list_head used_domains;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	struct dlb2_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+	struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+	struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+	struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
+struct dlb2_mbox {
+	u32 *mbox;
+	u32 *isr_in_progress;
+};
+
+struct dlb2_sw_mbox {
+	struct dlb2_mbox vdev_to_pf;
+	struct dlb2_mbox pf_to_vdev;
+	void (*pf_to_vdev_inject)(void *arg);
+	void *pf_to_vdev_inject_arg;
+};
+
+struct dlb2_hw {
+	/* BAR 0 address */
+	void  *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void  *func_kva;
+	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb2_hw_resources rsrcs;
+	struct dlb2_function_resources pf;
+	struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+	struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+	/* Virtualization */
+	int virt_mode;
+	struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
+	unsigned int pasid[DLB2_MAX_NUM_VDEVS];
+};
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
new file mode 100644
index 0000000..ce462c0
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
@@ -0,0 +1,596 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_BASE_DLB2_MBOX_H
+#define __DLB2_BASE_DLB2_MBOX_H
+
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+
+#define DLB2_MBOX_INTERFACE_VERSION 1
+
+/*
+ * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
+ * to send requests of its own (e.g. notifying a VF of an impending FLR).
+ * To avoid communication race conditions, e.g. the PF sends a response and then
+ * sends a request before the VF reads the response, the PF->VF mailbox is
+ * divided into two sections:
+ * - Bytes 0-47: PF responses
+ * - Bytes 48-63: PF requests
+ *
+ * Partitioning the PF->VF mailbox allows responses and requests to occupy the
+ * mailbox simultaneously.
+ */
+#define DLB2_PF2VF_RESP_BYTES	  48
+#define DLB2_PF2VF_RESP_BASE	  0
+#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
+
+#define DLB2_PF2VF_REQ_BYTES	  16
+#define DLB2_PF2VF_REQ_BASE	  (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
+#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
+
+/*
+ * Similarly, the VF->PF mailbox is divided into two sections:
+ * - Bytes 0-239: VF requests
+ * -- (Bytes 0-3 are unused due to a hardware errata)
+ * - Bytes 240-255: VF responses
+ */
+#define DLB2_VF2PF_REQ_BYTES	 236
+#define DLB2_VF2PF_REQ_BASE	 4
+#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
+
+#define DLB2_VF2PF_RESP_BYTES	  16
+#define DLB2_VF2PF_RESP_BASE	  (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
+#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
+
+/* VF-initiated commands */
+enum dlb2_mbox_cmd_type {
+	DLB2_MBOX_CMD_REGISTER,
+	DLB2_MBOX_CMD_UNREGISTER,
+	DLB2_MBOX_CMD_GET_NUM_RESOURCES,
+	DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
+	DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
+	DLB2_MBOX_CMD_CREATE_LDB_PORT,
+	DLB2_MBOX_CMD_CREATE_DIR_PORT,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT,
+	DLB2_MBOX_CMD_DISABLE_LDB_PORT,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT,
+	DLB2_MBOX_CMD_DISABLE_DIR_PORT,
+	DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_MAP_QID,
+	DLB2_MBOX_CMD_UNMAP_QID,
+	DLB2_MBOX_CMD_START_DOMAIN,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
+	DLB2_MBOX_CMD_ARM_CQ_INTR,
+	DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
+	DLB2_MBOX_CMD_GET_SN_ALLOCATION,
+	DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
+	DLB2_MBOX_CMD_GET_COS_BW,
+	DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
+	DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_QE_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_CMD_TYPES,
+};
+
+static const char dlb2_mbox_cmd_type_strings[][128] = {
+	"DLB2_MBOX_CMD_REGISTER",
+	"DLB2_MBOX_CMD_UNREGISTER",
+	"DLB2_MBOX_CMD_GET_NUM_RESOURCES",
+	"DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_LDB_PORT",
+	"DLB2_MBOX_CMD_CREATE_DIR_PORT",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_DISABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_DISABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_MAP_QID",
+	"DLB2_MBOX_CMD_UNMAP_QID",
+	"DLB2_MBOX_CMD_START_DOMAIN",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
+	"DLB2_MBOX_CMD_ARM_CQ_INTR",
+	"DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
+	"DLB2_MBOX_CMD_GET_SN_ALLOCATION",
+	"DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
+	"DLB2_MBOX_CMD_GET_COS_BW",
+	"DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
+	"DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
+};
+
+/* PF-initiated commands */
+enum dlb2_mbox_vf_cmd_type {
+	DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
+	DLB2_MBOX_VF_CMD_NOTIFICATION,
+	DLB2_MBOX_VF_CMD_IN_USE,
+
+	/* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_VF_CMD_TYPES,
+};
+
+static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
+	"DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
+	"DLB2_MBOX_VF_CMD_NOTIFICATION",
+	"DLB2_MBOX_VF_CMD_IN_USE",
+};
+
+#define DLB2_MBOX_CMD_TYPE(hdr) \
+	(((struct dlb2_mbox_req_hdr *)hdr)->type)
+#define DLB2_MBOX_CMD_STRING(hdr) \
+	dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
+
+enum dlb2_mbox_status_type {
+	DLB2_MBOX_ST_SUCCESS,
+	DLB2_MBOX_ST_INVALID_CMD_TYPE,
+	DLB2_MBOX_ST_VERSION_MISMATCH,
+	DLB2_MBOX_ST_INVALID_OWNER_VF,
+};
+
+static const char dlb2_mbox_status_type_strings[][128] = {
+	"DLB2_MBOX_ST_SUCCESS",
+	"DLB2_MBOX_ST_INVALID_CMD_TYPE",
+	"DLB2_MBOX_ST_VERSION_MISMATCH",
+	"DLB2_MBOX_ST_INVALID_OWNER_VF",
+};
+
+#define DLB2_MBOX_ST_TYPE(hdr) \
+	(((struct dlb2_mbox_resp_hdr *)hdr)->status)
+#define DLB2_MBOX_ST_STRING(hdr) \
+	dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
+
+/* This structure is always the first field in a request structure */
+struct dlb2_mbox_req_hdr {
+	u32 type;
+};
+
+/* This structure is always the first field in a response structure */
+struct dlb2_mbox_resp_hdr {
+	u32 status;
+};
+
+struct dlb2_mbox_register_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 min_interface_version;
+	u16 max_interface_version;
+};
+
+struct dlb2_mbox_register_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 interface_version;
+	u8 pf_id;
+	u8 vf_id;
+	u8 is_auxiliary_vf;
+	u8 primary_vf_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u16 num_sched_domains;
+	u16 num_ldb_queues;
+	u16 num_ldb_ports;
+	u16 num_cos_ldb_ports[4];
+	u16 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 max_contiguous_hist_list_entries;
+	u16 num_ldb_credits;
+	u16 num_dir_credits;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 num_ldb_queues;
+	u32 num_ldb_ports;
+	u32 num_cos_ldb_ports[4];
+	u32 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u8 cos_strict;
+	u8 padding0[3];
+	u32 padding1;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 num_sequence_numbers;
+	u32 num_qid_inflights;
+	u32 num_atomic_inflights;
+	u32 lock_id_comp_level;
+	u32 depth_threshold;
+	u32 padding;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 depth_threshold;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u16 cq_depth;
+	u16 cq_history_list_size;
+	u8 cos_id;
+	u8 cos_strict;
+	u16 padding1;
+	u64 cq_base_address;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u64 cq_base_address;
+	u16 cq_depth;
+	u16 padding0;
+	s32 queue_id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_map_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+	u32 priority;
+	u32 padding0;
+};
+
+struct dlb2_mbox_map_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_start_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+};
+
+struct dlb2_mbox_start_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 is_ldb;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding0;
+};
+
+/*
+ * The alert_id and aux_alert_data follows the format of the alerts defined in
+ * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
+ * the aux_alert_data value varies depending on the alert.
+ */
+struct dlb2_mbox_vf_alert_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 alert_id;
+	u32 aux_alert_data;
+};
+
+enum dlb2_mbox_vf_notification_type {
+	DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
+	DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
+
+	/* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
+	NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
+};
+
+struct dlb2_mbox_vf_notification_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 notification;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 in_use;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 num;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 cos_id;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 mode;
+};
+
+#endif /* __DLB2_BASE_DLB2_MBOX_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
new file mode 100644
index 0000000..c4c34eb
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
@@ -0,0 +1,230 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_H
+#define __DLB2_OSDEP_H
+
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <rte_string_fns.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include "../dlb2_main.h"
+#include "dlb2_resource.h"
+#include "../../dlb2_log.h"
+#include "../../dlb2_user.h"
+
+
+#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
+#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
+#define DLB2_CSR_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
+#define DLB2_FUNC_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
+/* Map to PMDs logging interface */
+#define DLB2_ERR(dev, fmt, args...) \
+	DLB2_LOG_ERR(fmt, ## args)
+
+#define DLB2_INFO(dev, fmt, args...) \
+	DLB2_LOG_INFO(fmt, ## args)
+
+#define DLB2_DEBUG(dev, fmt, args...) \
+	DLB2_LOG_DBG(fmt, ## args)
+
+/**
+ * os_udelay() - busy-wait for a number of microseconds
+ * @usecs: delay duration.
+ */
+static inline void os_udelay(int usecs)
+{
+	rte_delay_us(usecs);
+}
+
+/**
+ * os_msleep() - sleep for a number of milliseconds
+ * @usecs: delay duration.
+ */
+static inline void os_msleep(int msecs)
+{
+	rte_delay_ms(msecs);
+}
+
+#define DLB2_PP_BASE(__is_ldb) \
+	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
+
+/**
+ * os_map_producer_port() - map a producer port into the caller's address space
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ *
+ * This function maps the requested producer port memory into the caller's
+ * address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void *os_map_producer_port(struct dlb2_hw *hw,
+					 u8 port_id,
+					 bool is_ldb)
+{
+	uint64_t addr;
+	uint64_t pp_dma_base;
+
+	pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
+	addr = (pp_dma_base + (PAGE_SIZE * port_id));
+
+	return (void *)(uintptr_t)addr;
+}
+
+/**
+ * os_unmap_producer_port() - unmap a producer port
+ * @addr: mapped producer port address
+ *
+ * This function undoes os_map_producer_port() by unmapping the producer port
+ * memory from the caller's address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(addr);
+}
+
+/**
+ * os_fence_hcw() - fence an HCW to ensure it arrives at the device
+ * @hw: dlb2_hw handle for a particular device.
+ * @pp_addr: producer port address
+ */
+static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
+{
+	RTE_SET_USED(hw);
+
+	/* To ensure outstanding HCWs reach the device, read the PP address. IA
+	 * memory ordering prevents reads from passing older writes, and the
+	 * mfence also ensures this.
+	 */
+	rte_mb();
+
+	*(volatile u64 *)pp_addr;
+}
+
+/**
+ * DLB2_HW_ERR() - log an error message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_ERR(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_ERR(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/**
+ * DLB2_HW_DBG() - log an info message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_DBG(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_DEBUG(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/* The callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void *dlb2_complete_queue_map_unmap(void *__args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
+	int ret;
+
+	while (1) {
+		rte_spinlock_lock(&dlb2_dev->resource_mutex);
+
+		ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+		ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+		if (ret != 0) {
+			rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+			/* Relinquish the CPU so the application can process
+			 * its CQs, so this function doesn't deadlock.
+			 */
+			sched_yield();
+		} else {
+			break;
+		}
+	}
+
+	dlb2_dev->worker_launched = false;
+
+	rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+
+	return NULL;
+}
+
+
+/**
+ * os_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static inline void os_schedule_work(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+	pthread_t complete_queue_map_unmap_thread;
+	int ret;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
+				     "dlb_queue_unmap_waiter",
+				     NULL,
+				     dlb2_complete_queue_map_unmap,
+				     dlb2_dev);
+	if (ret)
+		DLB2_ERR(dlb2_dev,
+			 "Could not create queue complete map/unmap thread, err=%d\n",
+			 ret);
+	else
+		dlb2_dev->worker_launched = true;
+}
+
+/**
+ * os_worker_active() - query whether the map/unmap worker thread is active
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether a thread (launched by
+ * os_schedule_work()) is active. This function is used to determine
+ * whether or not to launch a worker thread.
+ */
+static inline bool os_worker_active(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	return dlb2_dev->worker_launched;
+}
+
+#endif /*  __DLB2_OSDEP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
new file mode 100644
index 0000000..423233b
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <rte_bitmap.h>
+#include <rte_string_fns.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include "../dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+	struct rte_bitmap *map;
+	unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+				    unsigned int len)
+{
+	struct dlb2_bitmap *bm;
+	void *mem;
+	uint32_t alloc_size;
+	uint32_t nbits = (uint32_t)len;
+
+	if (bitmap == NULL || nbits == 0)
+		return -EINVAL;
+
+	/* Allocate DLB2 bitmap control struct */
+	bm = rte_malloc("DLB2_PF",
+			sizeof(struct dlb2_bitmap),
+			RTE_CACHE_LINE_SIZE);
+
+	if (bm == NULL)
+		return -ENOMEM;
+
+	/* Allocate bitmap memory */
+	alloc_size = rte_bitmap_get_memory_footprint(nbits);
+	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
+	if (mem == NULL) {
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->map = rte_bitmap_init(len, mem, alloc_size);
+	if (bm->map == NULL) {
+		rte_free(mem);
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap == NULL)
+		return;
+
+	rte_free(bitmap->map);
+	rte_free(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++)
+		rte_bitmap_set(bitmap->map, i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	rte_bitmap_reset(bitmap->map);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set() - set a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
+				  unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_set(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+					unsigned int bit,
+					unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_set(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear() - clear a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
+				    unsigned int bit)
+{
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_clear(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+					  unsigned int bit,
+					  unsigned int len)
+{
+	unsigned int i;
+
+	if (bitmap  == NULL || bitmap->map == NULL)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	for (i = 0; i != len; i++)
+		rte_bitmap_clear(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+						 unsigned int len)
+{
+	unsigned int i, j = 0;
+
+	if (bitmap  == NULL || bitmap->map == NULL || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			if (++j == len)
+				return i - j + 1;
+		} else {
+			j = 0;
+		}
+	}
+
+	/* No set bit range of length len? */
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			return i;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+	int weight = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			weight++;
+	}
+	return weight;
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				      bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+	int max_len = 0, len = 0;
+	unsigned int i;
+
+	if (bitmap == NULL)
+		return -EINVAL;
+
+	if (bitmap->map == NULL)
+		return -EINVAL;
+
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			len++;
+		} else {
+			if (len > max_len)
+				max_len = len;
+			len = 0;
+		}
+	}
+
+	if (len > max_len)
+		max_len = len;
+
+	return max_len;
+}
+
+/**
+ * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
+ * @dest: pointer to dlb2_bitmap structure, which will contain the results of
+ *	  the 'or' of src1 and src2.
+ * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
+ * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
+ *
+ * This function 'or's two bitmaps together and stores the result in a third
+ * bitmap. The source and destination bitmaps can be the same.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - One of the bitmaps is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
+				 struct dlb2_bitmap *src1,
+				 struct dlb2_bitmap *src2)
+{
+	unsigned int i, min;
+	int numset = 0;
+
+	if (dest  == NULL || dest->map == NULL ||
+	    src1  == NULL || src1->map == NULL ||
+	    src2  == NULL || src2->map == NULL)
+		return -EINVAL;
+
+	min = dest->len;
+	min = (min > src1->len) ? src1->len : min;
+	min = (min > src2->len) ? src2->len : min;
+
+	for (i = 0; i != min; i++) {
+		if  (rte_bitmap_get(src1->map, i) ||
+		     rte_bitmap_get(src2->map, i)) {
+			rte_bitmap_set(dest->map, i);
+			numset++;
+		} else {
+			rte_bitmap_clear(dest->map, i);
+		}
+	}
+
+	return numset;
+}
+
+#endif /*  __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
new file mode 100644
index 0000000..5531739
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_LIST_H
+#define __DLB2_OSDEP_LIST_H
+
+#include <rte_tailq.h>
+
+struct dlb2_list_entry {
+	TAILQ_ENTRY(dlb2_list_entry) node;
+};
+
+/* Dummy - just a struct definition */
+TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
+
+/* =================
+ * TAILQ Supplements
+ * =================
+ */
+
+#ifndef TAILQ_FOREACH_ENTRY
+#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)		\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter)						\
+		&& (ptr = container_of(iter, typeof(*(ptr)), name)); \
+	    (iter) = TAILQ_NEXT((iter), node))
+#endif
+
+#ifndef TAILQ_FOREACH_ENTRY_SAFE
+#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)	\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter) &&						\
+		(ptr = container_of(iter, typeof(*(ptr)), name)) &&\
+		((tvar) = TAILQ_NEXT((iter), node), 1);	\
+	    (iter) = (tvar))
+#endif
+
+/***********************/
+/*** List operations ***/
+/***********************/
+
+/**
+ * dlb2_list_init_head() - initialize the head of a list
+ * @head: list head
+ */
+static inline void dlb2_list_init_head(struct dlb2_list_head *head)
+{
+	TAILQ_INIT(head);
+}
+
+/**
+ * dlb2_list_add() - add an entry to a list
+ * @head: list head
+ * @entry: new list entry
+ */
+static inline void
+dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
+{
+	TAILQ_INSERT_TAIL(head, entry, node);
+}
+
+/**
+ * dlb2_list_del() - delete an entry from a list
+ * @entry: list entry
+ * @head: list head
+ */
+static inline void dlb2_list_del(struct dlb2_list_head *head,
+				 struct dlb2_list_entry *entry)
+{
+	TAILQ_REMOVE(head, entry, node);
+}
+
+/**
+ * dlb2_list_empty() - check if a list is empty
+ * @head: list head
+ *
+ * Return:
+ * Returns 1 if empty, 0 if not.
+ */
+static inline int dlb2_list_empty(struct dlb2_list_head *head)
+{
+	return TAILQ_EMPTY(head);
+}
+
+/**
+ * dlb2_list_splice() - splice a list
+ * @src_head: list to be added
+ * @ head: where src_head will be inserted
+ */
+static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
+				    struct dlb2_list_head *head)
+{
+	TAILQ_CONCAT(head, src_head, node);
+}
+
+/**
+ * DLB2_LIST_HEAD() - retrieve the head of the list
+ * @head: list head
+ * @type: type of the list variable
+ * @name: name of the list field within the containing struct
+ */
+#define DLB2_LIST_HEAD(head, type, name)                       \
+	(TAILQ_FIRST(&head) ?					\
+		container_of(TAILQ_FIRST(&head), type, name) :	\
+		NULL)
+
+/**
+ * DLB2_LIST_FOR_EACH() - iterate over a list
+ * @head: list head
+ * @ptr: pointer to struct containing a struct list
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ */
+#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
+	TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
+
+/**
+ * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
+ * an element is removed from the list while processing it.
+ * @ptr: pointer to struct containing a struct list
+ * @ptr_tmp: pointer to struct containing a struct list (temporary)
+ * @head: list head
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ * @iter_tmp: iterator variable (temporary)
+ */
+#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
+	TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
+
+#endif /*  __DLB2_OSDEP_LIST_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
new file mode 100644
index 0000000..0a48f7e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_TYPES_H
+#define __DLB2_OSDEP_TYPES_H
+
+#include <linux/types.h>
+
+#include <inttypes.h>
+#include <ctype.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Types for user mode PF PMD */
+typedef uint8_t         u8;
+typedef int8_t          s8;
+typedef uint16_t        u16;
+typedef int16_t         s16;
+typedef uint32_t        u32;
+typedef int32_t         s32;
+typedef uint64_t        u64;
+
+#define __iomem
+
+/* END types for user mode PF PMD */
+
+#endif /* __DLB2_OSDEP_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
new file mode 100644
index 0000000..43ecad4
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
@@ -0,0 +1,2527 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "dlb2_osdep_types.h"
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+	struct {
+		u32 isr_pend : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+	struct {
+		u32 vf0_reset_in_progress : 1;
+		u32 vf1_reset_in_progress : 1;
+		u32 vf2_reset_in_progress : 1;
+		u32 vf3_reset_in_progress : 1;
+		u32 vf4_reset_in_progress : 1;
+		u32 vf5_reset_in_progress : 1;
+		u32 vf6_reset_in_progress : 1;
+		u32 vf7_reset_in_progress : 1;
+		u32 vf8_reset_in_progress : 1;
+		u32 vf9_reset_in_progress : 1;
+		u32 vf10_reset_in_progress : 1;
+		u32 vf11_reset_in_progress : 1;
+		u32 vf12_reset_in_progress : 1;
+		u32 vf13_reset_in_progress : 1;
+		u32 vf14_reset_in_progress : 1;
+		u32 vf15_reset_in_progress : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+	struct {
+		u32 vec_mask : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+	struct {
+		u32 func_vf_bar_dis : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+	struct {
+		u32 total_vas : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+	struct {
+		u32 total_dir_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+	struct {
+		u32 total_ldb_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+	struct {
+		u32 total_dir_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+	struct {
+		u32 total_ldb_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+	struct {
+		u32 total_dir_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+	struct {
+		u32 total_ldb_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 meas : 1;
+		u32 debug : 7;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 cq_int_rearm : 1;
+		u32 dsi_error : 1;
+		u32 rsvd0 : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 rsvd0 : 3;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+	struct {
+		u32 qid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+	struct {
+		u32 vqid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+	struct {
+		u32 qid : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 debug : 8;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 isz : 1;
+		u32 dsi_error : 1;
+		u32 dlbrsvd : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 vf_synd0_parity : 1;
+		u32 vf_synd1_parity : 1;
+		u32 vf_synd2_parity : 1;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+	struct {
+		u32 sn_cfg_v : 1;
+		u32 fid_cfg_v : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS  0
+#define DLB2_CQ_ISR_MODE_MSI  1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI  3
+union dlb2_sys_ldb_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+	struct {
+		u32 keep_pf_ppid : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+	struct {
+		u32 illegal_hcw : 1;
+		u32 illegal_pp : 1;
+		u32 illegal_pasid : 1;
+		u32 illegal_qid : 1;
+		u32 disabled_qid : 1;
+		u32 illegal_ldb_qid_cfg : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+	struct {
+		u32 msix_0_ack : 1;
+		u32 msix_1_ack : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+	struct {
+		u32 msix_0_passthru : 1;
+		u32 msix_1_passthru : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED     0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+	struct {
+		u32 mode : 1;
+		u32 poll_mode : 1;
+		u32 poll_mask : 1;
+		u32 poll_lock : 1;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+	struct {
+		u32 cq : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 alarm : 1;
+		u32 cwd : 1;
+		u32 vf_pf_mb : 1;
+		u32 rsvd0 : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+	struct {
+		u32 qid_fid_limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+	struct {
+		u32 compress_code : 3;
+		u32 rsvd0 : 29;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+	(DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+	struct {
+		u32 count : 15;
+		u32 rsvd0 : 17;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+	struct {
+		u32 sn : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+	struct {
+		u32 mode : 3;
+		u32 slot : 4;
+		u32 rsvz0 : 1;
+		u32 grp : 1;
+		u32 rsvz1 : 1;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+	struct {
+		u32 en : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+	struct {
+		u32 write_pointer : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+	struct {
+		u32 base : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+	struct {
+		u32 limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+	struct {
+		u32 pop_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+	struct {
+		u32 push_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+	struct {
+		u32 depth : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+	struct {
+		u32 write_pointer : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 dlb_cor_alarm_enable : 1;
+		u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+		u32 cfg_64bytes_qe_dir_cq_mode : 1;
+		u32 pad_write_ldb : 1;
+		u32 pad_write_dir : 1;
+		u32 pad_first_write_ldb : 1;
+		u32 pad_first_write_dir : 1;
+		u32 rsvz0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+	struct {
+		u32 egress_credit_status_empty : 1;
+		u32 egress_credit_status_afull : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+		u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+		u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+		u32 chp_rop_pipe_credit_status_empty : 1;
+		u32 chp_rop_pipe_credit_status_afull : 1;
+		u32 qed_to_cq_pipe_credit_status_empty : 1;
+		u32 qed_to_cq_pipe_credit_status_afull : 1;
+		u32 egress_lsp_token_credit_status_empty : 1;
+		u32 egress_lsp_token_credit_status_afull : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 rsvz0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+	struct {
+		u32 sn_mode_0 : 3;
+		u32 rszv0 : 5;
+		u32 sn_mode_1 : 3;
+		u32 rszv1 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+	struct {
+		u32 unit_single_step_mode : 1;
+		u32 rr_en : 1;
+		u32 rszv0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+	struct {
+		u32 prio : 24;
+		u32 v : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+	struct {
+		u32 qid_p0 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p1 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p2 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p3 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+	struct {
+		u32 qid_p4 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p5 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p6 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p7 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+	struct {
+		u32 token_depth_select : 4;
+		u32 disable_wb_opt : 1;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+	struct {
+		u32 token_count : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+	struct {
+		u32 thresh : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+	(DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+	(DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+	struct {
+		u32 depth : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+	struct {
+		u32 cq : 8;
+		u32 qidix : 3;
+		u32 value : 1;
+		u32 nalb_haswork_v : 1;
+		u32 rlist_haswork_v : 1;
+		u32 slist_haswork_v : 1;
+		u32 inflight_ok_v : 1;
+		u32 aqed_nfull_v : 1;
+		u32 rsvz0 : 15;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+	struct {
+		u32 transfer : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+	struct {
+		u32 bw_range : 9;
+		u32 rsvz0 : 22;
+		u32 no_extra_credit : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+	struct {
+		u32 disab_atq_empty_arb : 1;
+		u32 inc_tok_unit_idle : 1;
+		u32 disab_rlist_pri : 1;
+		u32 inc_cmp_unit_idle : 1;
+		u32 rsvz0 : 2;
+		u32 dir_single_op : 1;
+		u32 dir_half_bw : 1;
+		u32 dir_single_out : 1;
+		u32 dir_disab_multi : 1;
+		u32 atq_single_op : 1;
+		u32 atq_half_bw : 1;
+		u32 atq_single_out : 1;
+		u32 atq_disab_multi : 1;
+		u32 dirrpl_single_op : 1;
+		u32 dirrpl_half_bw : 1;
+		u32 dirrpl_single_out : 1;
+		u32 lbrpl_single_op : 1;
+		u32 lbrpl_half_bw : 1;
+		u32 lbrpl_single_out : 1;
+		u32 ldb_single_op : 1;
+		u32 ldb_half_bw : 1;
+		u32 ldb_disab_multi : 1;
+		u32 atm_single_sch : 1;
+		u32 atm_single_cmp : 1;
+		u32 ldb_ce_tog_arb : 1;
+		u32 rsvz1 : 1;
+		u32 smon0_valid_sel : 2;
+		u32 smon0_value_sel : 1;
+		u32 smon0_compare_sel : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+	struct {
+		u32 chp_pf_reset_done : 1;
+		u32 rop_pf_reset_done : 1;
+		u32 lsp_pf_reset_done : 1;
+		u32 nalb_pf_reset_done : 1;
+		u32 ap_pf_reset_done : 1;
+		u32 dp_pf_reset_done : 1;
+		u32 qed_pf_reset_done : 1;
+		u32 dqed_pf_reset_done : 1;
+		u32 aqed_pf_reset_done : 1;
+		u32 sys_pf_reset_done : 1;
+		u32 pf_reset_active : 1;
+		u32 flrsm_state : 7;
+		u32 rsvd0 : 13;
+		u32 dlb_proc_reset_done : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+	struct {
+		u32 chp_pipeidle : 1;
+		u32 rop_pipeidle : 1;
+		u32 lsp_pipeidle : 1;
+		u32 nalb_pipeidle : 1;
+		u32 ap_pipeidle : 1;
+		u32 dp_pipeidle : 1;
+		u32 qed_pipeidle : 1;
+		u32 dqed_pipeidle : 1;
+		u32 aqed_pipeidle : 1;
+		u32 sys_pipeidle : 1;
+		u32 chp_unit_idle : 1;
+		u32 rop_unit_idle : 1;
+		u32 lsp_unit_idle : 1;
+		u32 nalb_unit_idle : 1;
+		u32 ap_unit_idle : 1;
+		u32 dp_unit_idle : 1;
+		u32 qed_unit_idle : 1;
+		u32 dqed_unit_idle : 1;
+		u32 aqed_unit_idle : 1;
+		u32 sys_unit_idle : 1;
+		u32 rsvd1 : 4;
+		u32 mstr_cfg_ring_idle : 1;
+		u32 mstr_cfg_mstr_idle : 1;
+		u32 mstr_flr_clkreq_b : 1;
+		u32 mstr_proc_idle : 1;
+		u32 mstr_proc_idle_masked : 1;
+		u32 rsvd0 : 2;
+		u32 dlb_func_idle : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+	struct {
+		u32 prochot : 1;
+		u32 pgcb_dlb_idle : 1;
+		u32 pgcb_dlb_pg_rdy_ack_b : 1;
+		u32 pmsm_pgcb_req_b : 1;
+		u32 pgbc_pmc_pg_req_b : 1;
+		u32 pmc_pgcb_pg_ack_b : 1;
+		u32 pmc_pgcb_fet_en_b : 1;
+		u32 pgcb_fet_en_b : 1;
+		u32 rsvz0 : 1;
+		u32 rsvz1 : 1;
+		u32 fuse_force_on : 1;
+		u32 fuse_proc_disable : 1;
+		u32 rsvz2 : 1;
+		u32 rsvz3 : 1;
+		u32 pm_fsm_d0tod3_ok : 1;
+		u32 pm_fsm_d3tod0_ok : 1;
+		u32 dlb_in_d3 : 1;
+		u32 rsvz4 : 7;
+		u32 pmsm : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+	struct {
+		u32 disable : 1;
+		u32 rsvz0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+	struct {
+		u32 isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+	struct {
+		u32 pf_isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+	struct {
+		u32 isr_pend : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+	struct {
+		u32 reset_in_progress : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+	struct {
+		u32 vf_msi_isr : 32;
+	} field;
+	u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
new file mode 100644
index 0000000..503fdf3
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
@@ -0,0 +1,1913 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_osdep_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_start_domain() - start a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: start domain arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int dlb2_hw_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp,
+			 bool vdev_request,
+			 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_request,
+		    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding unmap procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_finish_map_qid_procedures() - finish any pending map procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding map procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ *					interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+			  int port_id,
+			  bool is_ldb,
+			  bool vdev_request,
+			  unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+					 u32 *ldb_interrupts,
+					 u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+				 u32 *ldb_interrupts,
+				 u32 *dir_interrupts);
+
+/**
+ * dlb2_read_vf_intr_status() - read the VF interrupt status register
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function can be called from a VF's interrupt handler to determine
+ * which interrupts have fired. The first 31 bits correspond to CQ interrupt
+ * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
+ *
+ * Return:
+ * Returns a bit vector indicating which interrupt vectors are active.
+ */
+u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_intr_status() - ack VF interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function ACKs a VF's interrupts. Its interrupts argument should be the
+ * value returned by dlb2_read_vf_intr_status().
+ */
+void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function clears the VF's MSI interrupt pending register. Its interrupts
+ * argument should be contain the MSI vectors to ACK. For example, if MSI MME
+ * is in mode 0, then one bit 0 should ever be set.
+ */
+void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When done processing the PF mailbox request, this function unsets
+ * the PF's mailbox ISR register.
+ */
+void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
+
+/**
+ * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
+ *					vdevs
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the vdev->PF ISR fires, this function can be called to determine which
+ * vdev(s) are requesting service. This bitvector must be passed to
+ * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
+ * vdevs.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested service.
+ */
+u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ *
+ * When done processing all VF mailbox requests, this function unsets the VF's
+ * mailbox ISR register.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
+ *				    FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the VF FLR ISR fires, this function can be called to determine which
+ * VF(s) are requesting FLRs. This bitvector must passed to
+ * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
+ */
+u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF FLR requests, this function unsets the VF's FLR
+ * ISR register.
+ */
+void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF requests, this function communicates to the
+ * hardware that processing is complete.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
+			     u32 mbox_bitvec,
+			     u32 flr_bitvec);
+
+/**
+ * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the watchdog timer interrupt cause registers to
+ * determine which port(s) had a watchdog timeout, and notifies the
+ * application(s) that own the port(s).
+ */
+void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads and logs the alarm syndrome, then acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_request,
+			      unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of resources in use by the PF or a VF. It
+ * fills in the fields that args points to, except the following:
+ * - max_contiguous_atomic_inflights
+ * - max_contiguous_hist_list_entries
+ * - max_contiguous_ldb_credits
+ * - max_contiguous_dir_credits
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
+				   struct dlb2_get_num_resources_args *arg,
+				   bool vdev_request,
+				   unsigned int vdev_id);
+
+/**
+ * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
+ *				       the PF
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function sends a VF->PF mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the PF has processed
+ * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
+ * when the PF has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
+ *				 request
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * processing a VF->PF mailbox request. It should only be called after sending
+ * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vf_flr_complete() - check the status of a VF FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * executing the VF FLR. It should only be called after setting the VF's FLR
+ * bit.
+ */
+bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
+ *					vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function sends a PF->vdev mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the vdev has
+ * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
+ * determine when the vdev has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
+ *			       request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function returns a boolean indicating whether the vdev has finished
+ * processing a PF->vdev mailbox request. It should only be called after
+ * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
+			     unsigned int vf_id,
+			     void *data,
+			     int len);
+
+/**
+ * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
+			       unsigned int vf_id,
+			       void *data,
+			       int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_reset_vdev() - reset the hardware owned by a virtual device
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function resets the hardware owned by a vdev, by resetting the vdev's
+ * domains one by one.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function returns whether or not the vdev's resource assignments are
+ * locked. If locked, no resources can be added to or subtracted from the
+ * group.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_lock_vdev() - lock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sets a flag indicating that the vdev is using its resources.
+ * When the vdev is locked, its resource assignment cannot be changed.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_unlock_vdev() - unlock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function unlocks the vdev's resource assignment, allowing it to be
+ * modified.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of scheduling domains to assign to this vdev
+ *
+ * This function assigns num scheduling domains to the specified vdev. If the
+ * vdev already has domains assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB queues to assign to this vdev
+ *
+ * This function assigns num LDB queues to the specified vdev. If the vdev
+ * already has LDB queues assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports to the specified vdev. If the vdev
+ * already has LDB ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @cos: class-of-service ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports from class-of-service cos to the
+ * specified vdev. If the vdev already has LDB ports from this class-of-service
+ * assigned, this existing assignment is adjusted accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
+				   u32 id,
+				   u32 cos,
+				   u32 num);
+
+/**
+ * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR ports to assign to this vdev
+ *
+ * This function assigns num DIR ports to the specified vdev. If the vdev
+ * already has DIR ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB credit credits to assign to this vdev
+ *
+ * This function assigns num LDB credit to the specified vdev. If the vdev
+ * already has LDB credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR credits to assign to this vdev
+ *
+ * This function assigns num DIR credit to the specified vdev. If the vdev
+ * already has DIR credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of history list entries to assign to this vdev
+ *
+ * This function assigns num history list entries to the specified vdev. If the
+ * vdev already has history list entries assigned, this existing assignment is
+ * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
+ * this function may fail if a sufficiently large contiguous chunk is not
+ * available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of atomic inflights to assign to this vdev
+ *
+ * This function assigns num atomic inflights to the specified vdev. If the vdev
+ * already has atomic inflights assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of entries, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function takes any resources currently assigned to the vdev and
+ * reassigns them to the PF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_notify_vf() - send an alarm to a VF
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID
+ * @notification: notification
+ *
+ * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
+ *
+ * Return:
+ * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
+ */
+int dlb2_notify_vf(struct dlb2_hw *hw,
+		   unsigned int vf_id,
+		   u32 notification);
+
+/**
+ * dlb2_vdev_in_use() - query whether a virtual device is in use
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sends a mailbox request to the vdev to query whether the vdev
+ * is in use.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
+ * an internal error occurs.
+ */
+int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+enum dlb2_virt_mode {
+	DLB2_VIRT_NONE,
+	DLB2_VIRT_SRIOV,
+	DLB2_VIRT_SIOV,
+
+	/* NUM_DLB2_VIRT_MODES must be last */
+	NUM_DLB2_VIRT_MODES,
+};
+
+/**
+ * dlb2_hw_set_virt_mode() - set the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: either none, SR-IOV, or Scalable IOV.
+ *
+ * This function sets the virtualization mode of the device. This controls
+ * whether the device uses a software or hardware mailbox.
+ *
+ * This should be called by the PF driver when either SR-IOV or Scalable IOV is
+ * selected as the virtualization mechanism, and by the VF/VDEV driver during
+ * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
+ *
+ * Errors:
+ * EINVAL - Invalid mode.
+ */
+int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
+
+/**
+ * dlb2_hw_get_virt_mode() - get the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function gets the virtualization mode of the device.
+ */
+enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_register_sw_mbox() - register a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
+ * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
+ * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
+ * @inject_arg: user argument for pf2vdev_inject callback.
+ *
+ * When Scalable IOV is enabled, the VDCM must register a software mailbox for
+ * every virtual device during vdev creation.
+ *
+ * This function notifies the driver to use a software mailbox using the
+ * provided pointers, instead of the device's hardware mailbox. When the driver
+ * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
+ * go to the software mailbox instead of the hardware one. This is used in
+ * Scalable IOV virtualization.
+ */
+void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
+			      unsigned int vdev_id,
+			      u32 *vdev2pf_mbox,
+			      u32 *pf2vdev_mbox,
+			      void (*pf2vdev_inject)(void *),
+			      void *inject_arg);
+
+/**
+ * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function notifies the driver to stop using a previously registered
+ * software mailbox.
+ */
+void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ * @addr_lo: least-significant 32 bits of address.
+ * @data: 32 data bits.
+ *
+ * This sets up the CQ's IMS entry with the provided address and data values.
+ * This function should only be called if the device is configured for Scalable
+ * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
+ * not needed.
+ */
+void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb,
+				u32 addr_lo,
+				u32 data);
+
+/**
+ * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ *
+ * This clears the CQ's IMS entry, reverting it to its reset state.
+ */
+void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb);
+
+/**
+ * dlb2_hw_register_pasid() - register a vdev's PASID
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @pasid: the vdev's PASID.
+ *
+ * This function stores the user-supplied PASID, and uses it when configuring
+ * the vdev's CQs.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+int dlb2_hw_register_pasid(struct dlb2_hw *hw,
+			   unsigned int vdev_id,
+			   unsigned int pasid);
+
+/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vf_request,
+				unsigned int vf_id);
+
+/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ *	to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ *	port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ *	    cause the total bandwidth across all classes of service to exceed
+ *	    100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+enum dlb2_wd_tmo {
+	/* 40s watchdog timeout */
+	DLB2_WD_TMO_40S,
+	/* 10s watchdog timeout */
+	DLB2_WD_TMO_10S,
+	/* 1s watchdog timeout */
+	DLB2_WD_TMO_1S,
+
+	/* Must be last */
+	NUM_DLB2_WD_TMOS,
+};
+
+/**
+ * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
+ *	caller-specified timeout.
+ * @hw: dlb2_hw handle for a particular device.
+ * @tmo: watchdog timeout.
+ *
+ * This function should be called during device initialization and after reset.
+ * The watchdog timer interrupt must also be enabled per-CQ, using either
+ * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid timeout.
+ */
+int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
+
+/**
+ * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid directed port ID.
+ */
+int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid load-balanced port ID.
+ */
+int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *	ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * load-balanced CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_ldb_cq_interrupt().
+ */
+void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical directed port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * directed CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_dir_cq_interrupt().
+ */
+void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+#endif /* __DLB2_RESOURCE_H */
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
new file mode 100644
index 0000000..3c51cab
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -0,0 +1,621 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "base/dlb2_resource.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_regs.h"
+#include "dlb2_main.h"
+#include "../dlb2_user.h"
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+
+#define PF_ID_ZERO 0	/* PF ONLY! */
+#define NO_OWNER_VF 0	/* PF ONLY! */
+#define NOT_VF_REQ false /* PF ONLY! */
+
+#define DLB2_PCI_CFG_SPACE_SIZE 256
+#define DLB2_PCI_CAP_POINTER 0x34
+#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
+#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
+#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
+#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
+#define DLB2_PCI_EXT_CAP_ID_ERR 1
+#define DLB2_PCI_ERR_UNCOR_MASK 8
+#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
+
+#define DLB2_PCI_EXP_DEVCTL 8
+#define DLB2_PCI_LNKCTL 16
+#define DLB2_PCI_SLTCTL 24
+#define DLB2_PCI_RTCTL 28
+#define DLB2_PCI_EXP_DEVCTL2 40
+#define DLB2_PCI_LNKCTL2 48
+#define DLB2_PCI_SLTCTL2 56
+#define DLB2_PCI_CMD 4
+#define DLB2_PCI_X_CMD 2
+#define DLB2_PCI_EXP_DEVSTA 10
+#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
+#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
+
+#define DLB2_PCI_CAP_ID_EXP       0x10
+#define DLB2_PCI_CAP_ID_MSIX      0x11
+#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
+#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
+#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
+
+#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
+#define DLB2_PCI_PRI_ALLOC_REQ           0xC
+#define DLB2_PCI_PRI_CTRL                0x4
+#define DLB2_PCI_MSIX_FLAGS              0x2
+#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
+#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
+#define DLB2_PCI_ERR_ROOT_STATUS         0x30
+#define DLB2_PCI_ERR_COR_STATUS          0x10
+#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
+#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
+#define DLB2_PCI_ACS_CAP                 0x4
+#define DLB2_PCI_ACS_CTRL                0x6
+#define DLB2_PCI_ACS_SV                  0x1
+#define DLB2_PCI_ACS_RR                  0x4
+#define DLB2_PCI_ACS_CR                  0x8
+#define DLB2_PCI_ACS_UF                  0x10
+#define DLB2_PCI_ACS_EC                  0x20
+
+/* Stubs: Allow building partial probe patch */
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	int ret = 0;
+	RTE_SET_USED(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+/* End stubs */
+
+static int
+dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint32_t hdr;
+	size_t sz;
+	int pos;
+
+	pos = DLB2_PCI_CFG_SPACE_SIZE;
+	sz = sizeof(hdr);
+
+	while (pos > 0xFF) {
+		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
+			return -1;
+
+		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
+			return pos;
+
+		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint8_t pos;
+	int ret;
+	uint16_t hdr;
+
+	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
+	pos &= 0xFC;
+
+	if (ret != 1)
+		return -1;
+
+	while (pos > 0x3F) {
+		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
+		if (ret != 2)
+			return -1;
+
+		if (DLB2_PCI_CAP_ID(hdr) == id)
+			return pos;
+
+		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
+			return -1;
+
+		pos = DLB2_PCI_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+	rte_spinlock_init(&dlb2_dev->resource_mutex);
+	rte_spinlock_init(&dlb2_dev->measurement_lock);
+
+	return 0;
+}
+
+static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
+{
+	u32 retries = 0;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
+		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+		union dlb2_cfg_mstr_cfg_pm_status pm_st;
+		u32 addr;
+
+		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+			break;
+
+		rte_delay_ms(1);
+	};
+
+	if (retries == DLB2_READY_RETRY_LIMIT) {
+		printf("[%s()] wait for device ready timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+struct dlb2_dev *
+dlb2_probe(struct rte_pci_device *pdev)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret = 0;
+
+	DLB2_INFO(dlb2_dev, "probe\n");
+
+	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
+			      RTE_CACHE_LINE_SIZE);
+
+	if (dlb2_dev == NULL) {
+		ret = -ENOMEM;
+		goto dlb2_dev_malloc_fail;
+	}
+
+	/* PCI Bus driver has already mapped bar space into process.
+	 * Save off our IO register and FUNC addresses.
+	 */
+
+	/* BAR 0 */
+	if (pdev->mem_resource[0].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
+	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.func_kva,
+		  (void *)dlb2_dev->hw.func_phys_addr,
+		  (void *)(pdev->mem_resource[0].len));
+
+	/* BAR 2 */
+	if (pdev->mem_resource[2].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
+	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.csr_kva,
+		  (void *)dlb2_dev->hw.csr_phys_addr,
+		  (void *)(pdev->mem_resource[2].len));
+
+	dlb2_dev->pdev = pdev;
+
+	/* PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb2_pf_enable_pm(dlb2_dev);
+
+	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb2_pf_reset(dlb2_dev);
+	if (ret)
+		goto dlb2_reset_fail;
+
+	ret = dlb2_pf_init_driver_state(dlb2_dev);
+	if (ret)
+		goto init_driver_state_fail;
+
+	ret = dlb2_resource_init(&dlb2_dev->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	return dlb2_dev;
+
+resource_init_fail:
+	dlb2_resource_free(&dlb2_dev->hw);
+init_driver_state_fail:
+dlb2_reset_fail:
+pci_mmap_bad_addr:
+wait_for_device_ready_fail:
+	rte_free(dlb2_dev);
+dlb2_dev_malloc_fail:
+	rte_errno = ret;
+	return NULL;
+}
+
+int
+dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
+{
+	int ret = 0;
+	int i = 0;
+	uint32_t dword[16];
+	uint16_t cmd;
+	off_t off;
+
+	uint16_t dev_ctl_word;
+	uint16_t dev_ctl2_word;
+	uint16_t lnk_word;
+	uint16_t lnk_word2;
+	uint16_t slt_word;
+	uint16_t slt_word2;
+	uint16_t rt_ctl_word;
+	uint32_t pri_reqs_dword;
+	uint16_t pri_ctrl_word;
+
+	int pcie_cap_offset;
+	int pri_cap_offset;
+	int msix_cap_offset;
+	int err_cap_offset;
+	int acs_cap_offset;
+	int wait_count;
+
+	uint16_t devsta_busy_word;
+	uint16_t devctl_word;
+
+	struct rte_pci_device *pdev = dlb2_dev->pdev;
+
+	/* Save PCI config state */
+
+	for (i = 0; i < 16; i++) {
+		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
+			return ret;
+	}
+
+	pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
+
+	if (pcie_cap_offset < 0) {
+		printf("[%s()] failed to find the pcie capability\n",
+		       __func__);
+		return pcie_cap_offset;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
+		dev_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
+		lnk_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
+		slt_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_RTCTL;
+	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
+		rt_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
+		dev_ctl2_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
+		lnk_word2 = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
+		slt_word2 = 0;
+
+	off = DLB2_PCI_EXT_CAP_ID_PRI;
+	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (pri_cap_offset >= 0) {
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
+			pri_reqs_dword = 0;
+	}
+
+	/* clear the PCI command register before issuing the FLR */
+
+	off = DLB2_PCI_CMD;
+	cmd = 0;
+	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+		printf("[%s()] failed to write the pci command\n",
+		       __func__);
+		return ret;
+	}
+
+	/* issue the FLR */
+	for (wait_count = 0; wait_count < 4; wait_count++) {
+		int sleep_time;
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
+		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to read the pci device status\n",
+			       __func__);
+			return ret;
+		}
+
+		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
+			break;
+
+		sleep_time = (1 << (wait_count)) * 100;
+		rte_delay_ms(sleep_time);
+	}
+
+	if (wait_count == 4) {
+		printf("[%s()] wait for pci pending transactions timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to read the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
+
+	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to write the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	rte_delay_ms(100);
+
+	/* Restore PCI config state */
+
+	if (pcie_cap_offset >= 0) {
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie device control at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_RTCTL;
+		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	if (pri_cap_offset >= 0) {
+		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
+
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
+		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ERR;
+	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (err_cap_offset >= 0) {
+		uint32_t tmp;
+
+		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	for (i = 16; i > 0; i--) {
+		off = (i - 1) * 4;
+		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_CMD;
+	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
+		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+			printf("[%s()] failed to write the pci command\n",
+			       __func__);
+			return ret;
+		}
+	}
+
+	msix_cap_offset = dlb2_pci_find_capability(pdev,
+						   DLB2_PCI_CAP_ID_MSIX);
+	if (msix_cap_offset >= 0) {
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
+			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ACS;
+	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (acs_cap_offset >= 0) {
+		uint16_t acs_cap, acs_ctrl, acs_mask;
+		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
+		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
+			acs_cap = 0;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
+		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
+		acs_ctrl |= (acs_cap & acs_mask);
+
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
+		acs_mask |= DLB2_PCI_ACS_EC;
+		acs_ctrl &= ~acs_mask;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
new file mode 100644
index 0000000..f082a94
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -0,0 +1,98 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+#endif
+
+#include "base/dlb2_hw_types.h"
+#include "../dlb2_user.h"
+
+#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
+
+struct dlb2_dev;
+
+struct dlb2_port_memory {
+	struct dlb2_list_head list;
+	void *cq_base;
+	bool valid;
+};
+
+struct dlb2_dev {
+	struct rte_pci_device *pdev;
+	struct dlb2_hw hw;
+	/* struct list_head list; */
+	struct device *dlb2_device;
+	bool domain_reset_failed;
+	/* The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	rte_spinlock_t resource_mutex;
+	rte_spinlock_t measurement_lock;
+	bool worker_launched;
+	u8 revision;
+};
+
+struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
+int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
+int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+
+#endif /* __DLB2_MAIN_H */
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
new file mode 100644
index 0000000..ae4d785
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -0,0 +1,196 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_devargs.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_pci.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "../dlb2_priv.h"
+#include "../dlb2_inline_fns.h"
+#include "dlb2_main.h"
+#include "base/dlb2_hw_types.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_resource.h"
+
+static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
+
+/* Stubs: Allow building partial probe patch */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(arg);
+	RTE_SET_USED(vdev_req);
+	RTE_SET_USED(vdev_id);
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	RTE_SET_USED(hw);
+}
+/* End stubs */
+
+static void
+dlb2_pf_iface_fn_ptrs_init(void)
+{
+/* flexible iface fn ptr assignments will go here */
+}
+
+/* PCI DEV HOOKS */
+static int
+dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
+{
+	int ret = 0;
+	struct rte_pci_device *pci_dev;
+	struct dlb2_devargs dlb2_args = {
+		.socket_id = rte_socket_id(),
+		.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+		.num_dir_credits_override = -1,
+		.qid_depth_thresholds = { {0} },
+		.cos_id = DLB2_COS_DEFAULT
+	};
+	struct dlb2_eventdev *dlb2;
+
+	DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
+		     eventdev->data->dev_id, eventdev->data->socket_id);
+
+	dlb2_pf_iface_fn_ptrs_init();
+
+	pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
+
+		/* Probe the DLB2 PF layer */
+		dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
+
+		if (dlb2->qm_instance.pf_dev == NULL) {
+			DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
+				     rte_errno);
+			ret = -rte_errno;
+			goto dlb2_probe_failed;
+		}
+
+		/* Were we invoked with runtime parameters? */
+		if (pci_dev->device.devargs) {
+			ret = dlb2_parse_params(pci_dev->device.devargs->args,
+						pci_dev->device.devargs->name,
+						&dlb2_args);
+			if (ret) {
+				DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
+					     ret, rte_errno);
+				goto dlb2_probe_failed;
+			}
+		}
+
+		ret = dlb2_primary_eventdev_probe(eventdev,
+						  event_dlb2_pf_name,
+						  &dlb2_args);
+	} else {
+		ret = dlb2_secondary_eventdev_probe(eventdev,
+						    event_dlb2_pf_name);
+	}
+	if (ret)
+		goto dlb2_probe_failed;
+
+	DLB2_LOG_INFO("DLB2 PF Probe success\n");
+
+	return 0;
+
+dlb2_probe_failed:
+
+	DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
+
+	return ret;
+}
+
+#define EVENTDEV_INTEL_VENDOR_ID 0x8086
+
+static const struct rte_pci_id pci_id_dlb2_map[] = {
+	{
+		RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
+			       PCI_DEVICE_ID_INTEL_DLB2_PF)
+	},
+	{
+		.vendor_id = 0,
+	},
+};
+
+static int
+event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
+		     struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
+					     sizeof(struct dlb2_eventdev),
+					     dlb2_eventdev_pci_init,
+					     event_dlb2_pf_name);
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+}
+
+static int
+event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_remove(pci_dev, NULL);
+
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+
+}
+
+static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
+	.id_table = pci_id_dlb2_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe = event_dlb2_pci_probe,
+	.remove = event_dlb2_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
-- 
2.6.4

[dpdk-dev] [PATCH v4 07/23] event/dlb2: add flexible interface

[Timothy McDaniel]

This commit introduces the flexible interface. This
interface allows the core code to operate in PF mode (direct
hardware access) or bifurcated mode (hardware configured via
kernel driver). This driver currently only supports PF mode
but bifurcated mode will be added in a future DPDK patch-set.
Note that the flexible interface is not used for data path
operations, and thus there are no performance concerns
related to the use of function pointers.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_iface.c | 28 ++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.h | 29 +++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build  |  1 +
 3 files changed, 58 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h

diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
new file mode 100644
index 0000000..0fc9991
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+
+#include "dlb2_priv.h"
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+void (*dlb2_iface_low_level_io_init)(void);
+
+int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+				     uint8_t *revision);
+
+void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+				   enum dlb2_cq_poll_modes *mode);
+
+int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
new file mode 100644
index 0000000..4fb416e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_IFACE_H_
+#define _DLB2_IFACE_H_
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+extern void (*dlb2_iface_low_level_io_init)(void);
+
+extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+					    uint8_t *revision);
+
+extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+					  enum dlb2_cq_poll_modes *mode);
+
+extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
+
+#endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 1eab9b9..491e76d 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -8,6 +8,7 @@ if not is_linux or not dpdk_conf.has('RTE_ARCH_X86_64')
 endif
 
 sources = files('dlb2.c',
+		'dlb2_iface.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c'
 )
-- 
2.6.4

[dpdk-dev] [PATCH v4 08/23] event/dlb2: add probe-time hardware init

[Timothy McDaniel]

This commit adds probe-time low level hardware
initialization.  It also adds probe-time init for both
primary and secondary DPDK processes.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 188 +++++++++++++++++++-
 drivers/event/dlb2/meson.build             |   3 +-
 drivers/event/dlb2/pf/base/dlb2_resource.c | 274 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  23 +--
 drivers/event/dlb2/pf/dlb2_main.h          |   1 -
 drivers/event/dlb2/pf/dlb2_pf.c            |  78 ++++++--
 6 files changed, 523 insertions(+), 44 deletions(-)
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 1b11979..16653db 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -31,6 +31,7 @@
 #include <rte_string_fns.h>
 
 #include "dlb2_priv.h"
+#include "dlb2_iface.h"
 #include "dlb2_inline_fns.h"
 
 /*
@@ -40,10 +41,108 @@
 #if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
 #error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
 #endif
+static struct rte_event_dev_info evdev_dlb2_default_info = {
+	.driver_name = "", /* probe will set */
+	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
+	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
+	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
+#else
+	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
+#endif
+	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
+	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
+	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
+	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
+	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
+	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+	.max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
+	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
+			  RTE_EVENT_DEV_CAP_EVENT_QOS |
+			  RTE_EVENT_DEV_CAP_BURST_MODE |
+			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
+			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
+			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
+};
 
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/* override defaults with value(s) provided on command line */
+static void
+dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
+				 int *qid_depth_thresholds)
+{
+	int q;
+
+	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
+		if (qid_depth_thresholds[q] != 0)
+			dlb2->ev_queues[q].depth_threshold =
+				qid_depth_thresholds[q];
+	}
+}
+
+static int
+dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
+	int ret;
+
+	/* Query driver resources provisioned for this device */
+
+	ret = dlb2_iface_get_num_resources(handle,
+					   &dlb2->hw_rsrc_query_results);
+	if (ret) {
+		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n", ret);
+		return ret;
+	}
+
+	/* Complete filling in device resource info returned to evdev app,
+	 * overriding any default values.
+	 * The capabilities (CAPs) were set at compile time.
+	 */
+
+	evdev_dlb2_default_info.max_event_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	evdev_dlb2_default_info.max_event_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	evdev_dlb2_default_info.max_num_events =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	/* Save off values used when creating the scheduling domain. */
+
+	handle->info.num_sched_domains =
+		dlb2->hw_rsrc_query_results.num_sched_domains;
+
+	handle->info.hw_rsrc_max.nb_events_limit =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	handle->info.hw_rsrc_max.num_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues +
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.num_ldb_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	handle->info.hw_rsrc_max.num_ldb_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	handle->info.hw_rsrc_max.num_dir_ports =
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.reorder_window_size =
+		dlb2->hw_rsrc_query_results.num_hist_list_entries;
+
+	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
+
+	return 0;
+}
+
 #define DLB2_BASE_10 10
 
 static int
@@ -235,14 +334,71 @@ set_qid_depth_thresh(const char *key __rte_unused,
 	return 0;
 }
 
+static void
+dlb2_entry_points_init(struct rte_eventdev *dev)
+{
+	RTE_SET_USED(dev);
+
+	/* Eventdev PMD entry points */
+}
+
 int
 dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    const char *name,
 			    struct dlb2_devargs *dlb2_args)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
-	RTE_SET_USED(dlb2_args);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	dlb2->event_dev = dev; /* backlink */
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	dlb2->max_num_events_override = dlb2_args->max_num_events;
+	dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
+	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
+					    &dlb2->revision);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_hardware_init(&dlb2->qm_instance);
+
+	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
+			     err);
+		return err;
+	}
+
+	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	dlb2_init_queue_depth_thresholds(dlb2,
+					 dlb2_args->qid_depth_thresholds.val);
 
 	return 0;
 }
@@ -251,8 +407,30 @@ int
 dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
 			      const char *name)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(name);
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 491e76d..28fef3f 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -10,7 +10,8 @@ endif
 sources = files('dlb2.c',
 		'dlb2_iface.c',
 		'pf/dlb2_main.c',
-		'pf/dlb2_pf.c'
+		'pf/dlb2_pf.c',
+		'pf/base/dlb2_resource.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
new file mode 100644
index 0000000..6de8b95
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_mbox.h"
+#include "dlb2_osdep.h"
+#include "dlb2_osdep_bitmap.h"
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+	int i;
+
+	dlb2_list_init_head(&domain->used_ldb_queues);
+	dlb2_list_init_head(&domain->used_dir_pq_pairs);
+	dlb2_list_init_head(&domain->avail_ldb_queues);
+	dlb2_list_init_head(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+	int i;
+
+	dlb2_list_init_head(&rsrc->avail_domains);
+	dlb2_list_init_head(&rsrc->used_domains);
+	dlb2_list_init_head(&rsrc->avail_ldb_queues);
+	dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
+	arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
+	arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
+	arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb2_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	struct dlb2_list_entry *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. This is application
+	 * dependent, but the driver interleaves port IDs as much as possible
+	 * to reduce the likelihood of this. This initial allocation maximizes
+	 * the average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	/* Zero-out resource tracking data structures */
+	memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+	memset(&hw->pf, 0, sizeof(hw->pf));
+
+	dlb2_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+		dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+		dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_domains, list);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_ldb_queues, list);
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+		struct dlb2_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
+			      &port->func_list);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+				DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+	if (ret)
+		goto unwind;
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+					DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+		if (ret)
+			goto unwind;
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb2_resource_free(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+	r0.field.disable = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 3c51cab..210b3186 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -19,6 +19,7 @@
 #include "dlb2_main.h"
 #include "../dlb2_user.h"
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 
 #define PF_ID_ZERO 0	/* PF ONLY! */
@@ -72,27 +73,6 @@
 #define DLB2_PCI_ACS_UF                  0x10
 #define DLB2_PCI_ACS_EC                  0x20
 
-/* Stubs: Allow building partial probe patch */
-void dlb2_resource_free(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-int dlb2_resource_init(struct dlb2_hw *hw)
-{
-	int ret = 0;
-	RTE_SET_USED(hw);
-
-	return ret;
-}
-
-void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
-{
-	RTE_SET_USED(hw);
-}
-
-/* End stubs */
-
 static int
 dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
 {
@@ -149,7 +129,6 @@ static int
 dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
 {
 	rte_spinlock_init(&dlb2_dev->resource_mutex);
-	rte_spinlock_init(&dlb2_dev->measurement_lock);
 
 	return 0;
 }
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
index f082a94..f3bee71 100644
--- a/drivers/event/dlb2/pf/dlb2_main.h
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -38,7 +38,6 @@ struct dlb2_dev {
 	 * hardware registers.
 	 */
 	rte_spinlock_t resource_mutex;
-	rte_spinlock_t measurement_lock;
 	bool worker_launched;
 	u8 revision;
 };
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index ae4d785..7d64309 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -32,6 +32,7 @@
 #include <rte_string_fns.h>
 
 #include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
 #include "../dlb2_inline_fns.h"
 #include "dlb2_main.h"
 #include "base/dlb2_hw_types.h"
@@ -40,35 +41,82 @@
 
 static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
 
-/* Stubs: Allow building partial probe patch */
-int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
-			      struct dlb2_get_num_resources_args *arg,
-			      bool vdev_req,
-			      unsigned int vdev_id)
+static void
+dlb2_pf_low_level_io_init(void)
+{
+	int i;
+	/* Addresses will be initialized at port create */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		/* First directed ports */
+		dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
+
+		/* Now load balanced ports */
+		dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
+	}
+}
+
+static int
+dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
+{
+	RTE_SET_USED(handle);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+static int
+dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
+			   uint8_t *revision)
 {
-	RTE_SET_USED(hw);
-	RTE_SET_USED(arg);
-	RTE_SET_USED(vdev_req);
-	RTE_SET_USED(vdev_id);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	*revision = dlb2_dev->revision;
 
 	return 0;
 }
 
-void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+static void
+dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+	dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_num_resources_args *rsrcs)
 {
-	RTE_SET_USED(hw);
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
 }
 
-void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+static int
+dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
+			 enum dlb2_cq_poll_modes *mode)
 {
-	RTE_SET_USED(hw);
+	RTE_SET_USED(handle);
+
+	*mode = DLB2_CQ_POLL_MODE_SPARSE;
+
+	return 0;
 }
-/* End stubs */
 
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
-/* flexible iface fn ptr assignments will go here */
+	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
+	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
+	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
+	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
+	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v4 09/23] event/dlb2: add xstats

[Timothy McDaniel]

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
Reviewed-by: Chen, Mike Ximing <mike.ximing.chen@intel.com>
---
 drivers/event/dlb2/dlb2.c        |   35 +-
 drivers/event/dlb2/dlb2_xstats.c | 1240 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build   |    1 +
 3 files changed, 1273 insertions(+), 3 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 16653db..925824e 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,6 +70,21 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/*
+ * DUMMY - added so that xstats path will compile/link.
+ * Will be replaced by real version in a subsequent
+ * patch.
+ */
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	RTE_SET_USED(dlb2);
+	RTE_SET_USED(queue);
+
+	return 0;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -337,9 +352,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
-	RTE_SET_USED(dev);
-
-	/* Eventdev PMD entry points */
+	/* Expose PMD's eventdev interface */
+	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dump             = dlb2_eventdev_dump,
+		.xstats_get       = dlb2_eventdev_xstats_get,
+		.xstats_get_names = dlb2_eventdev_xstats_get_names,
+		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
+		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+	};
+
+	dev->dev_ops = &dlb2_eventdev_entry_ops;
 }
 
 int
@@ -391,6 +413,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Complete xtstats runtime initialization */
+	err = dlb2_xstats_init(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
+		return err;
+	}
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
new file mode 100644
index 0000000..21391f7
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_xstats.c
@@ -0,0 +1,1240 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <inttypes.h>
+
+#include <rte_malloc.h>
+#include <rte_eventdev.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+enum dlb2_xstats_type {
+	/* common to device and port */
+	rx_ok,				/**< Receive an event */
+	rx_drop,                        /**< Error bit set in received QE */
+	rx_interrupt_wait,		/**< Wait on an interrupt */
+	rx_umonitor_umwait,		/**< Block using umwait */
+	tx_ok,				/**< Transmit an event */
+	total_polls,			/**< Call dequeue_burst */
+	zero_polls,			/**< Call dequeue burst and return 0 */
+	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
+	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
+	tx_nospc_inflight_max,		/**< Reach the new_event_threshold */
+	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
+	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
+	/* device specific */
+	nb_events_limit,
+	inflight_events,
+	ldb_pool_size,
+	dir_pool_size,
+	/* port specific */
+	tx_new,				/**< Send an OP_NEW event */
+	tx_fwd,				/**< Send an OP_FORWARD event */
+	tx_rel,				/**< Send an OP_RELEASE event */
+	tx_implicit_rel,		/**< Issue an implicit event release */
+	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED event */
+	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL event */
+	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event */
+	tx_sched_directed,		/**< Send a directed event */
+	tx_invalid,                     /**< Send an event with an invalid op */
+	outstanding_releases,		/**< # of releases a port owes */
+	max_outstanding_releases,	/**< max # of releases a port can owe */
+	rx_sched_ordered,		/**< Dequeue an ordered event */
+	rx_sched_unordered,		/**< Dequeue an unordered event */
+	rx_sched_atomic,		/**< Dequeue an atomic event */
+	rx_sched_directed,		/**< Dequeue an directed event */
+	rx_sched_invalid,               /**< Dequeue event sched type invalid */
+	/* common to port and queue */
+	is_configured,			/**< Port is configured */
+	is_load_balanced,		/**< Port is LDB */
+	hw_id,				/**< Hardware ID */
+	/* queue specific */
+	num_links,			/**< Number of ports linked */
+	sched_type,			/**< Queue sched type */
+	enq_ok,				/**< # events enqueued to the queue */
+	current_depth,			/**< Current queue depth */
+	depth_threshold,		/**< Programmed depth threshold */
+	depth_le50_threshold,
+	/**< Depth LE to 50% of the configured hardware threshold */
+	depth_gt50_le75_threshold,
+	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold */
+	depth_gt75_le100_threshold,
+	/**< Depth GT 75%. but LE to the configured hardware threshold */
+	depth_gt100_threshold
+	/**< Depth GT 100% of the configured hw threshold */
+};
+
+typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
+		uint16_t obj_idx, /* port or queue id */
+		enum dlb2_xstats_type stat, int extra_arg);
+
+enum dlb2_xstats_fn_type {
+	DLB2_XSTATS_FN_DEV,
+	DLB2_XSTATS_FN_PORT,
+	DLB2_XSTATS_FN_QUEUE
+};
+
+struct dlb2_xstats_entry {
+	struct rte_event_dev_xstats_name name;
+	uint64_t reset_value; /* an offset to be taken away to emulate resets */
+	enum dlb2_xstats_fn_type fn_id;
+	enum dlb2_xstats_type stat;
+	enum rte_event_dev_xstats_mode mode;
+	int extra_arg;
+	uint16_t obj_idx;
+	uint8_t reset_allowed; /* when set, this value can be reset */
+};
+
+/* Some device stats are simply a summation of the corresponding port values */
+static uint64_t
+dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
+			     int which_stat)
+{
+	int i;
+	uint64_t val = 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
+
+		if (!port->setup_done)
+			continue;
+
+		switch (which_stat) {
+		case rx_ok:
+			val += port->stats.traffic.rx_ok;
+			break;
+		case rx_drop:
+			val += port->stats.traffic.rx_drop;
+			break;
+		case rx_interrupt_wait:
+			val += port->stats.traffic.rx_interrupt_wait;
+			break;
+		case rx_umonitor_umwait:
+			val += port->stats.traffic.rx_umonitor_umwait;
+			break;
+		case tx_ok:
+			val += port->stats.traffic.tx_ok;
+			break;
+		case total_polls:
+			val += port->stats.traffic.total_polls;
+			break;
+		case zero_polls:
+			val += port->stats.traffic.zero_polls;
+			break;
+		case tx_nospc_ldb_hw_credits:
+			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
+			break;
+		case tx_nospc_dir_hw_credits:
+			val += port->stats.traffic.tx_nospc_dir_hw_credits;
+			break;
+		case tx_nospc_inflight_max:
+			val += port->stats.traffic.tx_nospc_inflight_max;
+			break;
+		case tx_nospc_new_event_limit:
+			val += port->stats.traffic.tx_nospc_new_event_limit;
+			break;
+		case tx_nospc_inflight_credits:
+			val += port->stats.traffic.tx_nospc_inflight_credits;
+			break;
+		default:
+			return -1;
+		}
+	}
+	return val;
+}
+
+static uint64_t
+get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
+	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	switch (type) {
+	case rx_ok:
+	case rx_drop:
+	case rx_interrupt_wait:
+	case rx_umonitor_umwait:
+	case tx_ok:
+	case total_polls:
+	case zero_polls:
+	case tx_nospc_ldb_hw_credits:
+	case tx_nospc_dir_hw_credits:
+	case tx_nospc_inflight_max:
+	case tx_nospc_new_event_limit:
+	case tx_nospc_inflight_credits:
+		return dlb2_device_traffic_stat_get(dlb2, type);
+	case nb_events_limit:
+		return dlb2->new_event_limit;
+	case inflight_events:
+		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
+	case ldb_pool_size:
+		return dlb2->num_ldb_credits;
+	case dir_pool_size:
+		return dlb2->num_dir_credits;
+	default: return -1;
+	}
+}
+
+static uint64_t
+get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
+
+	switch (type) {
+	case rx_ok: return ev_port->stats.traffic.rx_ok;
+
+	case rx_drop: return ev_port->stats.traffic.rx_drop;
+
+	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
+
+	case rx_umonitor_umwait:
+		return ev_port->stats.traffic.rx_umonitor_umwait;
+
+	case tx_ok: return ev_port->stats.traffic.tx_ok;
+
+	case total_polls: return ev_port->stats.traffic.total_polls;
+
+	case zero_polls: return ev_port->stats.traffic.zero_polls;
+
+	case tx_nospc_ldb_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
+
+	case tx_nospc_dir_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
+
+	case tx_nospc_inflight_max:
+		return ev_port->stats.traffic.tx_nospc_inflight_max;
+
+	case tx_nospc_new_event_limit:
+		return ev_port->stats.traffic.tx_nospc_new_event_limit;
+
+	case tx_nospc_inflight_credits:
+		return ev_port->stats.traffic.tx_nospc_inflight_credits;
+
+	case is_configured: return ev_port->setup_done;
+
+	case is_load_balanced: return !ev_port->qm_port.is_directed;
+
+	case hw_id: return ev_port->qm_port.id;
+
+	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
+
+	case tx_fwd: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
+
+	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
+
+	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
+
+	case tx_sched_ordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case tx_sched_unordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case tx_sched_atomic:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case tx_sched_directed:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case tx_invalid: return ev_port->stats.tx_invalid;
+
+	case outstanding_releases: return ev_port->outstanding_releases;
+
+	case max_outstanding_releases:
+		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	case rx_sched_ordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case rx_sched_unordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case rx_sched_atomic:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case rx_sched_directed:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
+
+	default: return -1;
+	}
+}
+
+static uint64_t
+dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
+{
+	int port = 0;
+	uint64_t tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
+
+	return tally;
+}
+
+static uint64_t
+dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
+{
+	int port = 0;
+	uint64_t enq_ok_tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
+
+	return enq_ok_tally;
+}
+
+static uint64_t
+get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_queue *ev_queue =
+		&dlb2->ev_queues[obj_idx];
+
+	switch (type) {
+	case is_configured: return ev_queue->setup_done;
+
+	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
+
+	case hw_id: return ev_queue->qm_queue.id;
+
+	case num_links: return ev_queue->num_links;
+
+	case sched_type: return ev_queue->qm_queue.sched_type;
+
+	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
+
+	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
+
+	case depth_threshold: return ev_queue->depth_threshold;
+
+	case depth_le50_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_LE50);
+
+	case depth_gt50_le75_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT50_LE75);
+
+	case depth_gt75_le100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT75_LE100);
+
+	case depth_gt100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT100);
+
+	default: return -1;
+	}
+}
+
+int
+dlb2_xstats_init(struct dlb2_eventdev *dlb2)
+{
+	/*
+	 * define the stats names and types. Used to build up the device
+	 * xstats array
+	 * There are multiple set of stats:
+	 *   - device-level,
+	 *   - per-port,
+	 *   - per-qid,
+	 *
+	 * For each of these sets, we have three parallel arrays, one for the
+	 * names, the other for the stat type parameter to be passed in the fn
+	 * call to get that stat. The third array allows resetting or not.
+	 * All these arrays must be kept in sync
+	 */
+	static const char * const dev_stats[] = {
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"nb_events_limit",
+		"inflight_events",
+		"ldb_pool_size",
+		"dir_pool_size",
+	};
+	static const enum dlb2_xstats_type dev_types[] = {
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		nb_events_limit,
+		inflight_events,
+		ldb_pool_size,
+		dir_pool_size,
+	};
+	/* Note: generated device stats are not allowed to be reset. */
+	static const uint8_t dev_reset_allowed[] = {
+		0, /* rx_ok */
+		0, /* rx_drop */
+		0, /* rx_interrupt_wait */
+		0, /* rx_umonitor_umwait */
+		0, /* tx_ok */
+		0, /* total_polls */
+		0, /* zero_polls */
+		0, /* tx_nospc_ldb_hw_credits */
+		0, /* tx_nospc_dir_hw_credits */
+		0, /* tx_nospc_inflight_max */
+		0, /* tx_nospc_new_event_limit */
+		0, /* tx_nospc_inflight_credits */
+		0, /* nb_events_limit */
+		0, /* inflight_events */
+		0, /* ldb_pool_size */
+		0, /* dir_pool_size */
+	};
+	static const char * const port_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"tx_new",
+		"tx_fwd",
+		"tx_rel",
+		"tx_implicit_rel",
+		"tx_sched_ordered",
+		"tx_sched_unordered",
+		"tx_sched_atomic",
+		"tx_sched_directed",
+		"tx_invalid",
+		"outstanding_releases",
+		"max_outstanding_releases",
+		"rx_sched_ordered",
+		"rx_sched_unordered",
+		"rx_sched_atomic",
+		"rx_sched_directed",
+		"rx_sched_invalid"
+	};
+	static const enum dlb2_xstats_type port_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		tx_new,
+		tx_fwd,
+		tx_rel,
+		tx_implicit_rel,
+		tx_sched_ordered,
+		tx_sched_unordered,
+		tx_sched_atomic,
+		tx_sched_directed,
+		tx_invalid,
+		outstanding_releases,
+		max_outstanding_releases,
+		rx_sched_ordered,
+		rx_sched_unordered,
+		rx_sched_atomic,
+		rx_sched_directed,
+		rx_sched_invalid
+	};
+	static const uint8_t port_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		1, /* rx_ok */
+		1, /* rx_drop */
+		1, /* rx_interrupt_wait */
+		1, /* rx_umonitor_umwait */
+		1, /* tx_ok */
+		1, /* total_polls */
+		1, /* zero_polls */
+		1, /* tx_nospc_ldb_hw_credits */
+		1, /* tx_nospc_dir_hw_credits */
+		1, /* tx_nospc_inflight_max */
+		1, /* tx_nospc_new_event_limit */
+		1, /* tx_nospc_inflight_credits */
+		1, /* tx_new */
+		1, /* tx_fwd */
+		1, /* tx_rel */
+		1, /* tx_implicit_rel */
+		1, /* tx_sched_ordered */
+		1, /* tx_sched_unordered */
+		1, /* tx_sched_atomic */
+		1, /* tx_sched_directed */
+		1, /* tx_invalid */
+		0, /* outstanding_releases */
+		0, /* max_outstanding_releases */
+		1, /* rx_sched_ordered */
+		1, /* rx_sched_unordered */
+		1, /* rx_sched_atomic */
+		1, /* rx_sched_directed */
+		1  /* rx_sched_invalid */
+	};
+
+	/* QID specific stats */
+	static const char * const qid_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"num_links",
+		"sched_type",
+		"enq_ok",
+		"current_depth",
+		"depth_threshold",
+		"depth_le50_threshold",
+		"depth_gt50_le75_threshold",
+		"depth_gt75_le100_threshold",
+		"depth_gt100_threshold",
+	};
+	static const enum dlb2_xstats_type qid_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		num_links,
+		sched_type,
+		enq_ok,
+		current_depth,
+		depth_threshold,
+		depth_le50_threshold,
+		depth_gt50_le75_threshold,
+		depth_gt75_le100_threshold,
+		depth_gt100_threshold,
+	};
+	static const uint8_t qid_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		0, /* num_links */
+		0, /* sched_type */
+		1, /* enq_ok */
+		0, /* current_depth */
+		0, /* depth_threshold */
+		1, /* depth_le50_threshold */
+		1, /* depth_gt50_le75_threshold */
+		1, /* depth_gt75_le100_threshold */
+		1, /* depth_gt100_threshold */
+	};
+
+	/* ---- end of stat definitions ---- */
+
+	/* check sizes, since a missed comma can lead to strings being
+	 * joined by the compiler.
+	 */
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
+
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_reset_allowed));
+
+	/* other vars */
+	const unsigned int count = RTE_DIM(dev_stats) +
+			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
+			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
+	unsigned int i, port, qid, stat_id = 0;
+
+	dlb2->xstats = rte_zmalloc_socket(NULL,
+			sizeof(dlb2->xstats[0]) * count, 0,
+			dlb2->qm_instance.info.socket_id);
+	if (dlb2->xstats == NULL)
+		return -ENOMEM;
+
+#define sname dlb2->xstats[stat_id].name.name
+	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
+		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
+			.fn_id = DLB2_XSTATS_FN_DEV,
+			.stat = dev_types[i],
+			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
+			.reset_allowed = dev_reset_allowed[i],
+		};
+		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
+	}
+	dlb2->xstats_count_mode_dev = stat_id;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
+		dlb2->xstats_offset_for_port[port] = stat_id;
+
+		uint32_t count_offset = stat_id;
+
+		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_PORT,
+				.obj_idx = port,
+				.stat = port_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_PORT,
+				.reset_allowed = port_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "port_%u_%s",
+				 port, port_stats[i]);
+		}
+
+		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_port = stat_id - dlb2->xstats_count_mode_dev;
+
+	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
+		uint32_t count_offset = stat_id;
+
+		dlb2->xstats_offset_for_qid[qid] = stat_id;
+
+		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_QUEUE,
+				.obj_idx = qid,
+				.stat = qid_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
+				.reset_allowed = qid_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "qid_%u_%s",
+				 qid, qid_stats[i]);
+		}
+
+		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_queue = stat_id -
+		(dlb2->xstats_count_mode_dev + dlb2->xstats_count_mode_port);
+#undef sname
+
+	dlb2->xstats_count = stat_id;
+
+	return 0;
+}
+
+void
+dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
+{
+	rte_free(dlb2->xstats);
+	dlb2->xstats_count = 0;
+}
+
+int
+dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstats_names,
+		unsigned int *ids, unsigned int size)
+{
+	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+	uint32_t start_offset = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			break;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			break;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
+		break;
+	default:
+		return -EINVAL;
+	};
+
+	if (xstats_mode_count > size || ids == NULL || xstats_names == NULL)
+		return xstats_mode_count;
+
+	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
+		if (dlb2->xstats[i].mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != dlb2->xstats[i].obj_idx)
+			continue;
+
+		xstats_names[xidx] = dlb2->xstats[i].name;
+		if (ids)
+			ids[xidx] = start_offset + xidx;
+		xidx++;
+	}
+	return xidx;
+}
+
+static int
+dlb2_xstats_update(struct dlb2_eventdev *dlb2,
+		enum rte_event_dev_xstats_mode mode,
+		uint8_t queue_port_id, const unsigned int ids[],
+		uint64_t values[], unsigned int n, const uint32_t reset)
+{
+	unsigned int i;
+	unsigned int xidx = 0;
+	uint32_t xstats_mode_count = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			goto invalid_value;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			goto invalid_value;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		break;
+	default:
+		goto invalid_value;
+	};
+
+	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
+		dlb2_xstats_fn fn;
+
+		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != xs->obj_idx)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			goto invalid_value;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+
+		if (values)
+			values[xidx] = val;
+
+		if (xs->reset_allowed && reset)
+			xs->reset_value += val;
+
+		xidx++;
+	}
+
+	return xidx;
+
+invalid_value:
+	return -EINVAL;
+}
+
+int
+dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	const uint32_t reset = 0;
+
+	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
+				  reset);
+}
+
+uint64_t
+dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+				 const char *name, unsigned int *id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	dlb2_xstats_fn fn;
+
+	for (i = 0; i < dlb2->xstats_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (strncmp(xs->name.name, name,
+			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
+			if (id != NULL)
+				*id = i;
+
+			switch (xs->fn_id) {
+			case DLB2_XSTATS_FN_DEV:
+				fn = get_dev_stat;
+				break;
+			case DLB2_XSTATS_FN_PORT:
+				fn = get_port_stat;
+				break;
+			case DLB2_XSTATS_FN_QUEUE:
+				fn = get_queue_stat;
+				break;
+			default:
+				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
+					  xs->fn_id);
+				return (uint64_t)-1;
+			}
+
+			return fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+		}
+	}
+	if (id != NULL)
+		*id = (uint32_t)-1;
+	return (uint64_t)-1;
+}
+
+static void
+dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
+			uint32_t num)
+{
+	uint32_t i;
+	dlb2_xstats_fn fn;
+
+	for (i = start; i < start + num; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (!xs->reset_allowed)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			return;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
+		xs->reset_value = val;
+	}
+}
+
+static int
+dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
+			const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_QUEUE,
+					queue_id, ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	if (ids == NULL)
+		dlb2_xstats_reset_range(dlb2,
+			dlb2->xstats_offset_for_qid[queue_id],
+			dlb2->xstats_count_per_qid[queue_id]);
+
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
+		       const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+	int offset = dlb2->xstats_offset_for_port[port_id];
+	int nb_stat = dlb2->xstats_count_per_port[port_id];
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_PORT, port_id,
+					ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
+		      uint32_t nb_ids)
+{
+	uint32_t i;
+
+	if (ids) {
+		for (i = 0; i < nb_ids; i++) {
+			uint32_t id = ids[i];
+
+			if (id >= dlb2->xstats_count_mode_dev)
+				return -EINVAL;
+			dlb2_xstats_reset_range(dlb2, id, 1);
+		}
+	} else {
+		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
+			dlb2_xstats_reset_range(dlb2, i, 1);
+	}
+
+	return 0;
+}
+
+int
+dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+			   enum rte_event_dev_xstats_mode mode,
+			   int16_t queue_port_id,
+			   const uint32_t ids[],
+			   uint32_t nb_ids)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	uint32_t i;
+
+	/* handle -1 for queue_port_id here, looping over all ports/queues */
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
+			return -EINVAL;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+				if (dlb2_xstats_reset_port(dlb2, i,
+							   ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
+			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
+						   ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
+				if (dlb2_xstats_reset_queue(dlb2, i,
+							    ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
+			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
+						    ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	};
+
+	return 0;
+}
+
+void
+dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	int i;
+
+	if (f == NULL) {
+		printf("Invalid file pointer\n");
+		return;
+	}
+
+	if (dev == NULL) {
+		fprintf(f, "Invalid event device\n");
+		return;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (dlb2 == NULL) {
+		fprintf(f, "DLB2 Event device cannot be dumped!\n");
+		return;
+	}
+
+	if (!dlb2->configured)
+		fprintf(f, "DLB2 Event device is not configured\n");
+
+	handle = &dlb2->qm_instance;
+
+	fprintf(f, "================\n");
+	fprintf(f, "DLB2 Device Dump\n");
+	fprintf(f, "================\n");
+
+	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
+		dlb2->umwait_allowed ? "yes" : "no");
+
+	/* Generic top level device information */
+
+	fprintf(f, "device is configured and run state =");
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		fprintf(f, "STOPPED\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
+		fprintf(f, "STOPPING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
+		fprintf(f, "STARTING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
+		fprintf(f, "STARTED\n");
+	else
+		fprintf(f, "UNEXPECTED\n");
+
+	fprintf(f, "domain ID=%u, socket_id=%u, evdev=%p\n",
+		handle->domain_id, handle->info.socket_id, dlb2->event_dev);
+
+	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
+		dlb2->num_dir_ports, dlb2->num_dir_queues);
+
+	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
+		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
+
+	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
+		handle->cfg.resources.num_atomic_inflights,
+		handle->cfg.resources.num_hist_list_entries);
+
+	fprintf(f, "results from most recent hw resource query:\n");
+
+	fprintf(f, "\tnum_sched_domains = %u\n",
+		dlb2->hw_rsrc_query_results.num_sched_domains);
+
+	fprintf(f, "\tnum_ldb_queues = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_queues);
+
+	fprintf(f, "\tnum_ldb_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_ports);
+
+	fprintf(f, "\tnum_dir_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_ports);
+
+	fprintf(f, "\tnum_atomic_inflights = %u\n",
+		dlb2->hw_rsrc_query_results.num_atomic_inflights);
+
+	fprintf(f, "\tnum_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.num_hist_list_entries);
+
+	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
+
+	fprintf(f, "\tnum_ldb_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_credits);
+
+	fprintf(f, "\tnum_dir_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_credits);
+
+	/* Port level information */
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
+		int j;
+
+		if (!p->enq_configured)
+			fprintf(f, "Port_%d is not configured\n", i);
+
+		fprintf(f, "Port_%d\n", i);
+		fprintf(f, "=======\n");
+
+		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
+			p->id, p->setup_done);
+
+		fprintf(f, "\tconfig state=%d, port state=%d\n",
+			p->qm_port.config_state, p->qm_port.state);
+
+		fprintf(f, "\tport is %s\n",
+			p->qm_port.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\toutstanding releases=%u\n",
+			p->outstanding_releases);
+
+		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
+			p->inflight_max, p->inflight_credits);
+
+		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
+			p->credit_update_quanta, p->implicit_release);
+
+		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (p->link[j].valid)
+				fprintf(f, "id=%u prio=%u ",
+					p->link[j].queue_id,
+					p->link[j].priority);
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
+
+		fprintf(f, "\tcached_ldb_credits=%u\n",
+			p->qm_port.cached_ldb_credits);
+
+		fprintf(f, "\tldb_credits = %u\n",
+			p->qm_port.ldb_credits);
+
+		fprintf(f, "\tcached_dir_credits = %u\n",
+			p->qm_port.cached_dir_credits);
+
+		fprintf(f, "\tdir_credits = %u\n",
+			p->qm_port.dir_credits);
+
+		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
+			p->qm_port.gen_bit,
+			p->qm_port.cq_idx,
+			p->qm_port.cq_depth);
+
+		fprintf(f, "\tinterrupt armed=%d\n",
+			p->qm_port.int_armed);
+
+		fprintf(f, "\tPort statistics\n");
+
+		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
+			p->stats.traffic.rx_ok);
+
+		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
+			p->stats.traffic.rx_drop);
+
+		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
+			p->stats.traffic.rx_interrupt_wait);
+
+		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
+			p->stats.traffic.rx_umonitor_umwait);
+
+		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
+			p->stats.traffic.tx_ok);
+
+		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
+			p->stats.traffic.total_polls);
+
+		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
+			p->stats.traffic.zero_polls);
+
+		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_ldb_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_dir_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_max);
+
+		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_new_event_limit);
+
+		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_credits);
+
+		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
+
+		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
+
+		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
+
+		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
+			p->stats.tx_implicit_rel);
+
+		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
+			p->stats.tx_invalid);
+
+		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
+			p->stats.rx_sched_invalid);
+	}
+
+	/* Queue level information */
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
+		int j, k;
+
+		if (!q->setup_done)
+			fprintf(f, "Queue_%d is not configured\n", i);
+
+		fprintf(f, "Queue_%d\n", i);
+		fprintf(f, "========\n");
+
+		fprintf(f, "\tevqueue_%u is set up\n", q->id);
+
+		fprintf(f, "\tqueue is %s\n",
+			q->qm_queue.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
+
+		for (j = 0; j < dlb2->num_ports; j++) {
+			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
+
+			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++) {
+				if (p->link[k].valid &&
+				    p->link[k].queue_id == q->id)
+					fprintf(f, "id=%u prio=%u ",
+						p->id, p->link[k].priority);
+			}
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\tcurrent depth: %u events\n",
+			dlb2_get_queue_depth(dlb2, q));
+
+		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
+			q->qm_queue.num_qid_inflights, q->qm_queue.sched_type);
+	}
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 28fef3f..dc16c96 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files('dlb2.c',
 		'dlb2_iface.c',
+		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c'
-- 
2.6.4

[dpdk-dev] [PATCH v4 10/23] event/dlb2: add infos get and configure

[Timothy McDaniel]

Add support for configuring the DLB2 hardware.
In particular, this patch configures the DLB2
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  116 +
 drivers/event/dlb2/dlb2.c                  |  313 +++
 drivers/event/dlb2/dlb2_iface.c            |    5 +
 drivers/event/dlb2/dlb2_iface.h            |    4 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 3237 ++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |   15 +
 drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
 7 files changed, 3734 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index e3091c9..5f6c486 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -32,9 +32,125 @@ detailed understanding of the hardware, but these details are important when
 writing high-performance code. This section describes the places where the
 eventdev API and DLB2 misalign.
 
+Scheduling Domain Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are 32 scheduling domainis the DLB2.
+When one is configured, it allocates load-balanced and
+directed queues, ports, credits, and other hardware resources. Some
+resource allocations are user-controlled -- the number of queues, for example
+-- and others, like credit pools (one directed and one load-balanced pool per
+scheduling domain), are not.
+
+The DLB2 is a closed system eventdev, and as such the ``nb_events_limit`` device
+setup argument and the per-port ``new_event_threshold`` argument apply as
+defined in the eventdev header file. The limit is applied to all enqueues,
+regardless of whether it will consume a directed or load-balanced credit.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Reconfiguration
+~~~~~~~~~~~~~~~
+
+The Eventdev API allows one to reconfigure a device, its ports, and its queues
+by first stopping the device, calling the configuration function(s), then
+restarting the device. The DLB2 does not support configuring an individual queue
+or port without first reconfiguring the entire device, however, so there are
+certain reconfiguration sequences that are valid in the eventdev API but not
+supported by the PMD.
+
+Specifically, the PMD supports the following configuration sequence:
+1. Configure and start the device
+2. Stop the device
+3. (Optional) Reconfigure the device
+4. (Optional) If step 3 is run:
+
+   a. Setup queue(s). The reconfigured queue(s) lose their previous port links.
+   b. The reconfigured port(s) lose their previous queue links.
+
+5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s)
+6. Restart the device. If the device is reconfigured in step 3 but one or more
+   of its ports or queues are not, the PMD will apply their previous
+   configuration (including port->queue links) at this time.
+
+The PMD does not support the following configuration sequences:
+1. Configure and start the device
+2. Stop the device
+3. Setup queue or setup port
+4. Start the device
+
+This sequence is not supported because the event device must be reconfigured
+before its ports or queues can be.
+
+Atomic Inflights Allocation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the last stage prior to scheduling an atomic event to a CQ, DLB2 holds the
+inflight event in a temporary buffer that is divided among load-balanced
+queues. If a queue's atomic buffer storage fills up, this can result in
+head-of-line-blocking. For example:
+
+- An LDB queue allocated N atomic buffer entries
+- All N entries are filled with events from flow X, which is pinned to CQ 0.
+
+Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be
+scheduled. The likelihood of this case depends on the eventdev configuration,
+traffic behavior, event processing latency, potential for a worker to be
+interrupted or otherwise delayed, etc.
+
+By default, the PMD allocates 16 buffer entries for each load-balanced queue,
+which provides an even division across all 128 queues but potentially wastes
+buffer space (e.g. if not all queues are used, or aren't used for atomic
+scheduling).
+
+The PMD provides a dev arg to override the default per-queue allocation. To
+increase a vdev's per-queue atomic-inflight allocation to (for example) 64:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,atm_inflights=64
+
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 925824e..6798e66 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -85,6 +85,25 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
 	return 0;
 }
 
+static void
+dlb2_free_qe_mem(struct dlb2_port *qm_port)
+{
+	if (qm_port == NULL)
+		return;
+
+	rte_free(qm_port->qe4);
+	qm_port->qe4 = NULL;
+
+	rte_free(qm_port->int_arm_qe);
+	qm_port->int_arm_qe = NULL;
+
+	rte_free(qm_port->consume_qe);
+	qm_port->consume_qe = NULL;
+
+	rte_memzone_free(dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz);
+	dlb2_port[qm_port->id][PORT_TYPE(qm_port)].mz = NULL;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -350,10 +369,304 @@ set_qid_depth_thresh(const char *key __rte_unused,
 }
 
 static void
+dlb2_eventdev_info_get(struct rte_eventdev *dev,
+		       struct rte_event_dev_info *dev_info)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret;
+
+	ret = dlb2_hw_query_resources(dlb2);
+	if (ret) {
+		const struct rte_eventdev_data *data = dev->data;
+
+		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+			     ret, data->dev_id);
+		/* fn is void, so fall through and return values set up in
+		 * probe
+		 */
+	}
+
+	/* Add num resources currently owned by this domain.
+	 * These would become available if the scheduling domain were reset due
+	 * to the application recalling eventdev_configure to *reconfigure* the
+	 * domain.
+	 */
+	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
+	evdev_dlb2_default_info.max_event_queues += dlb2->num_ldb_queues;
+	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
+
+	evdev_dlb2_default_info.max_event_queues =
+		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
+			RTE_EVENT_MAX_QUEUES_PER_DEV);
+
+	evdev_dlb2_default_info.max_num_events =
+		RTE_MIN(evdev_dlb2_default_info.max_num_events,
+			dlb2->max_num_events_override);
+
+	*dev_info = evdev_dlb2_default_info;
+}
+
+static int
+dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
+			    const struct dlb2_hw_rsrcs *resources_asked)
+{
+	int ret = 0;
+	struct dlb2_create_sched_domain_args *cfg;
+
+	if (resources_asked == NULL) {
+		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
+		ret = EINVAL;
+		goto error_exit;
+	}
+
+	/* Map generic qm resources to dlb2 resources */
+	cfg = &handle->cfg.resources;
+
+	/* DIR ports and queues */
+
+	cfg->num_dir_ports = resources_asked->num_dir_ports;
+
+	cfg->num_dir_credits = resources_asked->num_dir_credits;
+
+	/* LDB queues */
+
+	cfg->num_ldb_queues = resources_asked->num_ldb_queues;
+
+	/* LDB ports */
+
+	cfg->cos_strict = 0; /* Best effort */
+	cfg->num_cos_ldb_ports[0] = 0;
+	cfg->num_cos_ldb_ports[1] = 0;
+	cfg->num_cos_ldb_ports[2] = 0;
+	cfg->num_cos_ldb_ports[3] = 0;
+
+	switch (handle->cos_id) {
+	case DLB2_COS_0:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[0] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_1:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[1] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_2:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[2] = resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_3:
+		cfg->num_ldb_ports = 0; /* no don't care ports */
+		cfg->num_cos_ldb_ports[3] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_DEFAULT:
+		/* all ldb ports are don't care ports from a cos perspective */
+		cfg->num_ldb_ports =
+			resources_asked->num_ldb_ports;
+		break;
+	}
+
+	cfg->num_ldb_credits =
+		resources_asked->num_ldb_credits;
+
+	cfg->num_atomic_inflights =
+		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
+		cfg->num_ldb_queues;
+
+	cfg->num_hist_list_entries = resources_asked->num_ldb_ports *
+		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d\n",
+		     cfg->num_ldb_queues,
+		     resources_asked->num_ldb_ports,
+		     cfg->num_dir_ports,
+		     cfg->num_atomic_inflights,
+		     cfg->num_hist_list_entries,
+		     cfg->num_ldb_credits,
+		     cfg->num_dir_credits);
+
+	/* Configure the QM */
+
+	ret = dlb2_iface_sched_domain_create(handle, cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: domain create failed, ret = %d, extra status: %s\n",
+			     ret,
+			     dlb2_error_strings[cfg->response.status]);
+
+		goto error_exit;
+	}
+
+	handle->domain_id = cfg->response.id;
+	handle->cfg.configured = true;
+
+error_exit:
+
+	return ret;
+}
+
+static void
+dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	enum dlb2_configuration_state config_state;
+	int i, j;
+
+	dlb2_iface_domain_reset(dlb2);
+
+	/* Free all dynamically allocated port memory */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
+
+	/* If reconfiguring, mark the device's queues and ports as "previously
+	 * configured." If the user doesn't reconfigure them, the PMD will
+	 * reapply their previous configuration when the device is started.
+	 */
+	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
+		DLB2_NOT_CONFIGURED;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		dlb2->ev_ports[i].qm_port.config_state = config_state;
+		/* Reset setup_done so ports can be reconfigured */
+		dlb2->ev_ports[i].setup_done = false;
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			dlb2->ev_ports[i].link[j].mapped = false;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++)
+		dlb2->ev_queues[i].qm_queue.config_state = config_state;
+
+	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
+		dlb2->ev_queues[i].setup_done = false;
+
+	dlb2->num_ports = 0;
+	dlb2->num_ldb_ports = 0;
+	dlb2->num_dir_ports = 0;
+	dlb2->num_queues = 0;
+	dlb2->num_ldb_queues = 0;
+	dlb2->num_dir_queues = 0;
+	dlb2->configured = false;
+}
+
+/* Note: 1 QM instance per QM device, QM instance/device == event device */
+static int
+dlb2_eventdev_configure(const struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
+	const struct rte_eventdev_data *data = dev->data;
+	const struct rte_event_dev_config *config = &data->dev_conf;
+	int ret;
+
+	/* If this eventdev is already configured, we must release the current
+	 * scheduling domain before attempting to configure a new one.
+	 */
+	if (dlb2->configured) {
+		dlb2_hw_reset_sched_domain(dev, true);
+
+		ret = dlb2_hw_query_resources(dlb2);
+		if (ret) {
+			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+				     ret, data->dev_id);
+			return ret;
+		}
+	}
+
+	if (config->nb_event_queues > rsrcs->num_queues) {
+		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_queues,
+			     rsrcs->num_queues);
+		return -EINVAL;
+	}
+	if (config->nb_event_ports > (rsrcs->num_ldb_ports
+			+ rsrcs->num_dir_ports)) {
+		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_ports,
+			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
+		return -EINVAL;
+	}
+	if (config->nb_events_limit > rsrcs->nb_events_limit) {
+		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_events_limit,
+			     rsrcs->nb_events_limit);
+		return -EINVAL;
+	}
+
+	if (config->event_dev_cfg & RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT)
+		dlb2->global_dequeue_wait = false;
+	else {
+		uint32_t timeout32;
+
+		dlb2->global_dequeue_wait = true;
+
+		/* note size mismatch of timeout vals in eventdev lib. */
+		timeout32 = config->dequeue_timeout_ns;
+
+		dlb2->global_dequeue_wait_ticks =
+			timeout32 * (rte_get_timer_hz() / 1E9);
+	}
+
+	/* Does this platform support umonitor/umwait? */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
+		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
+			DLB2_LOG_ERR("invalid value (%d) for RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
+				     RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
+			return -EINVAL;
+		}
+		dlb2->umwait_allowed = true;
+	}
+
+	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
+	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs->num_dir_ports;
+	/* 1 dir queue per dir port */
+	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports;
+
+	/* Scale down nb_events_limit by 4 for directed credits, since there
+	 * are 4x as many load-balanced credits.
+	 */
+	rsrcs->num_ldb_credits = 0;
+	rsrcs->num_dir_credits = 0;
+
+	if (rsrcs->num_ldb_queues)
+		rsrcs->num_ldb_credits = config->nb_events_limit;
+	if (rsrcs->num_dir_ports)
+		rsrcs->num_dir_credits = config->nb_events_limit / 4;
+	if (dlb2->num_dir_credits_override != -1)
+		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
+
+	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
+		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
+		return -ENODEV;
+	}
+
+	dlb2->new_event_limit = config->nb_events_limit;
+	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
+
+	/* Save number of ports/queues for this event dev */
+	dlb2->num_ports = config->nb_event_ports;
+	dlb2->num_queues = config->nb_event_queues;
+	dlb2->num_dir_ports = rsrcs->num_dir_ports;
+	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
+	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
+	dlb2->num_dir_queues = dlb2->num_dir_ports;
+	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
+	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
+	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
+	dlb2->max_dir_credits = rsrcs->num_dir_credits;
+
+	dlb2->configured = true;
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dev_infos_get    = dlb2_eventdev_info_get,
+		.dev_configure    = dlb2_eventdev_configure,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 0fc9991..a829b9b 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -26,3 +26,8 @@ int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
+
+int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				struct dlb2_create_sched_domain_args *args);
+
+void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4fb416e..6663dab 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -26,4 +26,8 @@ extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
 
+extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				 struct dlb2_create_sched_domain_args *args);
+
+extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 6de8b95..63a68a6 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -12,6 +12,27 @@
 #include "dlb2_regs.h"
 #include "dlb2_resource.h"
 
+#include "../../dlb2_priv.h"
+#include "../../dlb2_inline_fns.h"
+
+#define DLB2_DOM_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, domain_list, iter)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, func_list, iter)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, domain_list, it, it_tmp)
+
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, func_list, it, it_tmp)
+
 static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
 {
 	int i;
@@ -272,3 +293,3219 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
 
 	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
 }
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+	union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+	r0.field.count = domain->num_ldb_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+	r1.field.count = domain->num_dir_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+		       struct dlb2_function_resources *rsrcs,
+		       u32 domain_id,
+		       u32 cos_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	RTE_SET_USED(iter);
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+				   typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				   struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_ports,
+				   u32 cos_id,
+				   struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_ldb_port *port;
+
+		port = dlb2_get_next_ldb_port(hw, rsrcs,
+					      domain->id.phys_id, cos_id);
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_ports[cos_id],
+			      &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_ports[cos_id],
+			      &port->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_create_sched_domain_args *args,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      num,
+						      i,
+						      resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+					ret = __dlb2_attach_ldb_ports(hw,
+								      rsrcs,
+								      domain,
+								      1,
+								      cos_id,
+								      resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret < 0)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      1,
+						      j,
+						      resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 u32 num_ports,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+					   typeof(*port));
+		if (port == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_dir_pq_pairs, &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_dir_pq_pairs, &port->domain_list);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+					struct dlb2_hw_domain *domain,
+					u32 num_atomic_inflights,
+					struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+				     struct dlb2_hw_domain *domain,
+				     u32 num_hist_list_entries,
+				     struct dlb2_cmd_response *resp)
+{
+	struct dlb2_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb2_bitmap_find_set_bit_range(bitmap,
+						      num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+				  struct dlb2_function_resources *rsrcs,
+				  struct dlb2_hw_domain *domain,
+				  u32 num_queues,
+				  struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb2_ldb_queue *queue;
+
+		queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+					    typeof(*queue));
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_queues, &queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_queues, &queue->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+			     struct dlb2_function_resources *rsrcs,
+			     struct dlb2_hw_domain *domain,
+			     struct dlb2_create_sched_domain_args *args,
+			     struct dlb2_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb2_attach_ldb_queues(hw,
+				     rsrcs,
+				     domain,
+				     args->num_ldb_queues,
+				     resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_ports(hw,
+				    rsrcs,
+				    domain,
+				    args,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_ports(hw,
+				    rsrcs,
+				    domain,
+				    args->num_dir_ports,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_credits(rsrcs,
+				      domain,
+				      args->num_ldb_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_credits(rsrcs,
+				      domain,
+				      args->num_dir_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+						   domain,
+						   args->num_hist_list_entries,
+						   resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_atomic_inflights(rsrcs,
+					   domain,
+					   args->num_atomic_inflights,
+					   resp);
+	if (ret < 0)
+		return ret;
+
+	dlb2_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+				  struct dlb2_create_sched_domain_args *args,
+				  struct dlb2_cmd_response *resp)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb2_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+				  struct dlb2_create_sched_domain_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		    args->num_ldb_queues);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		    args->num_ldb_ports);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		    args->num_cos_ldb_ports[0]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		    args->cos_strict);
+	DLB2_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		    args->num_dir_ports);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		    args->num_atomic_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		    args->num_hist_list_entries);
+	DLB2_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		    args->num_ldb_credits);
+	DLB2_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		    args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ *	domain and its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+	if (ret)
+		return ret;
+
+	domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available domains\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (domain->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_domains contains configured domains.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	dlb2_init_domain_rsrc_lists(domain);
+
+	ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to verify args.\n",
+			    __func__);
+
+		return ret;
+	}
+
+	dlb2_list_del(&rsrcs->avail_domains, &domain->func_list);
+
+	dlb2_list_add(&rsrcs->used_domains, &domain->func_list);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+	DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+			     struct dlb2_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb2_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	int ret;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_disable(hw, port);
+
+		ret = dlb2_drain_dir_cq(hw, port);
+		if (ret < 0)
+			return ret;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_dir_pq_pair *queue)
+{
+	union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *queue)
+{
+	return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+				      struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		dlb2_movdir64b(pp_addr, hcw);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			dlb2_movdir64b(pp_addr, hcw);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int ret, i;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			if (toggle_port)
+				dlb2_ldb_port_cq_disable(hw, port);
+
+			ret = dlb2_drain_ldb_cq(hw, port);
+			if (ret < 0)
+				return ret;
+
+			if (toggle_port)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+
+	return 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_atm_active r1;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_ldb_queue *queue)
+{
+	return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb2_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+					   struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to unmap domain queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = true;
+
+			dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+			   u32 id,
+			   bool vdev_req,
+			   unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iter1) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter2)
+			if (queue->id.virt_id == id)
+				return queue;
+	}
+
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue, iter1)
+		if (queue->id.virt_id == id)
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+						      u32 id,
+						      bool vdev_req,
+						      unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iteration;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	RTE_SET_USED(iteration);
+
+	if (id >= DLB2_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iteration)
+		if (domain->id.virt_id == id)
+			return domain;
+
+	return NULL;
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot,
+					   enum dlb2_qid_map_state new_state)
+{
+	enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb2_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB2_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB2_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB2_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB2_HW_DBG(hw,
+		    "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB2_HW_ERR(hw,
+		    "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+				enum dlb2_qid_map_state state,
+				int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+				      enum dlb2_qid_map_state state,
+				      struct dlb2_ldb_queue *queue,
+				      int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+					      struct dlb2_hw_domain *domain,
+					      struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain,
+					     struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+						struct dlb2_ldb_port *port,
+						int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 0;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 1;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+					struct dlb2_ldb_port *p,
+					struct dlb2_ldb_queue *q,
+					u8 priority)
+{
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_lsp_cq2qid0 r1;
+	union dlb2_atm_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix_00 r3;
+	union dlb2_lsp_qid2cqidix2_00 r4;
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+	    !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+	r0.field.v |= 1 << i;
+	r0.field.prio |= (priority & 0x7) << i * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+	else
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		r1.field.qid_p0 = q->id.phys_id;
+	if (i == 1 || i == 5)
+		r1.field.qid_p1 = q->id.phys_id;
+	if (i == 2 || i == 6)
+		r1.field.qid_p2 = q->id.phys_id;
+	if (i == 3 || i == 7)
+		r1.field.qid_p3 = q->id.phys_id;
+
+	if (i < 4)
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+	else
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r4.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+						  p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		r2.field.cq_p0 |= 1 << i;
+		r3.field.cq_p0 |= 1 << i;
+		r4.field.cq_p0 |= 1 << i;
+		break;
+
+	case 1:
+		r2.field.cq_p1 |= 1 << i;
+		r3.field.cq_p1 |= 1 << i;
+		r4.field.cq_p1 |= 1 << i;
+		break;
+
+	case 2:
+		r2.field.cq_p2 |= 1 << i;
+		r3.field.cq_p2 |= 1 << i;
+		r4.field.cq_p2 |= 1 << i;
+		break;
+
+	case 3:
+		r2.field.cq_p3 |= 1 << i;
+		r3.field.cq_p3 |= 1 << i;
+		r4.field.cq_p3 |= 1 << i;
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r3.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		    r4.val);
+
+	dlb2_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB2_QUEUE_MAPPED;
+
+	return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	/* Set the atomic scheduling haswork bit */
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.rlist_haswork_v = r0.field.count > 0;
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      u8 slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.rlist_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.nalb_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+					      struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+	r0.field.limit = queue->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+						struct dlb2_ldb_queue *queue)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+						struct dlb2_hw_domain *domain,
+						struct dlb2_ldb_port *port,
+						struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_lsp_qid_ldb_infl_cnt r0;
+	enum dlb2_qid_map_state state;
+	int slot, ret, i;
+	u8 prio;
+	RTE_SET_USED(iter);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: non-zero QID inflight count\n",
+			    __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			state = DLB2_QUEUE_MAPPED;
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+					 struct dlb2_ldb_port *port,
+					 struct dlb2_ldb_queue *queue,
+					 u8 priority)
+{
+	union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	int domain_id, slot, ret;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+		DLB2_HW_ERR(hw,
+			    "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		if (port->enabled)
+			dlb2_ldb_port_cq_enable(hw, port);
+
+		dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain,
+					struct dlb2_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		union dlb2_lsp_qid_ldb_infl_cnt r0;
+		struct dlb2_ldb_queue *queue;
+		int qid;
+
+		if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (queue == NULL) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: unable to find queue %d\n",
+				    __func__, qid);
+			continue;
+		}
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count)
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb2_ldb_port_cq_disable(hw, port);
+
+		dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count) {
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+
+			dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port,
+				   struct dlb2_ldb_queue *queue)
+{
+	enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_atm_qid2cqidix_00 r1;
+	union dlb2_lsp_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix2_00 r3;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB2_QUEUE_MAPPED;
+	in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: QID %d isn't mapped\n",
+			    __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+	r0.field.v &= ~(1 << i);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		r1.field.cq_p0 &= ~(1 << i);
+		r2.field.cq_p0 &= ~(1 << i);
+		r3.field.cq_p0 &= ~(1 << i);
+		break;
+
+	case 1:
+		r1.field.cq_p1 &= ~(1 << i);
+		r2.field.cq_p1 &= ~(1 << i);
+		r3.field.cq_p1 &= ~(1 << i);
+		break;
+
+	case 2:
+		r1.field.cq_p2 &= ~(1 << i);
+		r2.field.cq_p2 &= ~(1 << i);
+		r3.field.cq_p2 &= ~(1 << i);
+		break;
+
+	case 3:
+		r1.field.cq_p3 &= ~(1 << i);
+		r2.field.cq_p3 &= ~(1 << i);
+		r3.field.cq_p3 &= ~(1 << i);
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+		    r1.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+		    r3.val);
+
+	dlb2_flush_csr(hw);
+
+	unmapped = DLB2_QUEUE_UNMAPPED;
+
+	return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_ldb_port *port,
+				 struct dlb2_ldb_queue *queue,
+				 u8 prio)
+{
+	if (domain->started)
+		return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   int slot)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb2_ldb_port_qid_map *map;
+		struct dlb2_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain,
+					  struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (r0.field.count > 0)
+		return false;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = false;
+
+			dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_dir_vpp_v r1;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		unsigned int offs;
+		u32 virt_id;
+
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+	}
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_ldb_vpp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			unsigned int offs;
+			u32 virt_id;
+
+			if (hw->virt_mode == DLB2_VIRT_SRIOV)
+				virt_id = port->id.virt_id;
+			else
+				virt_id = port->id.phys_id;
+
+			offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+			DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+	union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+				    r0.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+				    r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+	union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+			    r0.val);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+			    r1.val);
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+		union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+		union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+				    r1.val);
+
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID_V(idx),
+				    r2.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID2QID(idx),
+				    r3.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID_V(idx),
+				    r1.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID2QID(idx),
+				    r2.val);
+		}
+	}
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+					       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_sn_chk_enbl r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.en = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			int i;
+
+			for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+				if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					    __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		port->enabled = false;
+
+		dlb2_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	union dlb2_sys_dir_pp_v r1;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+			    r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_ldb_pp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_ldb_queue *queue;
+	int i;
+	RTE_SET_USED(iter);
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty ldb queue %d\n",
+				    __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port, iter) {
+			if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb2_ldb_cq_token_count(hw, ldb_port)) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to empty ldb port %d\n",
+					    __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir queue %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb2_dir_cq_token_count(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir port %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						   struct dlb2_ldb_port *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP2PP(offs),
+			    DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP_V(offs),
+			    DLB2_SYS_VF_LDB_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    DLB2_SYS_LDB_PP_V_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_DSBL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_BASE_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID0(port->id.phys_id),
+		    DLB2_LSP_CQ2QID0_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID1(port->id.phys_id),
+		    DLB2_LSP_CQ2QID1_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+		    DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			__dlb2_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+				       struct dlb2_dir_pq_pair *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_DSBL_RST);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_FMT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP2PP(offs),
+			    DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP_V(offs),
+			    DLB2_SYS_VF_DIR_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		__dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+			    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_ITS(queue_id),
+			    DLB2_SYS_LDB_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN(queue_id),
+			    DLB2_CHP_ORD_QID_SN_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+			    DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_V(queue_id),
+			    DLB2_SYS_LDB_QID_V_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_CFG_V(queue_id),
+			    DLB2_SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB2_CSR_WR(hw,
+				    offs[queue->sn_group],
+				    DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX2(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX2_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_ATM_QID2CQIDIX(queue_id, i),
+				    DLB2_ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb2_domain_reset_dir_port_registers(hw, domain);
+
+	dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_dir_pq_pair *tmp_dir_port;
+	struct dlb2_ldb_queue *tmp_ldb_queue;
+	struct dlb2_ldb_port *tmp_ldb_port;
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_list_head *list;
+	int ret, i;
+	RTE_SET_USED(tmp_dir_port);
+	RTE_SET_USED(tmp_ldb_queue);
+	RTE_SET_USED(tmp_ldb_port);
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list = &domain->used_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb2_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		dlb2_list_del(&domain->used_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list = &domain->avail_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		ldb_queue->owned = false;
+
+		dlb2_list_del(&domain->avail_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		list = &domain->used_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB2_QUEUE_UNMAPPED;
+
+			dlb2_list_del(&domain->used_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list = &domain->avail_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			ldb_port->owned = false;
+
+			dlb2_list_del(&domain->avail_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list = &domain->used_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		dlb2_list_del(&domain->used_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list = &domain->avail_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				    domain->hist_list_entry_base,
+				    domain->total_hist_list_entries);
+	if (ret) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			    __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	dlb2_list_del(&rsrcs->used_domains, &domain->func_list);
+	dlb2_list_add(&rsrcs->avail_domains, &domain->func_list);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain,
+					    struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_ldb_port *port;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (!dlb2_list_empty(&domain->used_ldb_ports[i]))
+			break;
+	}
+
+	if (i == DLB2_NUM_COS_DOMAINS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: No configured LDB ports\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb2_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+	 * prior to this in dlb2_domain_drain_mapped_queues().
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings != 0 ||
+		    dlb2_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ *	hardware resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain  == NULL || !domain->configured)
+		return -EINVAL;
+
+	/* Disable VPPs */
+	if (vdev_req) {
+		dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+		dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+	}
+
+	/* Disable CQ interrupts */
+	dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+	dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+	dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb2_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_mapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	dlb2_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb2_domain_disable_dir_cqs(hw, domain);
+
+	/* Disable PPs */
+	dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+	ret = dlb2_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Reset the QID and port state. */
+	dlb2_domain_reset_registers(hw, domain);
+
+	/* Hardware reset complete. Reset the domain's software state */
+	ret = dlb2_domain_reset_software_state(hw, domain);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 210b3186..285ad2c 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -598,3 +598,18 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
 
 	return 0;
 }
+
+int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+			    struct dlb2_create_sched_domain_args *args,
+			    struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_sched_domain(hw, args, resp, NOT_VF_REQ,
+					   PF_ID_ZERO);
+}
+
+int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 7d64309..9310f72 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -108,15 +108,59 @@ dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
 	return 0;
 }
 
+static int
+dlb2_pf_sched_domain_create(struct dlb2_hw_dev *handle,
+			    struct dlb2_create_sched_domain_args *arg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (dlb2_dev->domain_reset_failed) {
+		response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = dlb2_pf_create_sched_domain(&dlb2_dev->hw, arg, &response);
+	if (ret)
+		goto done;
+
+done:
+
+	arg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static void
+dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret;
+
+	dlb2_dev = (struct dlb2_dev *)dlb2->qm_instance.pf_dev;
+	ret = dlb2_pf_reset_domain(&dlb2_dev->hw, dlb2->qm_instance.domain_id);
+	if (ret)
+		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
 	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
 	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_domain_reset = dlb2_pf_domain_reset;
 	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
 	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v4 11/23] event/dlb2: add queue and port default conf

[Timothy McDaniel]

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 6798e66..ef1c000 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -661,12 +661,42 @@ dlb2_eventdev_configure(const struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
+				    uint8_t port_id,
+				    struct rte_event_port_conf *port_conf)
+{
+	RTE_SET_USED(port_id);
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	port_conf->new_event_threshold = dlb2->new_event_limit;
+	port_conf->dequeue_depth = 32;
+	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
+	port_conf->event_port_cfg = 0;
+}
+
+static void
+dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
+				     uint8_t queue_id,
+				     struct rte_event_queue_conf *queue_conf)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(queue_id);
+
+	queue_conf->nb_atomic_flows = 1024;
+	queue_conf->nb_atomic_order_sequences = 64;
+	queue_conf->event_queue_cfg = 0;
+	queue_conf->priority = 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH v4 12/23] event/dlb2: add queue setup

[Timothy McDaniel]

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  73 +++--
 drivers/event/dlb2/dlb2.c                  | 312 +++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |  12 +
 drivers/event/dlb2/dlb2_iface.h            |  12 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 464 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  82 +++++
 7 files changed, 926 insertions(+), 39 deletions(-)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 5f6c486..e57e009 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -47,45 +47,40 @@ setup argument and the per-port ``new_event_threshold`` argument apply as
 defined in the eventdev header file. The limit is applied to all enqueues,
 regardless of whether it will consume a directed or load-balanced credit.
 
-Load-balanced and Directed Ports
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
-does not have the same concept, but it has a similar one: ports and queues that
-are singly-linked (i.e. linked to a single queue or port, respectively).
-
-The ``rte_event_dev_info_get()`` function reports the number of available
-event ports and queues (among other things). For the DLB2 PMD, max_event_ports
-and max_event_queues report the number of available load-balanced ports and
-queues, and max_single_link_event_port_queue_pairs reports the number of
-available directed ports and queues.
-
-When a scheduling domain is created in ``rte_event_dev_configure()``, the user
-specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
-control the total number of ports (load-balanced and directed) and the number
-of directed ports. Hence, the number of requested load-balanced ports is
-``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
-specifies the total number of queues (load-balanced and directed). The number
-of directed queues comes from ``nb_single_link_event_port_queues``, since
-directed ports and queues come in pairs.
-
-When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
-whether it should be configured as a directed (the flag is set) or a
-load-balanced (the flag is unset) port. Similarly, the
-``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
-whether it is a directed or load-balanced queue.
-
-Load-balanced ports can only be linked to load-balanced queues, and directed
-ports can only be linked to directed queues. Furthermore, directed ports can
-only be linked to a single directed queue (and vice versa), and that link
-cannot change after the eventdev is started.
-
-The eventdev API does not have a directed scheduling type. To support directed
-traffic, the dlb PMD detects when an event is being sent to a directed queue
-and overrides its scheduling type. Note that the originally selected scheduling
-type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
-will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
-port.
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
 
 Flow ID
 ~~~~~~~
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index ef1c000..8c1e06d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -688,6 +688,317 @@ dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
 	queue_conf->priority = 0;
 }
 
+static int32_t
+dlb2_get_sn_allocation(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_allocation_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_set_sn_allocation(struct dlb2_eventdev *dlb2, int group, int num)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_set_sn_allocation_args cfg;
+	int ret;
+
+	cfg.num = num;
+	cfg.group = group;
+
+	ret = dlb2_iface_set_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: set_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int32_t
+dlb2_get_sn_occupancy(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_occupancy_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_occupancy(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_occupancy ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+/* Query the current sequence number allocations and, if they conflict with the
+ * requested LDB queue configuration, attempt to re-allocate sequence numbers.
+ * This is best-effort; if it fails, the PMD will attempt to configure the
+ * load-balanced queue and return an error.
+ */
+static void
+dlb2_program_sn_allocation(struct dlb2_eventdev *dlb2,
+			   const struct rte_event_queue_conf *queue_conf)
+{
+	int grp_occupancy[DLB2_NUM_SN_GROUPS];
+	int grp_alloc[DLB2_NUM_SN_GROUPS];
+	int i, sequence_numbers;
+
+	sequence_numbers = (int)queue_conf->nb_atomic_order_sequences;
+
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		int total_slots;
+
+		grp_alloc[i] = dlb2_get_sn_allocation(dlb2, i);
+		if (grp_alloc[i] < 0)
+			return;
+
+		total_slots = DLB2_MAX_LDB_SN_ALLOC / grp_alloc[i];
+
+		grp_occupancy[i] = dlb2_get_sn_occupancy(dlb2, i);
+		if (grp_occupancy[i] < 0)
+			return;
+
+		/* DLB has at least one available slot for the requested
+		 * sequence numbers, so no further configuration required.
+		 */
+		if (grp_alloc[i] == sequence_numbers &&
+		    grp_occupancy[i] < total_slots)
+			return;
+	}
+
+	/* None of the sequence number groups are configured for the requested
+	 * sequence numbers, so we have to reconfigure one of them. This is
+	 * only possible if a group is not in use.
+	 */
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		if (grp_occupancy[i] == 0)
+			break;
+	}
+
+	if (i == DLB2_NUM_SN_GROUPS) {
+		printf("[%s()] No groups with %d sequence_numbers are available or have free slots\n",
+		       __func__, sequence_numbers);
+		return;
+	}
+
+	/* Attempt to configure slot i with the requested number of sequence
+	 * numbers. Ignore the return value -- if this fails, the error will be
+	 * caught during subsequent queue configuration.
+	 */
+	dlb2_set_sn_allocation(dlb2, i, sequence_numbers);
+}
+
+static int32_t
+dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 const struct rte_event_queue_conf *evq_conf)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_queue *queue = &ev_queue->qm_queue;
+	struct dlb2_create_ldb_queue_args cfg;
+	int32_t ret;
+	uint32_t qm_qid;
+	int sched_type = -1;
+
+	if (evq_conf == NULL)
+		return -EINVAL;
+
+	if (evq_conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_ALL_TYPES) {
+		if (evq_conf->nb_atomic_order_sequences != 0)
+			sched_type = RTE_SCHED_TYPE_ORDERED;
+		else
+			sched_type = RTE_SCHED_TYPE_PARALLEL;
+	} else
+		sched_type = evq_conf->schedule_type;
+
+	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
+	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
+	cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences;
+
+	if (sched_type != RTE_SCHED_TYPE_ORDERED) {
+		cfg.num_sequence_numbers = 0;
+		cfg.num_qid_inflights = 2048;
+	}
+
+	/* App should set this to the number of hardware flows they want, not
+	 * the overall number of flows they're going to use. E.g. if app is
+	 * using 64 flows and sets compression to 64, best-case they'll get
+	 * 64 unique hashed flows in hardware.
+	 */
+	switch (evq_conf->nb_atomic_flows) {
+	/* Valid DLB2 compression levels */
+	case 64:
+	case 128:
+	case 256:
+	case 512:
+	case (1 * 1024): /* 1K */
+	case (2 * 1024): /* 2K */
+	case (4 * 1024): /* 4K */
+	case (64 * 1024): /* 64K */
+		cfg.lock_id_comp_level = evq_conf->nb_atomic_flows;
+		break;
+	default:
+		/* Invalid compression level */
+		cfg.lock_id_comp_level = 0; /* no compression */
+	}
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create LB event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	qm_qid = cfg.response.id;
+
+	/* Save off queue config for debug, resource lookups, and reconfig */
+	queue->num_qid_inflights = cfg.num_qid_inflights;
+	queue->num_atm_inflights = cfg.num_atomic_inflights;
+
+	queue->sched_type = sched_type;
+	queue->config_state = DLB2_CONFIGURED;
+
+	DLB2_LOG_DBG("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n",
+		     qm_qid,
+		     cfg.num_atomic_inflights,
+		     cfg.num_sequence_numbers,
+		     cfg.num_qid_inflights);
+
+	return qm_qid;
+}
+
+static int
+dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int32_t qm_qid;
+
+	if (queue_conf->nb_atomic_order_sequences)
+		dlb2_program_sn_allocation(dlb2, queue_conf);
+
+	qm_qid = dlb2_hw_create_ldb_queue(dlb2, ev_queue, queue_conf);
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the load-balanced queue\n");
+
+		return qm_qid;
+	}
+
+	dlb2->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int dlb2_num_dir_queues_setup(struct dlb2_eventdev *dlb2)
+{
+	int i, num = 0;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].setup_done &&
+		    dlb2->ev_queues[i].qm_queue.is_directed)
+			num++;
+	}
+
+	return num;
+}
+
+static void
+dlb2_queue_link_teardown(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue)
+{
+	struct dlb2_eventdev_port *ev_port;
+	int i, j;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (!ev_port->link[j].valid ||
+			    ev_port->link[j].queue_id != ev_queue->id)
+				continue;
+
+			ev_port->link[j].valid = false;
+			ev_port->num_links--;
+		}
+	}
+
+	ev_queue->num_links = 0;
+}
+
+static int
+dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
+			  uint8_t ev_qid,
+			  const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_queue *ev_queue;
+	int ret;
+
+	if (queue_conf == NULL)
+		return -EINVAL;
+
+	if (ev_qid >= dlb2->num_queues)
+		return -EINVAL;
+
+	ev_queue = &dlb2->ev_queues[ev_qid];
+
+	ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg &
+		RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+	ev_queue->id = ev_qid;
+	ev_queue->conf = *queue_conf;
+
+	if (!ev_queue->qm_queue.is_directed) {
+		ret = dlb2_eventdev_ldb_queue_setup(dev, ev_queue, queue_conf);
+	} else {
+		/* The directed queue isn't setup until link time, at which
+		 * point we know its directed port ID. Directed queue setup
+		 * will only fail if this queue is already setup or there are
+		 * no directed queues left to configure.
+		 */
+		ret = 0;
+
+		ev_queue->qm_queue.config_state = DLB2_NOT_CONFIGURED;
+
+		if (ev_queue->setup_done ||
+		    dlb2_num_dir_queues_setup(dlb2) == dlb2->num_dir_queues)
+			ret = -EINVAL;
+	}
+
+	/* Tear down pre-existing port->queue links */
+	if (!ret && dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_queue_link_teardown(dlb2, ev_queue);
+
+	if (!ret)
+		ev_queue->setup_done = true;
+
+	return ret;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -696,6 +1007,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a829b9b..4c07574 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -31,3 +31,15 @@ int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				struct dlb2_create_sched_domain_args *args);
 
 void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_ldb_queue_args *cfg);
+
+int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_sn_allocation_args *args);
+
+int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_set_sn_allocation_args *args);
+
+int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_occupancy_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 6663dab..4c88fe0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -30,4 +30,16 @@ extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				 struct dlb2_create_sched_domain_args *args);
 
 extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+extern int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_queue_args *cfg);
+
+extern int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_allocation_args *args);
+
+extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_set_sn_allocation_args *args);
+
+extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				  struct dlb2_get_sn_occupancy_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 63a68a6..e6ea0d7 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3509,3 +3509,467 @@ unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
 
 	return num;
 }
+
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_ldb_queue *queue,
+				     struct dlb2_create_ldb_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+	union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+	union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+	union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+	union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+	union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+	union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+	union dlb2_sys_ldb_qid_its r7 = { {0} };
+	union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+	union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+	union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+	union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+	union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+	union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+	struct dlb2_sn_group *sn_group;
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r3.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	r4.field.limit = args->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+	r5.field.limit = queue->aqed_limit;
+
+	if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+		r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+		    r5.val);
+
+	switch (args->lock_id_comp_level) {
+	case 64:
+		r6.field.compress_code = 1;
+		break;
+	case 128:
+		r6.field.compress_code = 2;
+		break;
+	case 256:
+		r6.field.compress_code = 3;
+		break;
+	case 512:
+		r6.field.compress_code = 4;
+		break;
+	case 1024:
+		r6.field.compress_code = 5;
+		break;
+	case 2048:
+		r6.field.compress_code = 6;
+		break;
+	case 4096:
+		r6.field.compress_code = 7;
+		break;
+	case 0:
+	case 65536:
+		r6.field.compress_code = 0;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+		    r6.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r7.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+		    r7.val);
+
+	r8.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+		    r8.val);
+
+	r9.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+		    r9.val);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	r10.field.qid_fid_limit = 512;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+		    r10.val);
+
+	/* Configure SNs */
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	r11.field.mode = sn_group->mode;
+	r11.field.slot = queue->sn_slot;
+	r11.field.grp  = sn_group->id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+	r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+	r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		r0.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+		r1.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+		r2.field.vqid = queue->id.virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+			    r2.val);
+	}
+
+	r13.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+				  struct dlb2_ldb_queue *queue,
+				  struct dlb2_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb2_sn_group_full(group)) {
+			slot = dlb2_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no sequence number slots available\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+				struct dlb2_hw_domain *domain,
+				struct dlb2_ldb_queue *queue,
+				struct dlb2_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_ldb_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (dlb2_list_empty(&domain->avail_ldb_queues)) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb2_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_ldb_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		    args->num_sequence_numbers);
+	DLB2_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		    args->num_qid_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		    args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+
+	dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			    __func__, __LINE__);
+		return ret;
+	}
+
+	dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_queues, &queue->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_queues, &queue->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+						unsigned int group_id,
+						unsigned long val)
+{
+	DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+	DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+	DLB2_HW_DBG(hw, "\tValue:    %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val)
+{
+	u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+	union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+	struct dlb2_sn_group *group;
+	int mode;
+
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	group = &hw->rsrcs.sn_groups[group_id];
+
+	/*
+	 * Once the first load-balanced queue using an SN group is configured,
+	 * the group cannot be changed.
+	 */
+	if (group->slot_use_bitmap != 0)
+		return -EPERM;
+
+	for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+		if (val == valid_allocations[mode])
+			break;
+
+	if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+		return -EINVAL;
+
+	group->mode = mode;
+	group->sequence_numbers_per_queue = val;
+
+	r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+	r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+	DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+	dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 285ad2c..14bb04b 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -613,3 +613,13 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
 {
 	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_ldb_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 9310f72..a6cd178 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -150,6 +150,84 @@ dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
 		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
 }
 
+static int
+dlb2_pf_ldb_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_ldb_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_ldb_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw_dev *handle,
+			 struct dlb2_get_sn_occupancy_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_number_occupancy(&dlb2_dev->hw,
+						       args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_numbers(&dlb2_dev->hw, args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_set_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_set_group_sequence_numbers(&dlb2_dev->hw, args->group,
+					      args->num);
+
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -161,6 +239,10 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
+	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
+	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
+	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

[dpdk-dev] [PATCH v4 13/23] event/dlb2: add port setup

[Timothy McDaniel]

Configure the load balanded (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 doc/guides/eventdevs/dlb2.rst              |  75 +++
 drivers/event/dlb2/dlb2.c                  | 498 ++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   8 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 922 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  28 +
 drivers/event/dlb2/pf/dlb2_pf.c            | 180 ++++++
 7 files changed, 1719 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index e57e009..8027af7 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -82,6 +82,81 @@ The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
 the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
 load-balanced queues can use the full 16-bit flow ID range.
 
+Load-Balanced Queues
+~~~~~~~~~~~~~~~~~~~
+
+A load-balanced queue can support atomic and ordered scheduling, or atomic and
+unordered scheduling, but not atomic and unordered and ordered scheduling. A
+queue's scheduling types are controlled by the event queue configuration.
+
+If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
+``nb_atomic_order_sequences`` determines the supported scheduling types.
+With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
+and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
+supported by scheduling those events as ordered events.  Note that when the
+event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
+``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
+unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
+
+If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
+dictates the queue's scheduling type.
+
+The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
+queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
+group is configured to contain either 1 queue with 1024 reorder entries, 2
+queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
+
+When a load-balanced queue is created, the PMD will configure a new sequence
+number group on-demand if num_sequence_numbers does not match a pre-existing
+group with available reorder buffer entries. If all sequence number groups are
+in use, no new group will be created and queue configuration will fail. (Note
+that when the PMD is used with a virtual DLB2 device, it cannot change the
+sequence number configuration.)
+
+The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
+the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
+load-balanced queues can use the full 16-bit flow ID range.
+
+Load-balanced and Directed Ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DLB2 ports come in two flavors: load-balanced and directed. The eventdev API
+does not have the same concept, but it has a similar one: ports and queues that
+are singly-linked (i.e. linked to a single queue or port, respectively).
+
+The ``rte_event_dev_info_get()`` function reports the number of available
+event ports and queues (among other things). For the DLB2 PMD, max_event_ports
+and max_event_queues report the number of available load-balanced ports and
+queues, and max_single_link_event_port_queue_pairs reports the number of
+available directed ports and queues.
+
+When a scheduling domain is created in ``rte_event_dev_configure()``, the user
+specifies ``nb_event_ports`` and ``nb_single_link_event_port_queues``, which
+control the total number of ports (load-balanced and directed) and the number
+of directed ports. Hence, the number of requested load-balanced ports is
+``nb_event_ports - nb_single_link_event_ports``. The ``nb_event_queues`` field
+specifies the total number of queues (load-balanced and directed). The number
+of directed queues comes from ``nb_single_link_event_port_queues``, since
+directed ports and queues come in pairs.
+
+When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines
+whether it should be configured as a directed (the flag is set) or a
+load-balanced (the flag is unset) port. Similarly, the
+``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls
+whether it is a directed or load-balanced queue.
+
+Load-balanced ports can only be linked to load-balanced queues, and directed
+ports can only be linked to directed queues. Furthermore, directed ports can
+only be linked to a single directed queue (and vice versa), and that link
+cannot change after the eventdev is started.
+
+The eventdev API does not have a directed scheduling type. To support directed
+traffic, the dlb PMD detects when an event is being sent to a directed queue
+and overrides its scheduling type. Note that the originally selected scheduling
+type (atomic, ordered, or parallel) is not preserved, and an event's sched_type
+will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed
+port.
+
 Flow ID
 ~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 8c1e06d..9e8dcce 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -999,6 +999,503 @@ dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
 	return ret;
 }
 
+static int
+dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_cq_pop_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_cq_pop_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL)	{
+		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->consume_qe = qe;
+
+	qe->qe_valid = 0;
+	qe->qe_frag = 0;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
+	 * and so on.
+	 */
+	qe->tokens = 0;	/* set at run time */
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_enqueue_qe *qe;
+
+	qe = rte_zmalloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_enqueue_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL) {
+		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->int_arm_qe = qe;
+
+	/* V2 - INT ARM is CQ_TOKEN + FRAG */
+	qe->qe_valid = 0;
+	qe->qe_frag = 1;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
+{
+	int ret, sz;
+
+	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct dlb2_enqueue_qe);
+
+	qm_port->qe4 = rte_zmalloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
+
+	if (qm_port->qe4 == NULL) {
+		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	ret = dlb2_init_consume_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	return 0;
+
+error_exit:
+
+	dlb2_free_qe_mem(qm_port);
+
+	return ret;
+}
+
+static int
+dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_ldb_port_args cfg = {0};
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_CQ_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	cfg.cq_history_list_size = DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	if (handle->cos_id == DLB2_COS_DEFAULT)
+		cfg.cos_id = 0;
+	else
+		cfg.cos_id = handle->cos_id;
+
+	cfg.cos_strict = 0;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* If there are no directed ports, the kernel driver will ignore this
+	 * port's directed credit settings. Don't use enqueue_depth if it would
+	 * require more directed credits than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2; /* back ptr */
+	/*
+	 * Allocate and init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_ldb_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(qm_port->cfg.ldb));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     qm_port->ldb_credits,
+		     qm_port->dir_credits);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
+
+	return ret;
+}
+
+static void
+dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port)
+{
+	struct dlb2_eventdev_queue *ev_queue;
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (!ev_port->link[i].valid)
+			continue;
+
+		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
+
+		ev_port->link[i].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+}
+
+static int
+dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_port_args cfg = {0};
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (dlb2 == NULL || handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* Directed queues are configured at link time. */
+	cfg.queue_id = -1;
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* Don't use enqueue_depth if it would require more directed credits
+	 * than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2;  /* back ptr */
+
+	/*
+	 * Init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE to be aligned
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "dlb2_dir_port%d",
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = cfg.cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(qm_port->cfg.dir));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     dir_credit_high_watermark,
+		     ldb_credit_high_watermark);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_setup(struct rte_eventdev *dev,
+			 uint8_t ev_port_id,
+			 const struct rte_event_port_conf *port_conf)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_eventdev_port *ev_port;
+	int ret;
+
+	if (dev == NULL || port_conf == NULL) {
+		DLB2_LOG_ERR("Null parameter\n");
+		return -EINVAL;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
+		return -EINVAL;
+
+	if (port_conf->dequeue_depth >
+		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
+	    port_conf->enqueue_depth >
+		evdev_dlb2_default_info.max_event_port_enqueue_depth)
+		return -EINVAL;
+
+	ev_port = &dlb2->ev_ports[ev_port_id];
+	/* configured? */
+	if (ev_port->setup_done) {
+		DLB2_LOG_ERR("evport %d is already configured\n", ev_port_id);
+		return -EINVAL;
+	}
+
+	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
+		RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (!ev_port->qm_port.is_directed) {
+		ret = dlb2_hw_create_ldb_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the lB port ve portId=%d\n",
+				     ev_port_id);
+
+			return ret;
+		}
+	} else {
+		ret = dlb2_hw_create_dir_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the DIR port\n");
+			return ret;
+		}
+	}
+
+	/* Save off port config for reconfig */
+	ev_port->conf = *port_conf;
+
+	ev_port->id = ev_port_id;
+	ev_port->enq_configured = true;
+	ev_port->setup_done = true;
+	ev_port->inflight_max = port_conf->new_event_threshold;
+	ev_port->implicit_release = !(port_conf->event_port_cfg &
+		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
+	ev_port->outstanding_releases = 0;
+	ev_port->inflight_credits = 0;
+	ev_port->credit_update_quanta = RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
+	ev_port->dlb2 = dlb2; /* reverse link */
+
+	/* Tear down pre-existing port->queue links */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
+
+	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1009,6 +1506,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
+		.port_setup       = dlb2_eventdev_port_setup,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 4c07574..01818c7 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -43,3 +43,11 @@ int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				   struct dlb2_get_sn_occupancy_args *args);
+
+int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_dir_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4c88fe0..a6b923a 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -42,4 +42,12 @@ extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				  struct dlb2_get_sn_occupancy_args *args);
+
+extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_ldb_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_dir_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index e6ea0d7..a633d3e 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3973,3 +3973,925 @@ int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_ldb_port *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_pp2vas r0 = { {0} };
+	union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+		union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_ldb_port *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_ldb_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+	union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+	union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+	union dlb2_chp_hist_list_lim r5 = { {0} };
+	union dlb2_chp_hist_list_base r6 = { {0} };
+	union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+	union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+	union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+	union dlb2_sys_ldb_cq_at r10 = { {0} };
+	union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+	union dlb2_chp_ldb_cq2vas r12 = { {0} };
+	union dlb2_lsp_cq2priov r13 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r14.field.token_count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    r14.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	r5.field.limit = port->hist_list_entry_limit - 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+	r6.field.base = port->hist_list_entry_base;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	r7.field.limit = args->cq_history_list_size;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+	r8.field.push_ptr = r6.field.base;
+	r8.field.generation = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    r8.val);
+
+	r9.field.pop_ptr = r6.field.base;
+	r9.field.generation = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+	/* Disable the port's QID mappings */
+	r13.field.v = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+	return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_ldb_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb2_ldb_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+	if (ret < 0)
+		return ret;
+
+	dlb2_ldb_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_ldb_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		    args->cq_history_list_size);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+	DLB2_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		    args->cos_strict);
+}
+
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_ldb_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+		resp->status = DLB2_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		if (dlb2_list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	} else {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			if (!dlb2_list_empty(&domain->avail_ldb_ports[i]))
+				break;
+		}
+
+		if (i == DLB2_NUM_COS_DOMAINS) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ *	its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	int ret, cos_id, i;
+
+	dlb2_log_create_ldb_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->cos_strict) {
+		cos_id = args->cos_id;
+
+		port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+					  typeof(*port));
+	} else {
+		int idx;
+
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+			port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+						  typeof(*port));
+			if (port)
+				break;
+		}
+
+		cos_id = idx;
+	}
+
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (port->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_ldb_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_ports[cos_id], &port->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_ports[cos_id], &port->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_dir_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+}
+
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+			    bool vdev_req,
+			    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+
+	return NULL;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_dir_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the queue is already configured, validate
+	 * the queue ID, its domain, and whether the queue is configured.
+	 */
+	if (args->queue_id != -1) {
+		struct dlb2_dir_pq_pair *queue;
+
+		queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+						    vdev_req,
+						    domain);
+
+		if (queue == NULL || queue->domain_id.phys_id !=
+				domain->id.phys_id ||
+				!queue->queue_configured) {
+			resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the port's queue is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->queue_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_dir_pq_pair *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_dir_pp2vas r0 = { {0} };
+	union dlb2_sys_dir_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+		union dlb2_sys_dir_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_dir_pq_pair *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_dir_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+	union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+	union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+	union dlb2_sys_dir_cq_fmt r9 = { {0} };
+	union dlb2_sys_dir_cq_at r10 = { {0} };
+	union dlb2_sys_dir_cq_pasid r11 = { {0} };
+	union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r13.field.count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    r13.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.disable_wb_opt = 0;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	r9.field.keep_pf_ppid = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+	return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_dir_pq_pair *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_dir_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb2_dir_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+
+	if (ret < 0)
+		return ret;
+
+	dlb2_dir_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ *	and queue. The port/queue pair have the same ID and name.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *port;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (args->queue_id != -1)
+		port = dlb2_get_domain_used_dir_pq(args->queue_id,
+						   vdev_req,
+						   domain);
+	else
+		port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					  typeof(*port));
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_dir_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs, &port->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs, &port->domain_list);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 14bb04b..2a089f6 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -623,3 +623,31 @@ dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_ldb_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
+
+int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_dir_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a6cd178..2a82b8f 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -228,6 +228,184 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static void *
+dlb2_alloc_coherent_aligned(const struct rte_memzone **mz, uintptr_t *phys,
+			    size_t size, int align)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t core_id = rte_lcore_id();
+	unsigned int socket_id;
+
+	snprintf(mz_name, sizeof(mz_name) - 1, "event_dlb2_pf_%lx",
+		 (unsigned long)rte_get_timer_cycles());
+	if (core_id == (unsigned int)LCORE_ID_ANY)
+		core_id = rte_get_main_lcore();
+	socket_id = rte_lcore_to_socket_id(core_id);
+	*mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
+					 RTE_MEMZONE_IOVA_CONTIG, align);
+	if (*mz == NULL) {
+		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu bytes - %s\n",
+			     size, rte_strerror(rte_errno));
+		*phys = 0;
+		return NULL;
+	}
+	*phys = (*mz)->iova;
+	return (*mz)->addr;
+}
+
+static int
+dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_ldb_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret, cq_alloc_depth;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = false;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	cq_alloc_depth = RTE_MAX(cfg->cq_depth, DLB2_MIN_HARDWARE_CQ_DEPTH);
+	alloc_sz = cq_alloc_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].cq_base = (void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+	return ret;
+}
+
+static int
+dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_dir_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret;
+	uint8_t *port_base;
+	const struct rte_memzone *mz;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = true;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	alloc_sz = cfg->cq_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&mz, &cq_base, alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
+		(void *)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
+		(void *)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].mz = mz;
+
+	dlb2_list_init_head(&port_memory.list);
+
+	cfg->response = response;
+
+	return 0;
+
+create_port_err:
+
+	rte_memzone_free(mz);
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -240,6 +418,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v4 14/23] event/dlb2: add port link

[Timothy McDaniel]

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 302 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 633 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
 6 files changed, 1007 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 9e8dcce..c6593c7 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1496,6 +1496,307 @@ dlb2_eventdev_port_setup(struct rte_eventdev *dev,
 	return 0;
 }
 
+static int16_t
+dlb2_hw_map_ldb_qid_to_port(struct dlb2_hw_dev *handle,
+			    uint32_t qm_port_id,
+			    uint16_t qm_qid,
+			    uint8_t priority)
+{
+	struct dlb2_map_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	/* Build message */
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+	cfg.priority = EV_TO_DLB2_PRIO(priority);
+
+	ret = dlb2_iface_map_qid(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: map qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		DLB2_LOG_ERR("dlb2: grp=%d, qm_port=%d, qm_qid=%d prio=%d\n",
+			     handle->domain_id, cfg.port_id,
+			     cfg.qid,
+			     cfg.priority);
+	} else {
+		DLB2_LOG_DBG("dlb2: mapped queue %d to qm_port %d\n",
+			     qm_qid, qm_port_id);
+	}
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port,
+			  struct dlb2_eventdev_queue *ev_queue,
+			  uint8_t priority)
+{
+	int first_avail = -1;
+	int ret, i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid) {
+			if (ev_port->link[i].queue_id == ev_queue->id &&
+			    ev_port->link[i].priority == priority) {
+				if (ev_port->link[i].mapped)
+					return 0; /* already mapped */
+				first_avail = i;
+			}
+		} else if (first_avail == -1)
+			first_avail = i;
+	}
+	if (first_avail == -1) {
+		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
+			     ev_port->qm_port.id);
+		return -EINVAL;
+	}
+
+	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
+					  ev_port->qm_port.id,
+					  ev_queue->qm_queue.id,
+					  priority);
+
+	if (!ret)
+		ev_port->link[first_avail].mapped = true;
+
+	return ret;
+}
+
+static int32_t
+dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 int32_t qm_port_id)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_queue_args cfg;
+	int32_t ret;
+
+	/* The directed port is always configured before its queue */
+	cfg.port_id = qm_port_id;
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else
+		cfg.depth_threshold = ev_queue->depth_threshold;
+
+	ret = dlb2_iface_dir_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	int32_t qm_qid;
+
+	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port->qm_port.id);
+
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the DIR queue\n");
+		return qm_qid;
+	}
+
+	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int
+dlb2_do_port_link(struct rte_eventdev *dev,
+		  struct dlb2_eventdev_queue *ev_queue,
+		  struct dlb2_eventdev_port *ev_port,
+		  uint8_t prio)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int err;
+
+	/* Don't link until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	if (ev_queue->qm_queue.is_directed)
+		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue, ev_port);
+	else
+		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue, prio);
+
+	if (err) {
+		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
+			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
+			     ev_queue->id, ev_port->id);
+
+		rte_errno = err;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
+			uint8_t queue_id,
+			bool link_exists,
+			int index)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	bool port_is_dir, queue_is_dir;
+
+	if (queue_id > dlb2->num_queues) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	ev_queue = &dlb2->ev_queues[queue_id];
+
+	if (!ev_queue->setup_done &&
+	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	port_is_dir = ev_port->qm_port.is_directed;
+	queue_is_dir = ev_queue->qm_queue.is_directed;
+
+	if (port_is_dir != queue_is_dir) {
+		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
+			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
+			     port_is_dir ? "DIR" : "LDB", ev_port->id);
+
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if there is space for the requested link */
+	if (!link_exists && index == -1) {
+		DLB2_LOG_ERR("no space for new link\n");
+		rte_errno = -ENOSPC;
+		return -1;
+	}
+
+	/* Check if the directed port is already linked */
+	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if the directed queue is already linked */
+	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
+			     ev_queue->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
+			const uint8_t queues[], const uint8_t priorities[],
+			uint16_t nb_links)
+
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i, j;
+
+	RTE_SET_USED(dev);
+
+	if (ev_port == NULL) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!ev_port->setup_done &&
+	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
+	 * queues pointer.
+	 */
+	if (nb_links == 0) {
+		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
+		     nb_links,
+		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
+		     ev_port->id);
+
+	for (i = 0; i < nb_links; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		uint8_t queue_id, prio;
+		bool found = false;
+		int index = -1;
+
+		queue_id = queues[i];
+		prio = priorities[i];
+
+		/* Check if the link already exists. */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].valid) {
+				if (ev_port->link[j].queue_id == queue_id) {
+					found = true;
+					index = j;
+					break;
+				}
+			} else if (index == -1) {
+				index = j;
+			}
+
+		/* could not link */
+		if (index == -1)
+			break;
+
+		/* Check if already linked at the requested priority */
+		if (found && ev_port->link[j].priority == prio)
+			continue;
+
+		if (dlb2_validate_port_link(ev_port, queue_id, found, index))
+			break; /* return index of offending queue */
+
+		ev_queue = &dlb2->ev_queues[queue_id];
+
+		if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+			break; /* return index of offending queue */
+
+		ev_queue->num_links++;
+
+		ev_port->link[index].queue_id = queue_id;
+		ev_port->link[index].priority = prio;
+		ev_port->link[index].valid = true;
+		/* Entry already exists?  If so, then must be prio change */
+		if (!found)
+			ev_port->num_links++;
+	}
+	return i;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1507,6 +1808,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_link        = dlb2_eventdev_port_link,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 01818c7..7d1ba73 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -51,3 +51,9 @@ int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 				  struct dlb2_create_dir_port_args *cfg,
 				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_dir_queue_args *cfg);
+
+int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+			  struct dlb2_map_qid_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index a6b923a..4eddd14 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -50,4 +50,10 @@ extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 struct dlb2_create_dir_port_args *cfg,
 					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+					struct dlb2_create_dir_queue_args *cfg);
+
+extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+				 struct dlb2_map_qid_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index a633d3e..14ad262 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -4895,3 +4895,636 @@ int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_dir_pq_pair *queue,
+				     struct dlb2_create_dir_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_dir_vasqid_v r0 = { {0} };
+	union dlb2_sys_dir_qid_its r1 = { {0} };
+	union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+	union dlb2_sys_dir_qid_v r5 = { {0} };
+
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r0.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r1.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+		    r1.val);
+
+	r2.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+		    r2.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		r3.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+		r4.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+	}
+
+	r5.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+	queue->queue_configured = true;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_dir_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_dir_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = dlb2_get_domain_used_dir_pq(args->port_id,
+						   vdev_req,
+						   domain);
+
+		if (port == NULL || port->domain_id.phys_id !=
+				domain->id.phys_id || !port->port_configured) {
+			resp->status = DLB2_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the queue's port is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->port_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->port_id != -1)
+		queue = dlb2_get_domain_used_dir_pq(args->port_id,
+						    vdev_req,
+						    domain);
+	else
+		queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					   typeof(*queue));
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &queue->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs,
+			      &queue->domain_list);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot,
+					      struct dlb2_map_qid_args *args)
+{
+	union dlb2_lsp_cq2priov r0;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+	r0.field.v |= 1 << slot;
+	r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+	dlb2_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+					      struct dlb2_ldb_queue *queue,
+					      struct dlb2_cmd_response *resp)
+{
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB2_QUEUE_UNMAPPED;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+			  bool vdev_req,
+			  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter)
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+			      bool vdev_req,
+			      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+
+		DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+	}
+
+	return NULL;
+}
+
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+				    u32 domain_id,
+				    struct dlb2_map_qid_args *args,
+				    struct dlb2_cmd_response *resp,
+				    bool vdev_req,
+				    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	struct dlb2_ldb_queue *queue;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB2_QID_PRIORITIES) {
+		resp->status = DLB2_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_map_qid_args *args,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	DLB2_HW_DBG(hw, "\tPriority:  %d\n",
+		    args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	int ret, i, id;
+	u8 prio;
+
+	dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_map_qid_args(hw,
+				       domain_id,
+				       args,
+				       resp,
+				       vdev_req,
+				       vdev_id);
+	if (ret)
+		return ret;
+
+	prio = args->priority;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	if (port->num_pending_removals)
+		dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].pending_priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+		if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb2_qid_map_state st;
+
+			if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+				DLB2_HW_ERR(hw,
+					    "[%s():%d] Internal error: port slot tracking failed\n",
+					    __func__, __LINE__);
+				return -EFAULT;
+			}
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb2_port_slot_state_transition(hw, port, queue,
+							      i, st);
+			if (ret)
+				return ret;
+
+			DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 2a089f6..a010e25 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -651,3 +651,13 @@ dlb2_pf_create_dir_port(struct dlb2_hw *hw,
 				       NOT_VF_REQ,
 				       PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_dir_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 2a82b8f..37be4b1 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -406,6 +406,54 @@ dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_dir_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_dir_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_dir_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
+		struct dlb2_map_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_map_qid(&dlb2_dev->hw,
+			      handle->domain_id,
+			      cfg,
+			      &response,
+			      false,
+			      0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -419,7 +467,9 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
 	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
+	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v4 15/23] event/dlb2: add port unlink and port unlinks in progress

[Timothy McDaniel]

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 163 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 283 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  51 ++++++
 5 files changed, 509 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index c6593c7..4ef7b6d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1797,6 +1797,166 @@ dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
 	return i;
 }
 
+static int16_t
+dlb2_hw_unmap_ldb_qid_from_port(struct dlb2_hw_dev *handle,
+				uint32_t qm_port_id,
+				uint16_t qm_qid)
+{
+	struct dlb2_unmap_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+
+	ret = dlb2_iface_unmap_qid(handle, &cfg);
+	if (ret < 0)
+		DLB2_LOG_ERR("dlb2: unmap qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_detach_ldb(struct dlb2_eventdev *dlb2,
+			    struct dlb2_eventdev_port *ev_port,
+			    struct dlb2_eventdev_queue *ev_queue)
+{
+	int ret, i;
+
+	/* Don't unlink until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid &&
+		    ev_port->link[i].queue_id == ev_queue->id)
+			break; /* found */
+	}
+
+	/* This is expected with eventdev API!
+	 * It blindly attemmpts to unmap all queues.
+	 */
+	if (i == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_LOG_DBG("dlb2: ignoring LB QID %d not mapped for qm_port %d.\n",
+			     ev_queue->qm_queue.id,
+			     ev_port->qm_port.id);
+		return 0;
+	}
+
+	ret = dlb2_hw_unmap_ldb_qid_from_port(&dlb2->qm_instance,
+					      ev_port->qm_port.id,
+					      ev_queue->qm_queue.id);
+	if (!ret)
+		ev_port->link[i].mapped = false;
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
+			  uint8_t queues[], uint16_t nb_unlinks)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (queues == NULL || nb_unlinks == 0) {
+		DLB2_LOG_DBG("dlb2: queues is NULL or nb_unlinks is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	/* FIXME: How to handle unlink on a directed port? */
+	if (ev_port->qm_port.is_directed) {
+		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
+			     ev_port->id);
+		rte_errno = 0;
+		return nb_unlinks; /* as if success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	for (i = 0; i < nb_unlinks; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		int ret, j;
+
+		if (queues[i] >= dlb2->num_queues) {
+			DLB2_LOG_ERR("dlb2: invalid queue id %d\n", queues[i]);
+			rte_errno = -EINVAL;
+			return i; /* return index of offending queue */
+		}
+
+		ev_queue = &dlb2->ev_queues[queues[i]];
+
+		/* Does a link exist? */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].queue_id == queues[i] &&
+			    ev_port->link[j].valid)
+				break;
+
+		if (j == DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+			continue;
+
+		ret = dlb2_event_queue_detach_ldb(dlb2, ev_port, ev_queue);
+		if (ret) {
+			DLB2_LOG_ERR("unlink err=%d for port %d queue %d\n",
+				     ret, ev_port->id, queues[i]);
+			rte_errno = -ENOENT;
+			return i; /* return index of offending queue */
+		}
+
+		ev_port->link[j].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+
+	return nb_unlinks;
+}
+
+static int
+dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
+				       void *event_port)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	struct dlb2_pending_port_unmaps_args cfg;
+	int ret;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	cfg.port_id = ev_port->qm_port.id;
+	dlb2 = ev_port->dlb2;
+	handle = &dlb2->qm_instance;
+	ret = dlb2_iface_pending_port_unmaps(handle, &cfg);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: num_unlinks_in_progress ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1809,6 +1969,9 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
 		.port_link        = dlb2_eventdev_port_link,
+		.port_unlink      = dlb2_eventdev_port_unlink,
+		.port_unlinks_in_progress =
+				    dlb2_eventdev_port_unlinks_in_progress,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 7d1ba73..bd241f3 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -57,3 +57,9 @@ int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 			  struct dlb2_map_qid_args *cfg);
+
+int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+			    struct dlb2_unmap_qid_args *cfg);
+
+int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4eddd14..e892036 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -56,4 +56,10 @@ extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 				 struct dlb2_map_qid_args *cfg);
+
+extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+				   struct dlb2_unmap_qid_args *cfg);
+
+extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 14ad262..4a3d96d 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5528,3 +5528,286 @@ int dlb2_hw_map_qid(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_unmap_qid_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+		DLB2_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			    hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+				      u32 domain_id,
+				      struct dlb2_unmap_qid_args *args,
+				      struct dlb2_cmd_response *resp,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	struct dlb2_ldb_port *port;
+	int slot;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (queue == NULL || !queue->configured) {
+		DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			    __func__, args->qid);
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		return 0;
+
+	resp->status = DLB2_ST_INVALID_QID;
+	return -EINVAL;
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	bool unmap_complete;
+	int i, ret, id;
+
+	dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_unmap_qid_args(hw,
+					 domain_id,
+					 args,
+					 resp,
+					 vdev_req,
+					 vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (port == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (queue == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+		st = DLB2_QUEUE_UNMAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		st = DLB2_QUEUE_UNMAP_IN_PROG;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB2_QUEUE_MAPPED;
+	if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: no available CQ slots\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asynchronous procedure. It requires
+	 * stopping the DLB2 from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb2_ldb_port_cq_disable(hw, port);
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete && !os_worker_active(hw))
+		os_schedule_work(hw);
+
+unmap_qid_done:
+	resp->status = 0;
+
+	return 0;
+}
+
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+				  struct dlb2_pending_port_unmaps_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+
+	dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (port == NULL || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 37be4b1..33ea73e 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -454,6 +454,55 @@ dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw_dev *handle,
+		  struct dlb2_unmap_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_unmap_qid(&dlb2_dev->hw,
+				handle->domain_id,
+				cfg,
+				&response,
+				false,
+				0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
+			    struct dlb2_pending_port_unmaps_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_pending_port_unmaps(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -470,6 +519,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
+	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

[dpdk-dev] [PATCH v4 16/23] event/dlb2: add eventdev start

[Timothy McDaniel]

Add support for the eventdev start entry point.
We delay initializing some resources until
eventdev start, since the number of linked queues can be
used to determine if we are dealing with a ldb or dir resource.
If this is a device restart, then the previous configuration
will be reapplied.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 133 +++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   3 +
 drivers/event/dlb2/dlb2_iface.h            |   3 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 123 ++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +++
 drivers/event/dlb2/pf/dlb2_pf.c            |  25 ++++++
 6 files changed, 297 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 4ef7b6d..2d233bf 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1957,6 +1957,138 @@ dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
 	return cfg.response.id;
 }
 
+static int
+dlb2_eventdev_reapply_configuration(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret, i;
+
+	/* If an event queue or port was previously configured, but hasn't been
+	 * reconfigured, reapply its original configuration.
+	 */
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+
+		ev_queue = &dlb2->ev_queues[i];
+
+		if (ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_queue_setup(dev, i, &ev_queue->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure queue %d", i);
+			return ret;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+
+		if (ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_port_setup(dev, i, &ev_port->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure ev_port %d",
+				     i);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_apply_port_links(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int i;
+
+	/* Perform requested port->queue links */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+		int j;
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			struct dlb2_eventdev_queue *ev_queue;
+			uint8_t prio, queue_id;
+
+			if (!ev_port->link[j].valid)
+				continue;
+
+			prio = ev_port->link[j].priority;
+			queue_id = ev_port->link[j].queue_id;
+
+			if (dlb2_validate_port_link(ev_port, queue_id, true, j))
+				return -EINVAL;
+
+			ev_queue = &dlb2->ev_queues[queue_id];
+
+			if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_start(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_start_domain_args cfg;
+	int ret, i;
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_ERR("bad state %d for dev_start\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return -EINVAL;
+	}
+	dlb2->run_state = DLB2_RUN_STATE_STARTING;
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	/* If the device was configured more than once, some event ports and/or
+	 * queues may need to be reconfigured.
+	 */
+	ret = dlb2_eventdev_reapply_configuration(dev);
+	if (ret)
+		return ret;
+
+	/* The DLB PMD delays port links until the device is started. */
+	ret = dlb2_eventdev_apply_port_links(dev);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		if (!dlb2->ev_ports[i].setup_done) {
+			DLB2_LOG_ERR("dlb2: port %d not setup", i);
+			return -ESTALE;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0) {
+			DLB2_LOG_ERR("dlb2: queue %d is not linked", i);
+			return -ENOLINK;
+		}
+	}
+
+	ret = dlb2_iface_sched_domain_start(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: sched_domain_start ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STARTED;
+	DLB2_LOG_DBG("dlb2: sched_domain_start completed OK\n");
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1964,6 +2096,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.dev_start        = dlb2_eventdev_start,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index bd241f3..a86191d 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -63,3 +63,6 @@ int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				     struct dlb2_start_domain_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index e892036..7ead374 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -62,4 +62,7 @@ extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				struct dlb2_start_domain_args *cfg);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 4a3d96d..c14a2f3 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5811,3 +5811,126 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 struct dlb2_cmd_response *resp,
+					 bool vdev_req,
+					 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_start_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 start domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_hw_start_domain() - Lock the domain configuration
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @arg: User-provided arguments (unused, here for ioctl callback template).
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int
+dlb2_hw_start_domain(struct dlb2_hw *hw,
+		     u32 domain_id,
+		     __attribute((unused)) struct dlb2_start_domain_args *arg,
+		     struct dlb2_cmd_response *resp,
+		     bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_queue;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+	RTE_SET_USED(arg);
+	RTE_SET_USED(iter);
+
+	dlb2_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb2_verify_start_domain_args(hw,
+					    domain_id,
+					    resp,
+					    vdev_req,
+					    vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB2 will drop all
+	 * incoming traffic to those queues.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, ldb_queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r0.val);
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+	}
+
+	dlb2_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index a010e25..aee8336 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -661,3 +661,13 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_start_domain(struct dlb2_hw *hw,
+		     u32 id,
+		     struct dlb2_start_domain_args *args,
+		     struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_start_domain(hw, id, args, resp, NOT_VF_REQ,
+				    PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 33ea73e..fff2e57 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -503,6 +503,30 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
 
 	return ret;
 }
+
+static int
+dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
+			   struct dlb2_start_domain_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_start_domain(&dlb2_dev->hw,
+				   handle->domain_id,
+				   cfg,
+				   &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -520,6 +544,7 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v4 17/23] event/dlb2: add enqueue and its burst variants

[Timothy McDaniel]

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst | 118 +++++++++
 drivers/event/dlb2/dlb2.c     | 578 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 696 insertions(+)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 8027af7..b9607b7 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -163,6 +163,124 @@ Flow ID
 The flow ID field is preserved in the event when it is scheduled in the
 DLB2.
 
+Hardware Credits
+~~~~~~~~~~~~~~~~
+
+DLB2 uses a hardware credit scheme to prevent software from overflowing hardware
+event storage, with each unit of storage represented by a credit. A port spends
+a credit to enqueue an event, and hardware refills the ports with credits as the
+events are scheduled to ports. Refills come from credit pools, and each port is
+a member of a load-balanced credit pool and a directed credit pool. The
+load-balanced credits are used to enqueue to load-balanced queues, and directed
+credits are used for directed queues.
+
+A DLB2 eventdev contains one load-balanced and one directed credit pool. These
+pools' sizes are controlled by the nb_events_limit field in struct
+rte_event_dev_config. The load-balanced pool is sized to contain
+nb_events_limit credits, and the directed pool is sized to contain
+nb_events_limit/4 credits. The directed pool size can be overridden with the
+num_dir_credits vdev argument, like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,num_dir_credits=<value>
+
+This can be used if the default allocation is too low or too high for the
+specific application needs. The PMD also supports a vdev arg that limits the
+max_num_events reported by rte_event_dev_info_get():
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,max_num_events=<value>
+
+By default, max_num_events is reported as the total available load-balanced
+credits. If multiple DLB2-based applications are being used, it may be desirable
+to control how many load-balanced credits each application uses, particularly
+when application(s) are written to configure nb_events_limit equal to the
+reported max_num_events.
+
+Each port is a member of both credit pools. A port's credit allocation is
+defined by its low watermark, high watermark, and refill quanta. These three
+parameters are calculated by the dlb PMD like so:
+
+- The load-balanced high watermark is set to the port's enqueue_depth.
+  The directed high watermark is set to the minimum of the enqueue_depth and
+  the directed pool size divided by the total number of ports.
+- The refill quanta is set to half the high watermark.
+- The low watermark is set to the minimum of 16 and the refill quanta.
+
+When the eventdev is started, each port is pre-allocated a high watermark's
+worth of credits. For example, if an eventdev contains four ports with enqueue
+depths of 32 and a load-balanced credit pool size of 4096, each port will start
+with 32 load-balanced credits, and there will be 3968 credits available to
+replenish the ports. Thus, a single port is not capable of enqueueing up to the
+nb_events_limit (without any events being dequeued), since the other ports are
+retaining their initial credit allocation; in short, all ports must enqueue in
+order to reach the limit.
+
+If a port attempts to enqueue and has no credits available, the enqueue
+operation will fail and the application must retry the enqueue. Credits are
+replenished asynchronously by the DLB2 hardware.
+
+Software Credits
+~~~~~~~~~~~~~~~~
+
+The DLB2 is a "closed system" event dev, and the DLB2 PMD layers a software
+credit scheme on top of the hardware credit scheme in order to comply with
+the per-port backpressure described in the eventdev API.
+
+The DLB2's hardware scheme is local to a queue/pipeline stage: a port spends a
+credit when it enqueues to a queue, and credits are later replenished after the
+events are dequeued and released.
+
+In the software credit scheme, a credit is consumed when a new (.op =
+RTE_EVENT_OP_NEW) event is injected into the system, and the credit is
+replenished when the event is released from the system (either explicitly with
+RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()).
+
+In this model, an event is "in the system" from its first enqueue into eventdev
+until it is last dequeued. If the event goes through multiple event queues, it
+is still considered "in the system" while a worker thread is processing it.
+
+A port will fail to enqueue if the number of events in the system exceeds its
+``new_event_threshold`` (specified at port setup time). A port will also fail
+to enqueue if it lacks enough hardware credits to enqueue; load-balanced
+credits are used to enqueue to a load-balanced queue, and directed credits are
+used to enqueue to a directed queue.
+
+The out-of-credit situations are typically transient, and an eventdev
+application using the DLB2 ought to retry its enqueues if they fail.
+If enqueue fails, DLB2 PMD sets rte_errno as follows:
+
+- -ENOSPC: Credit exhaustion (either hardware or software)
+- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type.
+
+Depending on the pipeline the application has constructed, it's possible to
+enter a credit deadlock scenario wherein the worker thread lacks the credit
+to enqueue an event, and it must dequeue an event before it can recover the
+credit. If the worker thread retries its enqueue indefinitely, it will not
+make forward progress. Such deadlock is possible if the application has event
+"loops", in which an event in dequeued from queue A and later enqueued back to
+queue A.
+
+Due to this, workers should stop retrying after a time, release the events it
+is attempting to enqueue, and dequeue more events. It is important that the
+worker release the events and don't simply set them aside to retry the enqueue
+again later, because the port has limited history list size (by default, twice
+the port's dequeue_depth).
+
+Priority
+~~~~~~~~
+
+The DLB2 supports event priority and per-port queue service priority, as
+described in the eventdev header file. The DLB2 does not support 'global' event
+queue priority established at queue creation time.
+
+DLB2 supports 8 event and queue service priority levels. For both priority
+types, the PMD uses the upper three bits of the priority field to determine the
+DLB2 priority, discarding the 5 least significant bits. The 5 least significant
+event priority bits are not preserved when an event is enqueued.
+
 Reconfiguration
 ~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 2d233bf..90a5c9d 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2089,6 +2089,578 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
 	return 0;
 }
 
+static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
+	{
+		/* Load-balanced cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
+	},
+	{
+		/* Directed cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
+	},
+};
+
+static inline uint32_t
+dlb2_port_credits_get(struct dlb2_port *qm_port,
+		      enum dlb2_hw_queue_types type)
+{
+	uint32_t credits = *qm_port->credit_pool[type];
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	if (unlikely(credits < batch_size))
+		batch_size = credits;
+
+	if (likely(credits &&
+		   __atomic_compare_exchange_n(
+			qm_port->credit_pool[type],
+			&credits, credits - batch_size, false,
+			__ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
+		return batch_size;
+	else
+		return 0;
+}
+
+static inline void
+dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port)
+{
+	uint16_t quanta = ev_port->credit_update_quanta;
+
+	if (ev_port->inflight_credits >= quanta * 2) {
+		/* Replenish credits, saving one quanta for enqueues */
+		uint16_t val = ev_port->inflight_credits - quanta;
+
+		__atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits -= val;
+	}
+}
+
+static inline int
+dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
+						__ATOMIC_SEQ_CST);
+	const int num = 1;
+
+	if (unlikely(ev_port->inflight_max < sw_inflights)) {
+		DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
+		rte_errno = -ENOSPC;
+		return 1;
+	}
+
+	if (ev_port->inflight_credits < num) {
+		/* check if event enqueue brings ev_port over max threshold */
+		uint32_t credit_update_quanta = ev_port->credit_update_quanta;
+
+		if (sw_inflights + credit_update_quanta >
+				dlb2->new_event_limit) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_new_event_limit,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+
+		__atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
+				   __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits += (credit_update_quanta);
+
+		if (ev_port->inflight_credits < num) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_inflight_credits,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_ldb_credits == 0)) {
+		qm_port->cached_ldb_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_LDB_QUEUE);
+		if (unlikely(qm_port->cached_ldb_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
+			1);
+			DLB2_LOG_DBG("ldb credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_dir_credits == 0)) {
+		qm_port->cached_dir_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_DIR_QUEUE);
+		if (unlikely(qm_port->cached_dir_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
+			1);
+			DLB2_LOG_DBG("dir credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static __rte_always_inline void
+dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
+	      struct process_local_port_data *port_data)
+{
+	dlb2_movdir64b(port_data->pp_addr, qe4);
+}
+
+static inline int
+dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_cq_pop_qe *qe;
+
+	RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
+
+	qe = qm_port->consume_qe;
+
+	qe->tokens = num - 1;
+
+	/* No store fence needed since no pointer is being sent, and CQ token
+	 * pops can be safely reordered with other HCWs.
+	 */
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	dlb2_movntdq_single(port_data->pp_addr, qe);
+
+	DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
+
+	qm_port->owed_tokens = 0;
+
+	return 0;
+}
+
+static inline void
+dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
+		   bool do_sfence,
+		   struct process_local_port_data *port_data)
+{
+	/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
+	 * application writes complete before enqueueing the QE.
+	 */
+	if (do_sfence)
+		rte_wmb();
+
+	dlb2_pp_write(qm_port->qe4, port_data);
+}
+
+static inline void
+dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
+{
+	struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
+	int num = qm_port->owed_tokens;
+
+	qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
+	qe[idx].tokens = num - 1;
+
+	qm_port->owed_tokens = 0;
+}
+
+static inline void
+dlb2_event_build_hcws(struct dlb2_port *qm_port,
+		      const struct rte_event ev[],
+		      int num,
+		      uint8_t *sched_type,
+		      uint8_t *queue_id)
+{
+	struct dlb2_enqueue_qe *qe;
+	uint16_t sched_word[4];
+	__m128i sse_qe[2];
+	int i;
+
+	qe = qm_port->qe4;
+
+	sse_qe[0] = _mm_setzero_si128();
+	sse_qe[1] = _mm_setzero_si128();
+
+	switch (num) {
+	case 4:
+		/* Construct the metadata portion of two HCWs in one 128b SSE
+		 * register. HCW metadata is constructed in the SSE registers
+		 * like so:
+		 * sse_qe[0][63:0]:   qe[0]'s metadata
+		 * sse_qe[0][127:64]: qe[1]'s metadata
+		 * sse_qe[1][63:0]:   qe[2]'s metadata
+		 * sse_qe[1][127:64]: qe[3]'s metadata
+		 */
+
+		/* Convert the event operation into a command byte and store it
+		 * in the metadata:
+		 * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
+		 * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
+		 * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
+		 * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
+		 */
+#define DLB2_QE_CMD_BYTE 7
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[0].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[1].op],
+				DLB2_QE_CMD_BYTE + 8);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[2].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[3].op],
+				DLB2_QE_CMD_BYTE + 8);
+
+		/* Store priority, scheduling type, and queue ID in the sched
+		 * word array because these values are re-used when the
+		 * destination is a directed queue.
+		 */
+		sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
+				sched_type[0] << 8 |
+				queue_id[0];
+		sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
+				sched_type[1] << 8 |
+				queue_id[1];
+		sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
+				sched_type[2] << 8 |
+				queue_id[2];
+		sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
+				sched_type[3] << 8 |
+				queue_id[3];
+
+		/* Store the event priority, scheduling type, and queue ID in
+		 * the metadata:
+		 * sse_qe[0][31:16] = sched_word[0]
+		 * sse_qe[0][95:80] = sched_word[1]
+		 * sse_qe[1][31:16] = sched_word[2]
+		 * sse_qe[1][95:80] = sched_word[3]
+		 */
+#define DLB2_QE_QID_SCHED_WORD 1
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[0],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[1],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[2],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[3],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+
+		/* If the destination is a load-balanced queue, store the lock
+		 * ID. If it is a directed queue, DLB places this field in
+		 * bytes 10-11 of the received QE, so we format it accordingly:
+		 * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
+		 * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
+		 * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
+		 * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
+		 */
+#define DLB2_QE_LOCK_ID_WORD 2
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[0] == DLB2_SCHED_DIRECTED) ?
+					sched_word[0] : ev[0].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[1] == DLB2_SCHED_DIRECTED) ?
+					sched_word[1] : ev[1].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[2] == DLB2_SCHED_DIRECTED) ?
+					sched_word[2] : ev[2].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[3] == DLB2_SCHED_DIRECTED) ?
+					sched_word[3] : ev[3].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+
+		/* Store the event type and sub event type in the metadata:
+		 * sse_qe[0][15:0]  = flow_id[0]
+		 * sse_qe[0][79:64] = flow_id[1]
+		 * sse_qe[1][15:0]  = flow_id[2]
+		 * sse_qe[1][79:64] = flow_id[3]
+		 */
+#define DLB2_QE_EV_TYPE_WORD 0
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[0].sub_event_type << 8 |
+						ev[0].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[1].sub_event_type << 8 |
+						ev[1].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[2].sub_event_type << 8 |
+						ev[2].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[3].sub_event_type << 8 |
+						ev[3].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+
+		/* Store the metadata to memory (use the double-precision
+		 * _mm_storeh_pd because there is no integer function for
+		 * storing the upper 64b):
+		 * qe[0] metadata = sse_qe[0][63:0]
+		 * qe[1] metadata = sse_qe[0][127:64]
+		 * qe[2] metadata = sse_qe[1][63:0]
+		 * qe[3] metadata = sse_qe[1][127:64]
+		 */
+		_mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
+		_mm_storeh_pd((double *)&qe[1].u.opaque_data,
+			      (__m128d)sse_qe[0]);
+		_mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
+		_mm_storeh_pd((double *)&qe[3].u.opaque_data,
+			      (__m128d)sse_qe[1]);
+
+		qe[0].data = ev[0].u64;
+		qe[1].data = ev[1].u64;
+		qe[2].data = ev[2].u64;
+		qe[3].data = ev[3].u64;
+
+		break;
+	case 3:
+	case 2:
+	case 1:
+		/* At least one QE will be valid, so only zero out three */
+		qe[1].cmd_byte = 0;
+		qe[2].cmd_byte = 0;
+		qe[3].cmd_byte = 0;
+
+		for (i = 0; i < num; i++) {
+			qe[i].cmd_byte =
+				cmd_byte_map[qm_port->is_directed][ev[i].op];
+			qe[i].sched_type = sched_type[i];
+			qe[i].data = ev[i].u64;
+			qe[i].qid = queue_id[i];
+			qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
+			qe[i].lock_id = ev[i].flow_id;
+			if (sched_type[i] == DLB2_SCHED_DIRECTED) {
+				struct dlb2_msg_info *info =
+					(struct dlb2_msg_info *)&qe[i].lock_id;
+
+				info->qid = queue_id[i];
+				info->sched_type = DLB2_SCHED_DIRECTED;
+				info->priority = qe[i].priority;
+			}
+			qe[i].u.event_type.major = ev[i].event_type;
+			qe[i].u.event_type.sub = ev[i].sub_event_type;
+		}
+		break;
+	}
+}
+
+static inline int
+dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
+			struct dlb2_port *qm_port,
+			const struct rte_event ev[],
+			uint8_t *sched_type,
+			uint8_t *queue_id)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	uint16_t *cached_credits = NULL;
+	struct dlb2_queue *qm_queue;
+
+	ev_queue = &dlb2->ev_queues[ev->queue_id];
+	qm_queue = &ev_queue->qm_queue;
+	*queue_id = qm_queue->id;
+
+	/* Ignore sched_type and hardware credits on release events */
+	if (ev->op == RTE_EVENT_OP_RELEASE)
+		goto op_check;
+
+	if (!qm_queue->is_directed) {
+		/* Load balanced destination queue */
+
+		if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_ldb_credits;
+
+		switch (ev->sched_type) {
+		case RTE_SCHED_TYPE_ORDERED:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
+			if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
+				DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
+					     *queue_id);
+				rte_errno = -EINVAL;
+				return 1;
+			}
+			*sched_type = DLB2_SCHED_ORDERED;
+			break;
+		case RTE_SCHED_TYPE_ATOMIC:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
+			*sched_type = DLB2_SCHED_ATOMIC;
+			break;
+		case RTE_SCHED_TYPE_PARALLEL:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
+			if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
+				*sched_type = DLB2_SCHED_ORDERED;
+			else
+				*sched_type = DLB2_SCHED_UNORDERED;
+			break;
+		default:
+			DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
+			DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
+			rte_errno = -EINVAL;
+			return 1;
+		}
+	} else {
+		/* Directed destination queue */
+
+		if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_dir_credits;
+
+		DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
+
+		*sched_type = DLB2_SCHED_DIRECTED;
+	}
+
+op_check:
+	switch (ev->op) {
+	case RTE_EVENT_OP_NEW:
+		/* Check that a sw credit is available */
+		if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		ev_port->inflight_credits--;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_FORWARD:
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_RELEASE:
+		ev_port->inflight_credits++;
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+
+		/* Replenish s/w credits if enough are cached */
+		dlb2_replenish_sw_credits(dlb2, ev_port);
+		break;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+	if (ev->op != RTE_EVENT_OP_RELEASE) {
+		DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
+		DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
+	}
+#endif
+
+	return 0;
+}
+
+static inline uint16_t
+dlb2_event_enqueue_burst(void *event_port,
+			 const struct rte_event events[],
+			 uint16_t num)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct process_local_port_data *port_data;
+	int i, cnt;
+
+	RTE_ASSERT(ev_port->enq_configured);
+	RTE_ASSERT(events != NULL);
+
+	cnt = 0;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
+		int j = 0;
+
+		for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
+			const struct rte_event *ev = &events[i + j];
+
+			if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
+						    &sched_types[j],
+						    &queue_ids[j]))
+				break;
+		}
+
+		if (j == 0)
+			break;
+
+		dlb2_event_build_hcws(qm_port, &events[i], j,
+				      sched_types, queue_ids);
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+
+		if (j < DLB2_NUM_QES_PER_CACHE_LINE)
+			break;
+	}
+
+	return cnt;
+}
+
+static inline uint16_t
+dlb2_event_enqueue(void *event_port,
+		   const struct rte_event events[])
+{
+	return dlb2_event_enqueue_burst(event_port, events, 1);
+}
+
+static uint16_t
+dlb2_event_enqueue_new_burst(void *event_port,
+			     const struct rte_event events[],
+			     uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
+static uint16_t
+dlb2_event_enqueue_forward_burst(void *event_port,
+				 const struct rte_event events[],
+				 uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2112,7 +2684,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
 	};
 
+	/* Expose PMD's eventdev interface */
+
 	dev->dev_ops = &dlb2_eventdev_entry_ops;
+	dev->enqueue = dlb2_event_enqueue;
+	dev->enqueue_burst = dlb2_event_enqueue_burst;
+	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
+	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v4 18/23] event/dlb2: add dequeue and its burst variants

[Timothy McDaniel]

Add support for dequeue, dequeue_burst, ...

DLB2 does not currently support interrupts, but instead use
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB2 supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/eventdevs/dlb2.rst |  56 ++--
 drivers/event/dlb2/dlb2.c     | 761 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 782 insertions(+), 35 deletions(-)

diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index b9607b7..770227f 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -82,41 +82,6 @@ The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
 the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
 load-balanced queues can use the full 16-bit flow ID range.
 
-Load-Balanced Queues
-~~~~~~~~~~~~~~~~~~~
-
-A load-balanced queue can support atomic and ordered scheduling, or atomic and
-unordered scheduling, but not atomic and unordered and ordered scheduling. A
-queue's scheduling types are controlled by the event queue configuration.
-
-If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the
-``nb_atomic_order_sequences`` determines the supported scheduling types.
-With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic
-and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is
-supported by scheduling those events as ordered events.  Note that when the
-event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if
-``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and
-unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported.
-
-If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type
-dictates the queue's scheduling type.
-
-The ``nb_atomic_order_sequences`` queue configuration field sets the ordered
-queue's reorder buffer size.  DLB2 has 4 groups of ordered queues, where each
-group is configured to contain either 1 queue with 1024 reorder entries, 2
-queues with 512 reorder entries, and so on down to 32 queues with 32 entries.
-
-When a load-balanced queue is created, the PMD will configure a new sequence
-number group on-demand if num_sequence_numbers does not match a pre-existing
-group with available reorder buffer entries. If all sequence number groups are
-in use, no new group will be created and queue configuration will fail. (Note
-that when the PMD is used with a virtual DLB2 device, it cannot change the
-sequence number configuration.)
-
-The queue's ``nb_atomic_flows`` parameter is ignored by the DLB2 PMD, because
-the DLB2 does not limit the number of flows a queue can track. In the DLB2, all
-load-balanced queues can use the full 16-bit flow ID range.
-
 Load-balanced and Directed Ports
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -314,6 +279,27 @@ The PMD does not support the following configuration sequences:
 This sequence is not supported because the event device must be reconfigured
 before its ports or queues can be.
 
+Deferred Scheduling
+~~~~~~~~~~~~~~~~~~~
+
+The DLB2 PMD's default behavior for managing a CQ is to "pop" the CQ once per
+dequeued event before returning from rte_event_dequeue_burst(). This frees the
+corresponding entries in the CQ, which enables the DLB2 to schedule more events
+to it.
+
+To support applications seeking finer-grained scheduling control -- for example
+deferring scheduling to get the best possible priority scheduling and
+load-balancing -- the PMD supports a deferred scheduling mode. In this mode,
+the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst()
+call. This mode only applies to load-balanced event ports with dequeue depth of
+1.
+
+To enable deferred scheduling, use the defer_sched vdev argument like so:
+
+    .. code-block:: console
+
+       --vdev=dlb1_event,defer_sched=on
+
 Atomic Inflights Allocation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 90a5c9d..bd96055 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2661,9 +2661,761 @@ dlb2_event_enqueue_forward_burst(void *event_port,
 	return dlb2_event_enqueue_burst(event_port, events, num);
 }
 
+static inline void
+dlb2_port_credits_inc(struct dlb2_port *qm_port, int num)
+{
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	/* increment port credits, and return to pool if exceeds threshold */
+	if (!qm_port->is_directed) {
+		qm_port->cached_ldb_credits += num;
+		if (qm_port->cached_ldb_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_LDB_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_ldb_credits -= batch_size;
+		}
+	} else {
+		qm_port->cached_dir_credits += num;
+		if (qm_port->cached_dir_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_DIR_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_dir_credits -= batch_size;
+		}
+	}
+}
+
+static inline bool
+dlb2_cq_is_empty(struct dlb2_port *qm_port)
+{
+	volatile struct dlb2_dequeue_qe *qe_ptr;
+	struct dlb2_dequeue_qe qe;
+
+	qe_ptr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	qe = qe_ptr[qm_port->cq_idx];
+
+	return (qe.cq_gen != qm_port->gen_bit);
+}
+
+static inline int
+dlb2_dequeue_wait(struct dlb2_eventdev *dlb2,
+		  struct dlb2_eventdev_port *ev_port,
+		  struct dlb2_port *qm_port,
+		  uint64_t timeout,
+		  uint64_t start_ticks)
+{
+	struct process_local_port_data *port_data;
+	uint64_t elapsed_ticks;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	elapsed_ticks = rte_get_timer_cycles() - start_ticks;
+
+	/* Wait/poll time expired */
+	if (elapsed_ticks >= timeout) {
+		return 1;
+	} else if (dlb2->umwait_allowed) {
+		volatile struct dlb2_dequeue_qe *cq_base;
+
+		cq_base = port_data->cq_base;
+
+		/* Block on cache line write to CQ. Note: it's
+		 * safe to access the per-process cq_base
+		 * address here, since the PMD has already
+		 * attempted at least one CQ dequeue.
+		 */
+		dlb2_umonitor(&cq_base[qm_port->cq_idx]);
+
+		/* Avoid race condition. Check if still empty */
+		if (dlb2_cq_is_empty(qm_port)) {
+			dlb2_umwait(RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE,
+				    timeout + start_ticks);
+			DLB2_INC_STAT(
+				ev_port->stats.traffic.rx_umonitor_umwait, 1);
+		}
+	} else {
+		uint64_t poll_interval = RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL;
+		uint64_t curr_ticks = rte_get_timer_cycles();
+		uint64_t init_ticks = curr_ticks;
+
+		while ((curr_ticks - start_ticks < timeout) &&
+		       (curr_ticks - init_ticks < poll_interval))
+			curr_ticks = rte_get_timer_cycles();
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
+			 struct dlb2_port *qm_port,
+			 struct rte_event *events,
+			 struct dlb2_dequeue_qe *qes,
+			 int cnt)
+{
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	int i, num, evq_id;
+
+	for (i = 0, num = 0; i < cnt; i++) {
+		struct dlb2_dequeue_qe *qe = &qes[i];
+		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
+			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+			[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+		};
+
+		/* Fill in event information.
+		 * Note that flow_id must be embedded in the data by
+		 * the app, such as the mbuf RSS hash field if the data
+		 * buffer is a mbuf.
+		 */
+		if (unlikely(qe->error)) {
+			DLB2_LOG_ERR("QE error bit ON\n");
+			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
+			dlb2_consume_qe_immediate(qm_port, 1);
+			continue; /* Ignore */
+		}
+
+		events[num].u64 = qe->data;
+		events[num].flow_id = qe->flow_id;
+		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
+		events[num].event_type = qe->u.event_type.major;
+		events[num].sub_event_type = qe->u.event_type.sub;
+		events[num].sched_type = sched_type_map[qe->sched_type];
+		events[num].impl_opaque = qe->qid_depth;
+
+		/* qid not preserved for directed queues */
+		if (qm_port->is_directed)
+			evq_id = ev_port->link[0].queue_id;
+		else
+			evq_id = qid_mappings[qe->qid];
+
+		events[num].queue_id = evq_id;
+		DLB2_INC_STAT(
+			ev_port->stats.queue[evq_id].qid_depth[qe->qid_depth],
+			1);
+		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type], 1);
+		num++;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num);
+
+	return num;
+}
+
+static inline int
+dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
+			      struct dlb2_port *qm_port,
+			      struct rte_event *events,
+			      struct dlb2_dequeue_qe *qes)
+{
+	int sched_type_map[] = {
+		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+	};
+	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	__m128i sse_evt[2];
+
+	/* In the unlikely case that any of the QE error bits are set, process
+	 * them one at a time.
+	 */
+	if (unlikely(qes[0].error || qes[1].error ||
+		     qes[2].error || qes[3].error))
+		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
+						 qes, num_events);
+
+	events[0].u64 = qes[0].data;
+	events[1].u64 = qes[1].data;
+	events[2].u64 = qes[2].data;
+	events[3].u64 = qes[3].data;
+
+	/* Construct the metadata portion of two struct rte_events
+	 * in one 128b SSE register. Event metadata is constructed in the SSE
+	 * registers like so:
+	 * sse_evt[0][63:0]:   event[0]'s metadata
+	 * sse_evt[0][127:64]: event[1]'s metadata
+	 * sse_evt[1][63:0]:   event[2]'s metadata
+	 * sse_evt[1][127:64]: event[3]'s metadata
+	 */
+	sse_evt[0] = _mm_setzero_si128();
+	sse_evt[1] = _mm_setzero_si128();
+
+	/* Convert the hardware queue ID to an event queue ID and store it in
+	 * the metadata:
+	 * sse_evt[0][47:40]   = qid_mappings[qes[0].qid]
+	 * sse_evt[0][111:104] = qid_mappings[qes[1].qid]
+	 * sse_evt[1][47:40]   = qid_mappings[qes[2].qid]
+	 * sse_evt[1][111:104] = qid_mappings[qes[3].qid]
+	 */
+#define DLB_EVENT_QUEUE_ID_BYTE 5
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[0].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[1].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[2].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[3].qid],
+				     DLB_EVENT_QUEUE_ID_BYTE + 8);
+
+	/* Convert the hardware priority to an event priority and store it in
+	 * the metadata, while also returning the queue depth status
+	 * value captured by the hardware, storing it in impl_opaque, which can
+	 * be read by the application but not modified
+	 * sse_evt[0][55:48]   = DLB2_TO_EV_PRIO(qes[0].priority)
+	 * sse_evt[0][63:56]   = qes[0].qid_depth
+	 * sse_evt[0][119:112] = DLB2_TO_EV_PRIO(qes[1].priority)
+	 * sse_evt[0][127:120] = qes[1].qid_depth
+	 * sse_evt[1][55:48]   = DLB2_TO_EV_PRIO(qes[2].priority)
+	 * sse_evt[1][63:56]   = qes[2].qid_depth
+	 * sse_evt[1][119:112] = DLB2_TO_EV_PRIO(qes[3].priority)
+	 * sse_evt[1][127:120] = qes[3].qid_depth
+	 */
+#define DLB_EVENT_PRIO_IMPL_OPAQUE_WORD 3
+#define DLB_BYTE_SHIFT 8
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[0].priority) |
+			(qes[0].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[1].priority) |
+			(qes[1].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[2].priority) |
+			(qes[2].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[3].priority) |
+			(qes[3].qid_depth << DLB_BYTE_SHIFT),
+			DLB_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+
+	/* Write the event type, sub event type, and flow_id to the event
+	 * metadata.
+	 * sse_evt[0][31:0]   = qes[0].flow_id |
+	 *			qes[0].u.event_type.major << 28 |
+	 *			qes[0].u.event_type.sub << 20;
+	 * sse_evt[0][95:64]  = qes[1].flow_id |
+	 *			qes[1].u.event_type.major << 28 |
+	 *			qes[1].u.event_type.sub << 20;
+	 * sse_evt[1][31:0]   = qes[2].flow_id |
+	 *			qes[2].u.event_type.major << 28 |
+	 *			qes[2].u.event_type.sub << 20;
+	 * sse_evt[1][95:64]  = qes[3].flow_id |
+	 *			qes[3].u.event_type.major << 28 |
+	 *			qes[3].u.event_type.sub << 20;
+	 */
+#define DLB_EVENT_EV_TYPE_DW 0
+#define DLB_EVENT_EV_TYPE_SHIFT 28
+#define DLB_EVENT_SUB_EV_TYPE_SHIFT 20
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[0].flow_id |
+			qes[0].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[0].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[1].flow_id |
+			qes[1].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[1].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[2].flow_id |
+			qes[2].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT |
+			qes[2].u.event_type.sub <<  DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[3].flow_id |
+			qes[3].u.event_type.major << DLB_EVENT_EV_TYPE_SHIFT  |
+			qes[3].u.event_type.sub << DLB_EVENT_SUB_EV_TYPE_SHIFT,
+			DLB_EVENT_EV_TYPE_DW + 2);
+
+	/* Write the sched type to the event metadata. 'op' and 'rsvd' are not
+	 * set:
+	 * sse_evt[0][39:32]  = sched_type_map[qes[0].sched_type] << 6
+	 * sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6
+	 * sse_evt[1][39:32]  = sched_type_map[qes[2].sched_type] << 6
+	 * sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6
+	 */
+#define DLB_EVENT_SCHED_TYPE_BYTE 4
+#define DLB_EVENT_SCHED_TYPE_SHIFT 6
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[0].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[1].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[2].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[3].sched_type] << DLB_EVENT_SCHED_TYPE_SHIFT,
+		DLB_EVENT_SCHED_TYPE_BYTE + 8);
+
+	/* Store the metadata to the event (use the double-precision
+	 * _mm_storeh_pd because there is no integer function for storing the
+	 * upper 64b):
+	 * events[0].event = sse_evt[0][63:0]
+	 * events[1].event = sse_evt[0][127:64]
+	 * events[2].event = sse_evt[1][63:0]
+	 * events[3].event = sse_evt[1][127:64]
+	 */
+	_mm_storel_epi64((__m128i *)&events[0].event, sse_evt[0]);
+	_mm_storeh_pd((double *)&events[1].event, (__m128d) sse_evt[0]);
+	_mm_storel_epi64((__m128i *)&events[2].event, sse_evt[1]);
+	_mm_storeh_pd((double *)&events[3].event, (__m128d) sse_evt[1]);
+
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[0].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[1].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[2].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[3].sched_type], 1);
+
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[0].queue_id].
+			qid_depth[qes[0].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[1].queue_id].
+			qid_depth[qes[1].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[2].queue_id].
+			qid_depth[qes[2].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[3].queue_id].
+			qid_depth[qes[3].qid_depth],
+		1);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num_events);
+
+	return num_events;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe_sparse(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe)
+{
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	uint8_t xor_mask[2] = {0x0F, 0x00};
+	const uint8_t and_mask = 0x0F;
+	__m128i *qes = (__m128i *)qe;
+	uint8_t gen_bits, gen_bit;
+	uintptr_t addr[4];
+	uint16_t idx;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+
+	idx = qm_port->cq_idx;
+
+	/* Load the next 4 QEs */
+	addr[0] = (uintptr_t)&cq_addr[idx];
+	addr[1] = (uintptr_t)&cq_addr[(idx +  4) & qm_port->cq_depth_mask];
+	addr[2] = (uintptr_t)&cq_addr[(idx +  8) & qm_port->cq_depth_mask];
+	addr[3] = (uintptr_t)&cq_addr[(idx + 12) & qm_port->cq_depth_mask];
+
+	/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */
+	rte_prefetch0(&cq_addr[(idx + 16) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 20) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 24) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 28) & qm_port->cq_depth_mask]);
+
+	/* Correct the xor_mask for wrap-around QEs */
+	gen_bit = qm_port->gen_bit;
+	xor_mask[gen_bit] ^= !!((idx +  4) > qm_port->cq_depth_mask) << 1;
+	xor_mask[gen_bit] ^= !!((idx +  8) > qm_port->cq_depth_mask) << 2;
+	xor_mask[gen_bit] ^= !!((idx + 12) > qm_port->cq_depth_mask) << 3;
+
+	/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is
+	 * valid, then QEs[0:N-1] are too.
+	 */
+	qes[3] = _mm_load_si128((__m128i *)(void *)addr[3]);
+	rte_compiler_barrier();
+	qes[2] = _mm_load_si128((__m128i *)(void *)addr[2]);
+	rte_compiler_barrier();
+	qes[1] = _mm_load_si128((__m128i *)(void *)addr[1]);
+	rte_compiler_barrier();
+	qes[0] = _mm_load_si128((__m128i *)(void *)addr[0]);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[gen_bit];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask;
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline void
+dlb2_inc_cq_idx(struct dlb2_port *qm_port, int cnt)
+{
+	uint16_t idx = qm_port->cq_idx_unmasked + cnt;
+
+	qm_port->cq_idx_unmasked = idx;
+	qm_port->cq_idx = idx & qm_port->cq_depth_mask;
+	qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) & 0x1;
+}
+
+static int
+dlb2_event_release(struct dlb2_eventdev *dlb2,
+		   uint8_t port_id,
+		   int n)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_eventdev_port *ev_port;
+	struct dlb2_port *qm_port;
+	int i, cnt;
+
+	if (port_id > dlb2->num_ports) {
+		DLB2_LOG_ERR("Invalid port id %d in dlb2-event_release\n",
+			     port_id);
+		rte_errno = -EINVAL;
+		return rte_errno;
+	}
+
+	ev_port = &dlb2->ev_ports[port_id];
+	qm_port = &ev_port->qm_port;
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	cnt = 0;
+
+	if (qm_port->is_directed) {
+		cnt = n;
+		goto sw_credit_update;
+	}
+
+	for (i = 0; i < n; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		int j;
+
+		/* Zero-out QEs */
+		qm_port->qe4[0].cmd_byte = 0;
+		qm_port->qe4[1].cmd_byte = 0;
+		qm_port->qe4[2].cmd_byte = 0;
+		qm_port->qe4[3].cmd_byte = 0;
+
+		for (j = 0; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++)
+			qm_port->qe4[j].cmd_byte = DLB2_COMP_CMD_BYTE;
+
+		qm_port->issued_releases += j;
+
+		if (j == 0)
+			break;
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+	}
+
+sw_credit_update:
+	/* each release returns one credit */
+	if (!ev_port->outstanding_releases) {
+		DLB2_LOG_ERR("Unrecoverable application error. Outstanding releases underflowed.\n");
+		rte_errno = -ENOTRECOVERABLE;
+		return rte_errno;
+	}
+
+	ev_port->outstanding_releases -= cnt;
+	ev_port->inflight_credits += cnt;
+
+	/* Replenish s/w credits if enough releases are performed */
+	dlb2_replenish_sw_credits(dlb2, ev_port);
+	return 0;
+}
+
+static inline int16_t
+dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
+		       struct dlb2_eventdev_port *ev_port,
+		       struct rte_event *events,
+		       uint16_t max_num,
+		       uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe_sparse(qm_port, qes);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail << 2);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							      qm_port,
+							      &events[num],
+							      &qes[0]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							 qm_port,
+							 &events[num],
+							 &qes[0],
+							 num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe,
+	     uint8_t *offset)
+{
+	uint8_t xor_mask[2][4] = { {0x0F, 0x0E, 0x0C, 0x08},
+				   {0x00, 0x01, 0x03, 0x07} };
+	uint8_t and_mask[4] = {0x0F, 0x0E, 0x0C, 0x08};
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	__m128i *qes = (__m128i *)qe;
+	uint64_t *cache_line_base;
+	uint8_t gen_bits;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	cq_addr = &cq_addr[qm_port->cq_idx];
+
+	cache_line_base = (void *)(((uintptr_t)cq_addr) & ~0x3F);
+	*offset = ((uintptr_t)cq_addr & 0x30) >> 4;
+
+	/* Load the next CQ cache line from memory. Pack these reads as tight
+	 * as possible to reduce the chance that DLB invalidates the line while
+	 * the CPU is reading it. Read the cache line backwards to ensure that
+	 * if QE[N] (N > 0) is valid, then QEs[0:N-1] are too.
+	 *
+	 * (Valid QEs start at &qe[offset])
+	 */
+	qes[3] = _mm_load_si128((__m128i *)&cache_line_base[6]);
+	qes[2] = _mm_load_si128((__m128i *)&cache_line_base[4]);
+	qes[1] = _mm_load_si128((__m128i *)&cache_line_base[2]);
+	qes[0] = _mm_load_si128((__m128i *)&cache_line_base[0]);
+
+	/* Evict the cache line ASAP */
+	dlb2_cldemote(cache_line_base);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[qm_port->gen_bit][*offset];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask[*offset];
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline int16_t
+dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
+		struct dlb2_eventdev_port *ev_port,
+		struct rte_event *events,
+		uint16_t max_num,
+		uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t offset;
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe(qm_port, qes, &offset);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							     qm_port,
+							     &events[num],
+							     &qes[offset]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							qm_port,
+							&events[num],
+							&qes[offset],
+							num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+
+		dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static uint16_t
+dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
+			 uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue(void *event_port, struct rte_event *ev, uint64_t wait)
+{
+	return dlb2_event_dequeue_burst(event_port, ev, 1, wait);
+}
+
+static uint16_t
+dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
+				uint16_t num, uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
+			  uint64_t wait)
+{
+	return dlb2_event_dequeue_burst_sparse(event_port, ev, 1, wait);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
+	struct dlb2_eventdev *dlb2;
+
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
@@ -2691,6 +3443,15 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	dev->enqueue_burst = dlb2_event_enqueue_burst;
 	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
 	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
+
+	dlb2 = dev->data->dev_private;
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE) {
+		dev->dequeue = dlb2_event_dequeue_sparse;
+		dev->dequeue_burst = dlb2_event_dequeue_burst_sparse;
+	} else {
+		dev->dequeue = dlb2_event_dequeue;
+		dev->dequeue_burst = dlb2_event_dequeue_burst;
+	}
 }
 
 int
-- 
2.6.4

[dpdk-dev] [PATCH v4 19/23] event/dlb2: add eventdev stop and close

[Timothy McDaniel]

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 257 +++++++++++++++++++++++++++--
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c |  91 ++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  52 ++++++
 5 files changed, 396 insertions(+), 16 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index bd96055..b02a70c 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -70,21 +70,6 @@ static struct rte_event_dev_info evdev_dlb2_default_info = {
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
-/*
- * DUMMY - added so that xstats path will compile/link.
- * Will be replaced by real version in a subsequent
- * patch.
- */
-uint32_t
-dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
-		     struct dlb2_eventdev_queue *queue)
-{
-	RTE_SET_USED(dlb2);
-	RTE_SET_USED(queue);
-
-	return 0;
-}
-
 static void
 dlb2_free_qe_mem(struct dlb2_port *qm_port)
 {
@@ -1877,7 +1862,6 @@ dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
 		return 0; /* Ignore and return success */
 	}
 
-	/* FIXME: How to handle unlink on a directed port? */
 	if (ev_port->qm_port.is_directed) {
 		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
 			     ev_port->id);
@@ -3412,6 +3396,245 @@ dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
 }
 
 static void
+dlb2_flush_port(struct rte_eventdev *dev, int port_id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	eventdev_stop_flush_t flush;
+	struct rte_event ev;
+	uint8_t dev_id;
+	void *arg;
+	int i;
+
+	flush = dev->dev_ops->dev_stop_flush;
+	dev_id = dev->data->dev_id;
+	arg = dev->data->dev_stop_flush_arg;
+
+	while (rte_event_dequeue_burst(dev_id, port_id, &ev, 1, 0)) {
+		if (flush)
+			flush(dev_id, ev, arg);
+
+		if (dlb2->ev_ports[port_id].qm_port.is_directed)
+			continue;
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+	}
+
+	/* Enqueue any additional outstanding releases */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = dlb2->ev_ports[port_id].outstanding_releases; i > 0; i--)
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+}
+
+static uint32_t
+dlb2_get_ldb_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_ldb_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_ldb_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_ldb_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static uint32_t
+dlb2_get_dir_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_dir_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_dir_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_dir_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	if (queue->qm_queue.is_directed)
+		return dlb2_get_dir_queue_depth(dlb2, queue);
+	else
+		return dlb2_get_ldb_queue_depth(dlb2, queue);
+}
+
+static bool
+dlb2_queue_is_empty(struct dlb2_eventdev *dlb2,
+		    struct dlb2_eventdev_queue *queue)
+{
+	return dlb2_get_queue_depth(dlb2, queue) == 0;
+}
+
+static bool
+dlb2_linked_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0)
+			continue;
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static bool
+dlb2_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static void
+dlb2_drain(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_port *ev_port = NULL;
+	uint8_t dev_id;
+	int i;
+
+	dev_id = dev->data->dev_id;
+
+	while (!dlb2_linked_queues_empty(dlb2)) {
+		/* Flush all the ev_ports, which will drain all their connected
+		 * queues.
+		 */
+		for (i = 0; i < dlb2->num_ports; i++)
+			dlb2_flush_port(dev, i);
+	}
+
+	/* The queues are empty, but there may be events left in the ports. */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_flush_port(dev, i);
+
+	/* If the domain's queues are empty, we're done. */
+	if (dlb2_queues_empty(dlb2))
+		return;
+
+	/* Else, there must be at least one unlinked load-balanced queue.
+	 * Select a load-balanced port with which to drain the unlinked
+	 * queue(s).
+	 */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		if (!ev_port->qm_port.is_directed)
+			break;
+	}
+
+	if (i == dlb2->num_ports) {
+		DLB2_LOG_ERR("internal error: no LDB ev_ports\n");
+		return;
+	}
+
+	rte_errno = 0;
+	rte_event_port_unlink(dev_id, ev_port->id, NULL, 0);
+
+	if (rte_errno) {
+		DLB2_LOG_ERR("internal error: failed to unlink ev_port %d\n",
+			     ev_port->id);
+		return;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		uint8_t qid, prio;
+		int ret;
+
+		if (dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			continue;
+
+		qid = i;
+		prio = 0;
+
+		/* Link the ev_port to the queue */
+		ret = rte_event_port_link(dev_id, ev_port->id, &qid, &prio, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to link ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+
+		/* Flush the queue */
+		while (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			dlb2_flush_port(dev, ev_port->id);
+
+		/* Drain any extant events in the ev_port. */
+		dlb2_flush_port(dev, ev_port->id);
+
+		/* Unlink the ev_port from the queue */
+		ret = rte_event_port_unlink(dev_id, ev_port->id, &qid, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to unlink ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+	}
+}
+
+static void
+dlb2_eventdev_stop(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_DBG("Internal error: already stopped\n");
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	} else if (dlb2->run_state != DLB2_RUN_STATE_STARTED) {
+		DLB2_LOG_ERR("Internal error: bad state %d for dev_stop\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPING;
+
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	dlb2_drain(dev);
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPED;
+}
+
+static int
+dlb2_eventdev_close(struct rte_eventdev *dev)
+{
+	dlb2_hw_reset_sched_domain(dev, false);
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3421,6 +3644,8 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.dev_start        = dlb2_eventdev_start,
+		.dev_stop         = dlb2_eventdev_stop,
+		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index a86191d..5471dd8 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -66,3 +66,9 @@ int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				     struct dlb2_start_domain_args *cfg);
+
+int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 7ead374..b508eb0 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -65,4 +65,10 @@ extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				struct dlb2_start_domain_args *cfg);
+
+extern int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+extern int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index c14a2f3..ae5ef2f 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5934,3 +5934,94 @@ dlb2_hw_start_domain(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_dir_queue_depth(hw, queue);
+
+	return 0;
+}
+
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+
+	dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (domain == NULL) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (queue == NULL) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index fff2e57..632c4e0 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -527,6 +527,56 @@ dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_ldb_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_ldb_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_dir_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret = 0;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_dir_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -544,6 +594,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth;
+	dlb2_iface_get_dir_queue_depth = dlb2_pf_get_dir_queue_depth;
 	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
-- 
2.6.4

[dpdk-dev] [PATCH v4 20/23] event/dlb2: add PMD's token pop public interface

[Timothy McDaniel]

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/api/doxy-api-index.md         |  1 +
 drivers/event/dlb2/dlb2.c         | 53 ++++++++++++++++++++++++----
 drivers/event/dlb2/dlb2_priv.h    |  3 ++
 drivers/event/dlb2/meson.build    |  5 ++-
 drivers/event/dlb2/rte_pmd_dlb2.c | 39 +++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2.h | 72 +++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/version.map    |  6 ++++
 7 files changed, 172 insertions(+), 7 deletions(-)
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index a9c12d1..00f13e2 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -52,6 +52,7 @@ The public API headers are grouped by topics:
   [dpaa2_cmdif]        (@ref rte_pmd_dpaa2_cmdif.h),
   [dpaa2_qdma]         (@ref rte_pmd_dpaa2_qdma.h),
   [crypto_scheduler]   (@ref rte_cryptodev_scheduler.h)
+  [dlb2]	       (@ref rte_pmd_dlb2.h)
 
 - **memory**:
   [memseg]             (@ref rte_memory.h),
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index b02a70c..b25b213 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1196,7 +1196,7 @@ dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
 	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
 
 	qm_port->dequeue_depth = dequeue_depth;
-
+	qm_port->token_pop_thresh = dequeue_depth;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -1364,6 +1364,8 @@ dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
 
 	qm_port->dequeue_depth = dequeue_depth;
 
+	/* Directed ports are auto-pop, by default. */
+	qm_port->token_pop_mode = AUTO_POP;
 	qm_port->owed_tokens = 0;
 	qm_port->issued_releases = 0;
 
@@ -2611,6 +2613,14 @@ dlb2_event_enqueue_burst(void *event_port,
 		dlb2_event_build_hcws(qm_port, &events[i], j,
 				      sched_types, queue_ids);
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
@@ -2619,6 +2629,11 @@ dlb2_event_enqueue_burst(void *event_port,
 			break;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
 	return cnt;
 }
 
@@ -3101,11 +3116,25 @@ dlb2_event_release(struct dlb2_eventdev *dlb2,
 		if (j == 0)
 			break;
 
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
 		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
 
 		cnt += j;
 	}
 
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
 sw_credit_update:
 	/* each release returns one credit */
 	if (!ev_port->outstanding_releases) {
@@ -3189,8 +3218,8 @@ dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3316,8 +3345,8 @@ dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
 	qm_port->owed_tokens += num;
 
 	if (num) {
-
-		dlb2_consume_qe_immediate(qm_port, num);
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
 
 		ev_port->outstanding_releases += num;
 
@@ -3332,6 +3361,7 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 			 uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3347,6 +3377,9 @@ dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3366,6 +3399,7 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 				uint16_t num, uint64_t wait)
 {
 	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
 	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
 	uint16_t cnt;
 
@@ -3381,6 +3415,9 @@ dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
 		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
 	}
 
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
 	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
 
 	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
@@ -3685,7 +3722,7 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 			    struct dlb2_devargs *dlb2_args)
 {
 	struct dlb2_eventdev *dlb2;
-	int err;
+	int err, i;
 
 	dlb2 = dev->data->dev_private;
 
@@ -3735,6 +3772,10 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Initialize each port's token pop mode */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
+		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
+
 	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
 
 	dlb2_iface_low_level_io_init();
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 61567a6..b73cf3f 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -12,6 +12,7 @@
 #include <rte_config.h>
 #include "dlb2_user.h"
 #include "dlb2_log.h"
+#include "rte_pmd_dlb2.h"
 
 #ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
 #define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
@@ -290,6 +291,7 @@ struct dlb2_port {
 	bool gen_bit;
 	uint16_t dir_credits;
 	uint32_t dequeue_depth;
+	enum dlb2_token_pop_mode token_pop_mode;
 	union dlb2_port_config cfg;
 	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
 	uint16_t cached_ldb_credits;
@@ -298,6 +300,7 @@ struct dlb2_port {
 	bool int_armed;
 	uint16_t owed_tokens;
 	int16_t issued_releases;
+	int16_t token_pop_thresh;
 	int cq_depth;
 	uint16_t cq_idx;
 	uint16_t cq_idx_unmasked;
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index dc16c96..b734ff6 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,3 +1,4 @@
+
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
@@ -12,7 +13,9 @@ sources = files('dlb2.c',
 		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
-		'pf/base/dlb2_resource.c'
+		'pf/base/dlb2_resource.c',
+		'rte_pmd_dlb2.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
+install_headers('rte_pmd_dlb2.h')
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
new file mode 100644
index 0000000..b09b585
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "rte_pmd_dlb2.h"
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode)
+{
+	struct dlb2_eventdev *dlb2;
+	struct rte_eventdev *dev;
+
+	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+	dev = &rte_eventdevs[dev_id];
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (mode >= NUM_TOKEN_POP_MODES)
+		return -EINVAL;
+
+	/* The event device must be configured, but not yet started */
+	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
+		return -EINVAL;
+
+	/* The token pop mode must be set before configuring the port */
+	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
+		return -EINVAL;
+
+	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
new file mode 100644
index 0000000..74399db
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+/*!
+ *  @file      rte_pmd_dlb2.h
+ *
+ *  @brief     DLB PMD-specific functions
+ */
+
+#ifndef _RTE_PMD_DLB2_H_
+#define _RTE_PMD_DLB2_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Selects the token pop mode for a DLB2 port.
+ */
+enum dlb2_token_pop_mode {
+	/* Pop the CQ tokens immediately after dequeueing. */
+	AUTO_POP,
+	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
+	 * Supported on load-balanced ports only.
+	 */
+	DELAYED_POP,
+	/* Pop the CQ tokens during next dequeue operation. */
+	DEFERRED_POP,
+
+	/* NUM_TOKEN_POP_MODES must be last */
+	NUM_TOKEN_POP_MODES
+};
+
+/*!
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Configure the token pop mode for a DLB2 port. By default, all ports use
+ * AUTO_POP. This function must be called before calling rte_event_port_setup()
+ * for the port, but after calling rte_event_dev_configure().
+ *
+ * @param dev_id
+ *    The identifier of the event device.
+ * @param port_id
+ *    The identifier of the event port.
+ * @param mode
+ *    The token pop mode.
+ *
+ * @return
+ * - 0: Success
+ * - EINVAL: Invalid dev_id, port_id, or mode
+ * - EINVAL: The DLB2 is not configured, is already running, or the port is
+ *   already setup
+ */
+
+__rte_experimental
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_DLB2_H_ */
diff --git a/drivers/event/dlb2/version.map b/drivers/event/dlb2/version.map
index 4a76d1d..b1e4dff 100644
--- a/drivers/event/dlb2/version.map
+++ b/drivers/event/dlb2/version.map
@@ -1,3 +1,9 @@
 DPDK_21 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmd_dlb2_set_token_pop_mode;
+};
-- 
2.6.4

[dpdk-dev] [PATCH v4 21/23] event/dlb2: add PMD self-tests

[Timothy McDaniel]

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 app/test/test_eventdev.c           |    7 +
 drivers/event/dlb2/dlb2.c          |    1 +
 drivers/event/dlb2/dlb2_selftest.c | 1570 ++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build     |    3 +-
 4 files changed, 1580 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c

diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
index 62019c1..bcfaa53 100644
--- a/app/test/test_eventdev.c
+++ b/app/test/test_eventdev.c
@@ -1030,6 +1030,12 @@ test_eventdev_selftest_dpaa2(void)
 	return test_eventdev_selftest_impl("event_dpaa2", "");
 }
 
+static int
+test_eventdev_selftest_dlb2(void)
+{
+	return test_eventdev_selftest_impl("dlb2_event", "");
+}
+
 REGISTER_TEST_COMMAND(eventdev_common_autotest, test_eventdev_common);
 REGISTER_TEST_COMMAND(eventdev_selftest_sw, test_eventdev_selftest_sw);
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
@@ -1037,3 +1043,4 @@ REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
 		test_eventdev_selftest_octeontx2);
 REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2, test_eventdev_selftest_dpaa2);
+REGISTER_TEST_COMMAND(eventdev_selftest_dlb2, test_eventdev_selftest_dlb2);
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index b25b213..113fedf 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3696,6 +3696,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
 		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+		.dev_selftest     = test_dlb2_eventdev,
 	};
 
 	/* Expose PMD's eventdev interface */
diff --git a/drivers/event/dlb2/dlb2_selftest.c b/drivers/event/dlb2/dlb2_selftest.c
new file mode 100644
index 0000000..418a53b
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_selftest.c
@@ -0,0 +1,1570 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <unistd.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_eventdev.h>
+#include <rte_pause.h>
+
+#include "dlb2_priv.h"
+#include "rte_pmd_dlb2.h"
+
+#define MAX_PORTS 32
+#define MAX_QIDS 32
+#define DEFAULT_NUM_SEQ_NUMS 64
+
+static struct rte_mempool *eventdev_func_mempool;
+static int evdev;
+
+struct test {
+	struct rte_mempool *mbuf_pool;
+	int nb_qids;
+};
+
+/* initialization and config */
+static inline int
+init(struct test *t, int nb_queues, int nb_ports)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	memset(t, 0, sizeof(*t));
+
+	t->mbuf_pool = eventdev_func_mempool;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = nb_queues;
+	config.nb_event_ports = nb_ports;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0)
+		printf("%d: Error configuring device\n", __LINE__);
+
+	return ret;
+}
+
+static inline int
+create_ports(int num_ports)
+{
+	int i;
+
+	if (num_ports > MAX_PORTS)
+		return -1;
+
+	for (i = 0; i < num_ports; i++) {
+		struct rte_event_port_conf conf;
+
+		if (rte_event_port_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default port conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		if (rte_event_port_setup(evdev, i, &conf) < 0) {
+			printf("%d: Error setting up port %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+create_lb_qids(struct test *t, int num_qids, uint32_t flags)
+{
+	int i;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		struct rte_event_queue_conf conf;
+
+		if (rte_event_queue_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default queue conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		conf.schedule_type = flags;
+
+		if (conf.schedule_type == RTE_SCHED_TYPE_PARALLEL)
+			conf.nb_atomic_order_sequences = 0;
+		else
+			conf.nb_atomic_order_sequences = DEFAULT_NUM_SEQ_NUMS;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+static inline int
+create_atomic_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ATOMIC);
+}
+
+static inline int
+create_ordered_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ORDERED);
+}
+
+static inline int
+create_unordered_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_PARALLEL);
+}
+
+static inline int
+create_directed_qids(struct test *t, int num_qids, const uint8_t ports[])
+{
+	static struct rte_event_queue_conf conf;
+	int i;
+
+	conf.priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
+	conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		uint8_t queue_id;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, ports[i - t->nb_qids],
+					&queue_id, NULL, 1) != 1) {
+			printf("%d: error creating link for qid %d\n",
+			       __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+/* destruction */
+static inline void
+cleanup(void)
+{
+	int ret = 0;
+
+	rte_event_dev_stop(evdev);
+	ret = rte_event_dev_close(evdev);
+
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+};
+
+static inline int
+enqueue_timeout(uint8_t port_id, struct rte_event *ev, uint64_t tmo_us)
+{
+	const uint64_t start = rte_get_timer_cycles();
+	const uint64_t ticks = (tmo_us * rte_get_timer_hz()) / 1E6;
+
+	while ((rte_get_timer_cycles() - start) < ticks) {
+		if (rte_event_enqueue_burst(evdev, port_id, ev, 1) == 1)
+			return 0;
+
+		if (rte_errno != -ENOSPC) {
+			printf("enqueue_burst returned rte_errno %d\n",
+			       rte_errno);
+			return -1;
+		}
+	}
+	printf("%s time out\n", __func__);
+	return -1;
+}
+
+static void
+flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused)
+{
+	rte_pktmbuf_free(event.mbuf);
+}
+
+static int
+test_stop_flush(struct test *t) /* test to check we can properly flush events */
+{
+	struct rte_event ev;
+	uint32_t dequeue_depth;
+	unsigned int i, count;
+	uint8_t queue_id;
+
+	ev.op = RTE_EVENT_OP_NEW;
+
+	if (init(t, 2, 1) < 0 ||
+	    create_ports(1) < 0 ||
+	    create_atomic_qids(t, 2) < 0) {
+		printf("%d: Error initializing device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_port_link(evdev, 0, NULL, NULL, 0) != 2) {
+		printf("%d: Error linking queues to the port\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: Error with start call\n", __LINE__);
+		goto err;
+	}
+
+	/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked
+	 * queue.
+	 */
+	queue_id = 1;
+
+	if (rte_event_port_unlink(evdev, 0, &queue_id, 1) != 1) {
+		printf("%d: Error unlinking queue 1 from port\n", __LINE__);
+		goto err;
+	}
+
+	count = rte_mempool_avail_count(t->mbuf_pool);
+
+	if (rte_event_port_attr_get(evdev,
+				    0,
+				    RTE_EVENT_PORT_ATTR_DEQ_DEPTH,
+				    &dequeue_depth)) {
+		printf("%d: Error retrieveing dequeue depth\n", __LINE__);
+		goto err;
+	}
+
+	/* Send QEs to queue 0 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 0;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Send QEs to queue 1 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 1;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Now the DLB is scheduling events from the port to the IQ, and at
+	 * least one event should be remaining in each queue.
+	 */
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, flush, NULL)) {
+		printf("%d: Error installing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+
+	if (count != rte_mempool_avail_count(t->mbuf_pool)) {
+		printf("%d: Error executing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, NULL, NULL)) {
+		printf("%d: Error uninstalling the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	return 0;
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_single_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Create a directed port */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) == 0) {
+		printf("%d: port 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Create a load-balanced port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a directed queue */
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed queue */
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) == 0) {
+		printf("%d: queue 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a load-balanced queue */
+	queue_conf.event_queue_cfg = 0;
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to link directed and load-balanced resources */
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 0 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 1 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Link ports to queues */
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+#define NUM_LDB_PORTS 64
+#define NUM_LDB_QUEUES 32
+
+static int
+test_info_get(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	config.nb_event_ports = info.max_event_ports;
+	config.nb_event_queues = NUM_LDB_QUEUES + info.max_event_ports / 2;
+	config.nb_single_link_event_port_queues = info.max_event_ports / 2;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		goto err;
+	}
+
+	/* The DLB2 PMD only reports load-balanced ports and queues in its
+	 * info_get function. Confirm that these values don't include the
+	 * directed port or queue counts.
+	 */
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret) {
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+		return -1;
+	}
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+static int
+test_reconfiguration_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 2 LDB queues */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+			printf("%d: P0->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P1->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+			printf("%d: P1->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Configure device with 2 DIR ports and 2 DIR queues */
+	config.nb_single_link_event_port_queues = 2;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q1 */
+	queue_id = 1;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: P0->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Link P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: P1->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_queues = 5;
+	config.nb_event_ports = 5;
+	config.nb_single_link_event_port_queues = 1;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < config.nb_event_queues - 1; i++) {
+		port_conf.event_port_cfg = 0;
+		queue_conf.event_queue_cfg = 0;
+
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: P%d->Q%d link expected to succeed\n",
+			       __LINE__, i, i);
+			goto err;
+		}
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+		printf("%d: port %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+		printf("%d: queue %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	queue_id = i;
+
+	if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+		printf("%d: P%d->Q%d link expected to succeed\n",
+		       __LINE__, i, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_ports += 1;
+
+	/* Reconfigure device with 1 more load-balanced port */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Configure the new port */
+	if (rte_event_port_setup(evdev, config.nb_event_ports - 1,
+				 &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_load_balanced_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_directed_traffic(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 DIR port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (ev.queue_id != 0) {
+		printf("%d: invalid dequeued event queue ID (%d)\n",
+		       __LINE__, ev.queue_id);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_deferred_sched(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	const int num_events = 128;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 1 queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 1, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.dequeue_depth = 1;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.schedule_type = RTE_SCHED_TYPE_PARALLEL;
+	queue_conf.nb_atomic_order_sequences = 0;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 128 NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_PARALLEL;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue one event from port 0 */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue (and release) all other events from port 1. Deferred
+	 * scheduling ensures no other events are scheduled to port 0 without a
+	 * subsequent rte_event_dequeue_burst() call.
+	 */
+	for (i = 0; i < num_events - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 1, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		if (rte_event_enqueue_burst(evdev, 1, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+test_delayed_pop(void)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	int ret, i, num_events;
+	struct rte_event ev;
+	uint8_t queue_id;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DELAYED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	num_events = 2 * port_conf.dequeue_depth;
+
+	/* Enqueue 2 * dequeue_depth NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue dequeue_depth events but only release dequeue_depth - 2.
+	 * Delayed pop won't perform the pop and no more events will be
+	 * scheduled.
+	 */
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = 0; i < port_conf.dequeue_depth - 2; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	ret = rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout);
+	if (ret != 0) {
+		printf("%d: event dequeue expected to fail (ret = %d)\n",
+		       __LINE__, ret);
+		goto err;
+	}
+
+	/* Release one more event. This will trigger the token pop, and
+	 * dequeue_depth - 1 more events will be scheduled to the device.
+	 */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: RELEASE enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 0) {
+		printf("%d: event dequeue expected to fail\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup();
+	return 0;
+
+err:
+	cleanup();
+	return -1;
+}
+
+static int
+do_selftest(void)
+{
+	struct test t;
+	int ret;
+
+	/* Only create mbuf pool once, reuse for each test run */
+	if (!eventdev_func_mempool) {
+		eventdev_func_mempool =
+			rte_pktmbuf_pool_create("EVENTDEV_DLB2_ST_POOL",
+						(1 << 12), /* 4k buffers */
+						32 /*MBUF_CACHE_SIZE*/,
+						0,
+						512, /* use very small mbufs */
+						rte_socket_id());
+		if (!eventdev_func_mempool) {
+			printf("ERROR creating mempool\n");
+			goto test_fail;
+		}
+	}
+	t.mbuf_pool = eventdev_func_mempool;
+
+	printf("*** Running Stop Flush test...\n");
+	ret = test_stop_flush(&t);
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Single Link test...\n");
+	ret = test_single_link();
+	if (ret != 0) {
+		printf("ERROR - Single Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Info Get test...\n");
+	ret = test_info_get();
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Reconfiguration Link test...\n");
+	ret = test_reconfiguration_link();
+	if (ret != 0) {
+		printf("ERROR - Reconfiguration Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Load-Balanced Traffic test...\n");
+	ret = test_load_balanced_traffic();
+	if (ret != 0) {
+		printf("ERROR - Load-Balanced Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Directed Traffic test...\n");
+	ret = test_directed_traffic();
+	if (ret != 0) {
+		printf("ERROR - Directed Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Deferred Scheduling test...\n");
+	ret = test_deferred_sched();
+	if (ret != 0) {
+		printf("ERROR - Deferred Scheduling test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Delayed Pop test...\n");
+	ret = test_delayed_pop();
+	if (ret != 0) {
+		printf("ERROR - Delayed Pop test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	return 0;
+
+test_fail:
+	return -1;
+}
+
+int
+test_dlb2_eventdev(void)
+{
+	const char *dlb2_eventdev_name = "dlb2_event";
+	uint8_t num_evdevs = rte_event_dev_count();
+	int i, ret = 0;
+	int found = 0, skipped = 0, passed = 0, failed = 0;
+	struct rte_event_dev_info info;
+
+	for (i = 0; found + skipped < num_evdevs && i < RTE_EVENT_MAX_DEVS;
+	     i++) {
+		ret = rte_event_dev_info_get(i, &info);
+		if (ret < 0)
+			continue;
+
+		/* skip non-dlb2 event devices */
+		if (strncmp(info.driver_name, dlb2_eventdev_name,
+			    sizeof(*info.driver_name)) != 0) {
+			skipped++;
+			continue;
+		}
+
+		evdev = rte_event_dev_get_dev_id(info.driver_name);
+		if (evdev < 0) {
+			printf("Could not get dev_id for eventdev with name %s, i=%d\n",
+			       info.driver_name, i);
+			skipped++;
+			continue;
+		}
+		found++;
+		printf("Running selftest on eventdev %s\n", info.driver_name);
+		ret = do_selftest();
+		if (ret == 0) {
+			passed++;
+			printf("Selftest passed for eventdev %s\n",
+			       info.driver_name);
+		} else {
+			failed++;
+			printf("Selftest failed for eventdev %s, err=%d\n",
+			       info.driver_name, ret);
+		}
+	}
+
+	printf("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n",
+	       found, skipped, passed, failed);
+	return ret;
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index b734ff6..340bc57 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -14,7 +14,8 @@ sources = files('dlb2.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c',
-		'rte_pmd_dlb2.c'
+		'rte_pmd_dlb2.c',
+		'dlb2_selftest.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci']
-- 
2.6.4

[dpdk-dev] [PATCH v4 22/23] event/dlb2: add queue and port release

[Timothy McDaniel]

DLB does not support reconfiguring individual queues
or ports on the fly. The entire device must be reconfigured.
Previously allocated port QE and memzone memory
is freed in this patch.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 drivers/event/dlb2/dlb2.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 113fedf..4b4719f 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3672,6 +3672,28 @@ dlb2_eventdev_close(struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(id);
+
+	/* This function intentionally left blank. */
+}
+
+static void
+dlb2_eventdev_port_release(void *port)
+{
+	struct dlb2_eventdev_port *ev_port = port;
+	struct dlb2_port *qm_port;
+
+	if (ev_port) {
+		qm_port = &ev_port->qm_port;
+		if (qm_port->config_state == DLB2_CONFIGURED)
+			dlb2_free_qe_mem(qm_port);
+	}
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3685,8 +3707,10 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
+		.queue_release    = dlb2_eventdev_queue_release,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_release     = dlb2_eventdev_port_release,
 		.port_link        = dlb2_eventdev_port_link,
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
-- 
2.6.4

[dpdk-dev] [PATCH v4 23/23] event/dlb2: add timeout ticks entry point

[Timothy McDaniel]

Adds the timeout ticks conversion function.

Adds announcement of availabililty for the new driver
for Intel Dynamic Load Balancer 2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
---
 doc/guides/rel_notes/release_20_11.rst |  5 +++++
 drivers/event/dlb2/dlb2.c              | 13 +++++++++++++
 2 files changed, 18 insertions(+)

diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
index d8ac359..dbce3b1 100644
--- a/doc/guides/rel_notes/release_20_11.rst
+++ b/doc/guides/rel_notes/release_20_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
+
+  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
+  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
+
 * **Added write combining store APIs.**
 
   Added ``rte_write32_wc`` and ``rte_write32_wc_relaxed`` APIs
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 4b4719f..dea61fc 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3693,6 +3693,18 @@ dlb2_eventdev_port_release(void *port)
 	}
 }
 
+static int
+dlb2_eventdev_timeout_ticks(struct rte_eventdev *dev, uint64_t ns,
+			    uint64_t *timeout_ticks)
+{
+	RTE_SET_USED(dev);
+	uint64_t cycles_per_ns = rte_get_timer_hz() / 1E9;
+
+	*timeout_ticks = ns * cycles_per_ns;
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -3715,6 +3727,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
 				    dlb2_eventdev_port_unlinks_in_progress,
+		.timeout_ticks    = dlb2_eventdev_timeout_ticks,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

[dpdk-dev] [PATCH 02/22] event/dlb2: add dynamic logging

[Timothy McDaniel]

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_log.h | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_log.h

diff --git a/drivers/event/dlb2/dlb2_log.h b/drivers/event/dlb2/dlb2_log.h
new file mode 100644
index 0000000..dc1481e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_log.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_EVDEV_LOG_H_
+#define _DLB2_EVDEV_LOG_H_
+
+extern int eventdev_dlb2_log_level;
+
+/* Dynamic logging */
+#define DLB2_LOG_IMPL(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, eventdev_dlb2_log_level, "%s" fmt "\n", \
+		__func__, ##args)
+
+#define DLB2_LOG_INFO(fmt, args...) \
+	DLB2_LOG_IMPL(INFO, fmt, ## args)
+
+#define DLB2_LOG_ERR(fmt, args...) \
+	DLB2_LOG_IMPL(ERR, fmt, ## args)
+
+/* remove debug logs at compile time unless actually debugging */
+#define DLB2_LOG_DBG(fmt, args...) \
+	RTE_LOG_DP(DEBUG, PMD, fmt, ## args)
+
+#endif /* _DLB2_EVDEV_LOG_H_ */
-- 
2.6.4

Re: [dpdk-dev] [PATCH 02/22] event/dlb2: add dynamic logging

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 02/22] event/dlb2: add dynamic logging
> 
> This commit adds base support for dynamic logging.
> The default log level is NOTICE. Dynamic logging
> is used exclusively throughout this patchset.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH 03/22] event/dlb2: add private data structures and constants

[Timothy McDaniel]

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_priv.h    | 614 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2.h |  59 ++++
 2 files changed, 673 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_priv.h
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h

diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
new file mode 100644
index 0000000..7bec835
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -0,0 +1,614 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_PRIV_H_
+#define _DLB2_PRIV_H_
+
+#include <emmintrin.h>
+#include <stdbool.h>
+
+#include <rte_eventdev.h>
+
+#include "rte_config.h"
+#include "dlb2_user.h"
+#include "dlb2_log.h"
+#include "rte_pmd_dlb2.h"
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+#define DLB2_INC_STAT(_stat, _incr_val) ((_stat) += _incr_val)
+#else
+#define DLB2_INC_STAT(_stat, _incr_val)
+#endif
+
+#define EVDEV_DLB2_NAME_PMD dlb2_event
+
+/*  command line arg strings */
+#define NUMA_NODE_ARG "numa_node"
+#define DLB2_MAX_NUM_EVENTS "max_num_events"
+#define DLB2_NUM_DIR_CREDITS "num_dir_credits"
+#define DEV_ID_ARG "dev_id"
+#define DLB2_DEFER_SCHED_ARG "defer_sched"
+#define DLB2_QID_DEPTH_THRESH_ARG "qid_depth_thresh"
+#define DLB2_COS_ARG "cos"
+
+/* Begin HW related defines and structs */
+
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_VFS 16
+#define DLB2_MAX_NUM_LDB_QUEUES 32
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS 64
+#define DLB2_MAX_NUM_DIR_QUEUES 64
+#define DLB2_MAX_NUM_FLOWS (64 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDITS (8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS (2 * 1024)
+#define DLB2_MAX_NUM_LDB_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_DIR_CREDIT_POOLS 64
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQOS_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_MAX_DEVICE_PATH 32
+#define DLB2_MIN_DEQUEUE_TIMEOUT_NS 1
+/* Note: "- 1" here to support the timeout range check in eventdev_autotest */
+#define DLB2_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1)
+#define DLB2_SW_CREDIT_BATCH_SZ 32
+#define DLB2_NUM_SN_GROUPS 2
+#define DLB2_MAX_LDB_SN_ALLOC 1024
+#define DLB2_MAX_QUEUE_DEPTH_THRESHOLD 8191
+
+/* 2048 total hist list entries and 64 total ldb ports, which
+ * makes for 2048/64 == 32 hist list entries per port. However, CQ
+ * depth must be a power of 2 and must also be >= HIST LIST entries.
+ * As a result we just limit the maximum dequeue depth to 32.
+ */
+#define DLB2_MIN_CQ_DEPTH 1
+#define DLB2_MAX_CQ_DEPTH 32
+#define DLB2_MIN_HARDWARE_CQ_DEPTH 8
+#define DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \
+	DLB2_MAX_CQ_DEPTH
+
+/*
+ * Static per queue/port provisioning values
+ */
+#define DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 64
+
+#define CQ_BASE(is_dir) ((is_dir) ? DLB2_DIR_CQ_BASE : DLB2_LDB_CQ_BASE)
+#define CQ_SIZE(is_dir) ((is_dir) ? DLB2_DIR_CQ_MAX_SIZE : \
+				    DLB2_LDB_CQ_MAX_SIZE)
+#define PP_BASE(is_dir) ((is_dir) ? DLB2_DIR_PP_BASE : DLB2_LDB_PP_BASE)
+
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+
+#define DLB2_NUM_QES_PER_CACHE_LINE 4
+
+#define DLB2_MAX_ENQUEUE_DEPTH 64
+#define DLB2_MIN_ENQUEUE_DEPTH 4
+
+#define DLB2_NAME_SIZE 64
+
+#define DLB2_1K 1024
+#define DLB2_2K (2 * DLB2_1K)
+#define DLB2_4K (4 * DLB2_1K)
+#define DLB2_16K (16 * DLB2_1K)
+#define DLB2_32K (32 * DLB2_1K)
+#define DLB2_1MB (DLB2_1K * DLB2_1K)
+#define DLB2_16MB (16 * DLB2_1MB)
+
+/* Use the upper 3 bits of the event priority to select the DLB2 priority */
+#define EV_TO_DLB2_PRIO(x) ((x) >> 5)
+#define DLB2_TO_EV_PRIO(x) ((x) << 5)
+
+enum dlb2_hw_port_types {
+	DLB2_LDB_PORT,
+	DLB2_DIR_PORT,
+	DLB2_NUM_PORT_TYPES /* Must be last */
+};
+
+enum dlb2_hw_queue_types {
+	DLB2_LDB_QUEUE,
+	DLB2_DIR_QUEUE,
+	DLB2_NUM_QUEUE_TYPES /* Must be last */
+};
+
+#define PORT_TYPE(p) ((p)->is_directed ? DLB2_DIR_PORT : DLB2_LDB_PORT)
+
+/* Do not change - must match hardware! */
+enum dlb2_hw_sched_type {
+	DLB2_SCHED_ATOMIC = 0,
+	DLB2_SCHED_UNORDERED,
+	DLB2_SCHED_ORDERED,
+	DLB2_SCHED_DIRECTED,
+	/* DLB2_NUM_HW_SCHED_TYPES must be last */
+	DLB2_NUM_HW_SCHED_TYPES
+};
+
+/* TODO - these structs  have been converted from qm - some fields may not be
+ * required
+ */
+
+struct dlb2_hw_rsrcs {
+	int32_t nb_events_limit;
+	uint32_t num_queues;		/**> Total queues (lb + dir) */
+	uint32_t num_ldb_queues;	/**> Number of available ldb queues */
+	uint32_t num_ldb_ports;         /**< Number of load balanced ports */
+	uint32_t num_dir_ports;         /**< Number of directed ports */
+	uint32_t num_ldb_credits;       /**< Number of load balanced credits */
+	uint32_t num_dir_credits;       /**< Number of directed credits */
+	uint32_t reorder_window_size;   /**< Size of reorder window */
+};
+
+struct dlb2_hw_resource_info {
+	/**> Max resources that can be provided */
+	struct dlb2_hw_rsrcs hw_rsrc_max;
+	int num_sched_domains;
+	uint32_t socket_id;
+	/**> EAL flags passed to this QM instance, allowing the application to
+	 * identify the pmd backend indicating hardware or software.
+	 */
+	const char *eal_flags;
+};
+
+enum DLB2_ENQUEUE_TYPE {
+	/**>
+	 * New : Used to inject a new packet into the QM.
+	 */
+	DLB2_ENQ_NEW,
+	/**>
+	 * Forward : Enqueues a packet, and
+	 *  - if atomic: release any lock it holds in the QM
+	 *  - if ordered: release the packet for egress re-ordering
+	 */
+	DLB2_ENQ_FWD,
+	/**>
+	 * Enqueue Drop : Release an inflight packet. Must be called with
+	 * event == NULL. Used to drop a packet.
+	 *
+	 * Note that all packets dequeued from a load-balanced port must be
+	 * released, either with DLB2_ENQ_DROP or DLB2_ENQ_FWD.
+	 */
+	DLB2_ENQ_DROP,
+
+	/* marker for array sizing etc. */
+	_DLB2_NB_ENQ_TYPES
+};
+
+/* hw-specific format - do not change */
+
+struct dlb2_event_type {
+	uint8_t major:4;
+	uint8_t unused:4;
+	uint8_t sub;
+};
+
+union dlb2_opaque_data {
+	uint16_t opaque_data;
+	struct dlb2_event_type event_type;
+};
+
+struct dlb2_msg_info {
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+};
+
+#define DLB2_NEW_CMD_BYTE 0x08
+#define DLB2_FWD_CMD_BYTE 0x0A
+#define DLB2_COMP_CMD_BYTE 0x02
+#define DLB2_POP_CMD_BYTE 0x01
+#define DLB2_NOOP_CMD_BYTE 0x00
+
+/* hw-specific format - do not change */
+struct dlb2_enqueue_qe {
+	uint64_t data;
+	/* Word 3 */
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	/* Word 4 */
+	uint16_t lock_id;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_cq_pop_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t tokens:10;
+	uint16_t rsvd2:6;
+	uint8_t meas_lat:1;
+	uint8_t rsvd1:2;
+	uint8_t no_dec:1;
+	uint8_t cmp_id:4;
+	union {
+		uint8_t cmd_byte;
+		struct {
+			uint8_t cq_token:1;
+			uint8_t qe_comp:1;
+			uint8_t qe_frag:1;
+			uint8_t qe_valid:1;
+			uint8_t rsvd3:1;
+			uint8_t error:1;
+			uint8_t rsvd:2;
+		};
+	};
+};
+
+/* hw-specific format - do not change */
+struct dlb2_dequeue_qe {
+	uint64_t data;
+	union dlb2_opaque_data u;
+	uint8_t qid;
+	uint8_t sched_type:2;
+	uint8_t priority:3;
+	uint8_t msg_type:3;
+	uint16_t flow_id:16; /* was pp_id in v1 */
+	uint8_t debug;
+	uint8_t cq_gen:1;
+	uint8_t qid_depth:2; /* 2 bits in v2 */
+	uint8_t rsvd1:2;
+	uint8_t error:1;
+	uint8_t rsvd2:2;
+};
+
+union dlb2_port_config {
+	struct dlb2_create_ldb_port_args ldb;
+	struct dlb2_create_dir_port_args dir;
+};
+
+enum DLB2_PORT_STATE {
+	PORT_CLOSED,
+	PORT_STARTED,
+	PORT_STOPPED
+};
+
+enum dlb2_configuration_state {
+	/* The resource has not been configured */
+	DLB2_NOT_CONFIGURED,
+	/* The resource was configured, but the device was stopped */
+	DLB2_PREV_CONFIGURED,
+	/* The resource is currently configured */
+	DLB2_CONFIGURED
+};
+
+struct dlb2_port {
+	uint32_t id;
+	bool is_directed;
+	bool gen_bit;
+	uint16_t dir_credits;
+	uint32_t dequeue_depth;
+	enum dlb2_token_pop_mode token_pop_mode;
+	union dlb2_port_config cfg;
+	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic builtins */
+	uint16_t cached_ldb_credits;
+	uint16_t ldb_credits;
+	uint16_t cached_dir_credits;
+	bool int_armed;
+	uint16_t owed_tokens;
+	int16_t issued_releases;
+	int16_t token_pop_thresh;
+	int cq_depth;
+	uint16_t cq_idx;
+	uint16_t cq_idx_unmasked;
+	uint16_t cq_depth_mask;
+	uint16_t gen_bit_shift;
+	enum DLB2_PORT_STATE state;
+	enum dlb2_configuration_state config_state;
+	int num_mapped_qids;
+	uint8_t *qid_mappings;
+	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
+	struct dlb2_enqueue_qe *int_arm_qe;
+	struct dlb2_cq_pop_qe *consume_qe;
+	struct dlb2_eventdev *dlb2; /* back ptr */
+	struct dlb2_eventdev_port *ev_port; /* back ptr */
+};
+
+/* Per-process per-port mmio and memory pointers */
+struct process_local_port_data {
+	uint64_t *pp_addr;
+	struct dlb2_dequeue_qe *cq_base;
+	bool mmaped;
+};
+
+struct dlb2_eventdev;
+
+struct dlb2_port_low_level_io_functions {
+	void (*pp_enqueue_four)(void *qe4,
+				void *pp_addr);
+};
+
+/* TODO - Optimize memory use and layout */
+struct dlb2_config {
+	int configured;
+	int reserved;
+	uint32_t num_ldb_credits;
+	uint32_t num_dir_credits;
+	struct dlb2_create_sched_domain_args resources;
+};
+
+enum dlb2_cos {
+	DLB2_COS_DEFAULT = -1,
+	DLB2_COS_0 = 0,
+	DLB2_COS_1,
+	DLB2_COS_2,
+	DLB2_COS_3
+};
+
+struct dlb2_hw_dev {
+	char device_name[DLB2_NAME_SIZE];
+	char device_path[DLB2_MAX_DEVICE_PATH];
+	int device_path_id;
+	struct dlb2_config cfg;
+	struct dlb2_hw_resource_info info;
+	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
+	int device_id;
+	uint32_t domain_id;
+	int domain_id_valid;
+	enum dlb2_cos cos_id;
+	rte_spinlock_t resource_lock; /* for MP support */
+}; __rte_cache_aligned
+
+/* End HW related defines and structs */
+
+/* Begin DLB2 PMD Eventdev related defines and structs */
+
+#define DLB2_MAX_NUM_QUEUES \
+	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
+
+#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS + DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
+
+/* Used for parsing dir ports/queues. */
+
+/* Note: eventdev currently is limited to 64 ports, but DLB hardware has 128
+ * directed ports/queues.
+ */
+struct dlb2_dir_resource_list {
+	int entries;
+	uint32_t id[DLB2_MAX_NUM_DIR_PORTS];
+};
+
+/** Structure to hold the queue to port link establishment attributes */
+
+struct dlb2_event_queue_link {
+	uint8_t queue_id;
+	uint8_t priority;
+	bool mapped;
+	bool valid;
+};
+
+struct dlb2_traffic_stats {
+	uint64_t rx_ok;
+	uint64_t rx_drop;
+	uint64_t rx_interrupt_wait;
+	uint64_t rx_umonitor_umwait;
+	uint64_t tx_ok;
+	uint64_t total_polls;
+	uint64_t zero_polls;
+	uint64_t tx_nospc_ldb_hw_credits;
+	uint64_t tx_nospc_dir_hw_credits;
+	uint64_t tx_nospc_inflight_max;
+	uint64_t tx_nospc_new_event_limit;
+	uint64_t tx_nospc_inflight_credits;
+};
+
+/* DLB2 HW sets the 2bit qid_depth in rx QEs based on the programmable depth
+ * threshold. The global default value in config/common_base (or rte_config.h)
+ * can be overridden on a per-qid basis using a vdev command line parameter.
+ * 3: depth > threshold
+ * 2: threshold >= depth > 3/4 threshold
+ * 1: 3/4 threshold >= depth > 1/2 threshold
+ * 0: depth <= 1/2 threshold.
+ */
+#define DLB2_QID_DEPTH_LE50 0
+#define DLB2_QID_DEPTH_GT50_LE75 1
+#define DLB2_QID_DEPTH_GT75_LE100 2
+#define DLB2_QID_DEPTH_GT100 3
+#define DLB2_NUM_QID_DEPTH_STAT_VALS 4 /* 2 bits */
+
+struct dlb2_queue_stats {
+	uint64_t enq_ok;
+	uint64_t qid_depth[DLB2_NUM_QID_DEPTH_STAT_VALS];
+};
+
+struct dlb2_port_stats {
+	struct dlb2_traffic_stats traffic;
+	uint64_t tx_op_cnt[4]; /* indexed by rte_event.op */
+	uint64_t tx_implicit_rel;
+	uint64_t tx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t tx_invalid;
+	uint64_t rx_sched_cnt[DLB2_NUM_HW_SCHED_TYPES];
+	uint64_t rx_sched_invalid;
+	struct dlb2_queue_stats queue[DLB2_MAX_NUM_QUEUES];
+};
+
+struct dlb2_eventdev_port {
+	struct dlb2_port qm_port; /* hw specific data structure */
+	struct rte_event_port_conf conf; /* user-supplied configuration */
+	uint16_t inflight_credits; /* num credits this port has right now */
+	uint16_t credit_update_quanta;
+	struct dlb2_eventdev *dlb2; /* backlink optimization */
+	struct dlb2_port_stats stats __rte_cache_aligned;
+	struct dlb2_event_queue_link link[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	int num_links;
+	uint32_t id; /* port id */
+	/* num releases yet to be completed on this port.
+	 * Only applies to load-balanced ports.
+	 */
+	uint16_t outstanding_releases;
+	uint16_t inflight_max; /* app requested max inflights for this port */
+	/* setup_done is set when the event port is setup */
+	bool setup_done;
+	/* enq_configured is set when the qm port is created */
+	bool enq_configured;
+	uint8_t implicit_release; /* release events before dequeueing */
+}  __rte_cache_aligned;
+
+struct dlb2_queue {
+	uint32_t num_qid_inflights; /* User config */
+	uint32_t num_atm_inflights; /* User config */
+	enum dlb2_configuration_state config_state;
+	int  sched_type; /* LB queue only */
+	uint32_t id;
+	bool	 is_directed;
+};
+
+struct dlb2_eventdev_queue {
+	struct dlb2_queue qm_queue;
+	struct rte_event_queue_conf conf; /* User config */
+	int depth_threshold; /* use default if 0 */
+	uint32_t id;
+	bool setup_done;
+	uint8_t num_links;
+};
+
+struct dlb2_device_stats {
+	/* Device specific */
+};
+
+enum dlb2_run_state {
+	DLB2_RUN_STATE_STOPPED = 0,
+	DLB2_RUN_STATE_STOPPING,
+	DLB2_RUN_STATE_STARTING,
+	DLB2_RUN_STATE_STARTED
+};
+
+struct dlb2_eventdev {
+	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
+	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each queue */
+	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
+	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
+	/* store num stats and offset of the stats for each port */
+	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
+	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
+	struct dlb2_get_num_resources_args hw_rsrc_query_results;
+	uint32_t xstats_count_mode_queue;
+	struct dlb2_hw_dev qm_instance; /* strictly hw related */
+	uint64_t global_dequeue_wait_ticks;
+	struct dlb2_xstats_entry *xstats;
+	struct rte_eventdev *event_dev; /* backlink to dev */
+	uint32_t xstats_count_mode_dev;
+	uint32_t xstats_count_mode_port;
+	uint32_t xstats_count;
+	uint32_t inflights; /* use __atomic builtins */
+	uint32_t new_event_limit;
+	int max_num_events_override;
+	int num_dir_credits_override;
+	volatile enum dlb2_run_state run_state;
+	uint16_t num_dir_queues; /* total num of evdev dir queues requested */
+	uint16_t num_dir_credits;
+	uint16_t num_ldb_credits;
+	uint16_t num_queues; /* total queues */
+	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested */
+	uint16_t num_ports; /* total num of evdev ports requested */
+	uint16_t num_ldb_ports; /* total num of ldb ports requested */
+	uint16_t num_dir_ports; /* total num of dir ports requested */
+	bool umwait_allowed;
+	bool global_dequeue_wait; /* Not using per dequeue wait if true */
+	bool defer_sched;
+	enum dlb2_cq_poll_modes poll_mode;
+	uint8_t revision;
+	bool configured;
+	uint16_t max_ldb_credits;
+	uint16_t max_dir_credits;
+
+	/* force hw credit pool counters into exclusive cache lines */
+
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t ldb_credit_pool __rte_cache_aligned;
+	/* use __atomic builtins */ /* shared hw cred */
+	uint32_t dir_credit_pool __rte_cache_aligned;
+
+	/* Device stats */
+	struct dlb2_device_stats stats __rte_cache_aligned;
+};
+
+/* used for collecting and passing around the dev args */
+struct dlb2_qid_depth_thresholds {
+	int val[DLB2_MAX_NUM_QUEUES];
+};
+struct dlb2_devargs {
+	int socket_id;
+	int max_num_events;
+	int num_dir_credits_override;
+	int dev_id;
+	int defer_sched;
+	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
+	enum dlb2_cos cos_id;
+};
+
+/* End Eventdev related defines and structs */
+
+/* Forwards for non-inlined functions */
+
+int dlb2_uninit(const char *name);
+
+void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
+
+int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
+
+void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
+
+int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n);
+
+int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstat_names,
+		unsigned int *ids, unsigned int size);
+
+uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+					  const char *name, unsigned int *id);
+
+int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode,
+		int16_t queue_port_id,
+		const uint32_t ids[],
+		uint32_t nb_ids);
+
+int test_dlb2_eventdev(void);
+
+int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+				const char *name,
+				struct dlb2_devargs *dlb2_args);
+
+int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+				  const char *name);
+
+uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue);
+
+/* arg parsing helper functions */
+int dlb2_parse_params(const char *params,
+		      const char *name,
+		      struct dlb2_devargs *dlb2_args);
+
+int dlb2_string_to_int(int *result, const char *str);
+#endif	/* _DLB2_PRIV_H_ */
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.h b/drivers/event/dlb2/rte_pmd_dlb2.h
new file mode 100644
index 0000000..ff914af
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.h
@@ -0,0 +1,59 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef _RTE_PMD_DLB2_H_
+#define _RTE_PMD_DLB2_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+/**
+ * Selects the token pop mode for a DLB2 port.
+ */
+enum dlb2_token_pop_mode {
+	/* Pop the CQ tokens immediately after dequeueing. */
+	AUTO_POP,
+	/* Pop CQ tokens after (dequeue_depth - 1) events are released.
+	 * Supported on load-balanced ports only.
+	 */
+	DELAYED_POP,
+	/* Pop the CQ tokens during next dequeue operation. */
+	DEFERRED_POP,
+
+	/* NUM_TOKEN_POP_MODES must be last */
+	NUM_TOKEN_POP_MODES
+};
+
+/*!
+ * Configure the token pop mode for a DLB2 port. By default, all ports use
+ * AUTO_POP. This function must be called before calling rte_event_port_setup()
+ * for the port, but after calling rte_event_dev_configure().
+ *
+ * @param dev_id
+ *    The identifier of the event device.
+ * @param port_id
+ *    The identifier of the event port.
+ * @param mode
+ *    The token pop mode.
+ *
+ * @return
+ * - 0: Success
+ * - EINVAL: Invalid dev_id, port_id, or mode
+ * - EINVAL: The DLB2 is not configured, is already running, or the port is
+ *   already setup
+ */
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PMD_DLB2_H_ */
-- 
2.6.4

Re: [dpdk-dev] [PATCH 03/22] event/dlb2: add private data structures and constants

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 03/22] event/dlb2: add private data structures and constants
> 
> The header file dlb2_priv.h is used internally by the PMD.
> It include constants, macros for device resources,
> structure definitions for hardware interfaces and
> software state, and various forward-declarations.
> The header file rte_pmd_dlb2.h will be exported in a
> subsequent patch, but is included here due to a data
> structure dependency.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  drivers/event/dlb2/dlb2_priv.h    | 614
> ++++++++++++++++++++++++++++++++++++++
>  drivers/event/dlb2/rte_pmd_dlb2.h |  59 ++++
>  2 files changed, 673 insertions(+)
>  create mode 100644 drivers/event/dlb2/dlb2_priv.h
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
> 
> diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
> new file mode 100644
> index 0000000..7bec835
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2_priv.h
> @@ -0,0 +1,614 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef _DLB2_PRIV_H_
> +#define _DLB2_PRIV_H_
> +
> +#include <emmintrin.h>
> +#include <stdbool.h>
> +
> +#include <rte_eventdev.h>
> +
> +#include "rte_config.h"

This should be included with angle brackets and immediately follow
rte_eventdev.h.

[...]

> +/* Do not change - must match hardware! */
> +enum dlb2_hw_sched_type {
> +	DLB2_SCHED_ATOMIC = 0,
> +	DLB2_SCHED_UNORDERED,
> +	DLB2_SCHED_ORDERED,
> +	DLB2_SCHED_DIRECTED,
> +	/* DLB2_NUM_HW_SCHED_TYPES must be last */
> +	DLB2_NUM_HW_SCHED_TYPES
> +};
> +
> +/* TODO - these structs  have been converted from qm - some fields may not
> be
> + * required
> + */

Leftover TODO -- still valid?

> +
> +struct dlb2_hw_rsrcs {
> +	int32_t nb_events_limit;
> +	uint32_t num_queues;		/**> Total queues (lb + dir) */

I don't believe "**>" is correct doxygen format (angle bracket's facing the wrong
direction), but since this is an internal/private header file it won't be included in
the doxygen output anyway. I'd recommend either correcting the inconsistency
in angle brackets or just dropping the doxygen style here altogether.

> +	uint32_t num_ldb_queues;	/**> Number of available ldb queues */
> +	uint32_t num_ldb_ports;         /**< Number of load balanced ports */
> +	uint32_t num_dir_ports;         /**< Number of directed ports */
> +	uint32_t num_ldb_credits;       /**< Number of load balanced credits */
> +	uint32_t num_dir_credits;       /**< Number of directed credits */
> +	uint32_t reorder_window_size;   /**< Size of reorder window */
> +};
> +
> +struct dlb2_hw_resource_info {
> +	/**> Max resources that can be provided */
> +	struct dlb2_hw_rsrcs hw_rsrc_max;
> +	int num_sched_domains;
> +	uint32_t socket_id;
> +	/**> EAL flags passed to this QM instance, allowing the application to
> +	 * identify the pmd backend indicating hardware or software.
> +	 */
> +	const char *eal_flags;

Doesn't look like eal_flags is used in any of the later patches.

[...]

> +
> +enum DLB2_PORT_STATE {
> +	PORT_CLOSED,
> +	PORT_STARTED,
> +	PORT_STOPPED
> +};

Nit: other enum names in this file are lower-case.

> +
> +enum dlb2_configuration_state {
> +	/* The resource has not been configured */
> +	DLB2_NOT_CONFIGURED,
> +	/* The resource was configured, but the device was stopped */
> +	DLB2_PREV_CONFIGURED,
> +	/* The resource is currently configured */
> +	DLB2_CONFIGURED
> +};
> +
> +struct dlb2_port {
> +	uint32_t id;
> +	bool is_directed;
> +	bool gen_bit;
> +	uint16_t dir_credits;
> +	uint32_t dequeue_depth;
> +	enum dlb2_token_pop_mode token_pop_mode;
> +	union dlb2_port_config cfg;
> +	uint32_t *credit_pool[DLB2_NUM_QUEUE_TYPES]; /* use __atomic
> builtins */
> +	uint16_t cached_ldb_credits;
> +	uint16_t ldb_credits;
> +	uint16_t cached_dir_credits;
> +	bool int_armed;
> +	uint16_t owed_tokens;
> +	int16_t issued_releases;
> +	int16_t token_pop_thresh;
> +	int cq_depth;
> +	uint16_t cq_idx;
> +	uint16_t cq_idx_unmasked;
> +	uint16_t cq_depth_mask;
> +	uint16_t gen_bit_shift;
> +	enum DLB2_PORT_STATE state;
> +	enum dlb2_configuration_state config_state;
> +	int num_mapped_qids;
> +	uint8_t *qid_mappings;
> +	struct dlb2_enqueue_qe *qe4; /* Cache line's worth of QEs (4) */
> +	struct dlb2_enqueue_qe *int_arm_qe;
> +	struct dlb2_cq_pop_qe *consume_qe;
> +	struct dlb2_eventdev *dlb2; /* back ptr */
> +	struct dlb2_eventdev_port *ev_port; /* back ptr */
> +};
> +
> +/* Per-process per-port mmio and memory pointers */
> +struct process_local_port_data {
> +	uint64_t *pp_addr;
> +	struct dlb2_dequeue_qe *cq_base;
> +	bool mmaped;
> +};
> +
> +struct dlb2_eventdev;
> +
> +struct dlb2_port_low_level_io_functions {
> +	void (*pp_enqueue_four)(void *qe4,
> +				void *pp_addr);

In the DLB PMD v4 patchset, I noticed the pp_enqueue_four indirection is dropped
in favor of directly calling the MOVDIR64 function -- can that be done here as well?

> +};
> +
> +/* TODO - Optimize memory use and layout */

Leftover TODO

> +struct dlb2_config {
> +	int configured;
> +	int reserved;
> +	uint32_t num_ldb_credits;
> +	uint32_t num_dir_credits;
> +	struct dlb2_create_sched_domain_args resources;
> +};
> +
> +enum dlb2_cos {
> +	DLB2_COS_DEFAULT = -1,
> +	DLB2_COS_0 = 0,
> +	DLB2_COS_1,
> +	DLB2_COS_2,
> +	DLB2_COS_3
> +};
> +
> +struct dlb2_hw_dev {
> +	char device_name[DLB2_NAME_SIZE];
> +	char device_path[DLB2_MAX_DEVICE_PATH];
> +	int device_path_id;
> +	struct dlb2_config cfg;
> +	struct dlb2_hw_resource_info info;
> +	void *pf_dev; /* opaque pointer to PF PMD dev (struct dlb2_dev) */
> +	int device_id;
> +	uint32_t domain_id;
> +	int domain_id_valid;
> +	enum dlb2_cos cos_id;
> +	rte_spinlock_t resource_lock; /* for MP support */
> +}; __rte_cache_aligned

Attribute needs to come before the semi-colon.

> +
> +/* End HW related defines and structs */
> +
> +/* Begin DLB2 PMD Eventdev related defines and structs */
> +
> +#define DLB2_MAX_NUM_QUEUES \
> +	(DLB2_MAX_NUM_DIR_QUEUES + DLB2_MAX_NUM_LDB_QUEUES)
> +
> +#define DLB2_MAX_NUM_PORTS (DLB2_MAX_NUM_DIR_PORTS +
> DLB2_MAX_NUM_LDB_PORTS)
> +#define DLB2_MAX_INPUT_QUEUE_DEPTH 256
> +
> +/* Used for parsing dir ports/queues. */

Does this comment refer to the following code? Seems out of place.

> +
> +/* Note: eventdev currently is limited to 64 ports, but DLB hardware has 128
> + * directed ports/queues.
> + */

Does this refer to RTE_EVENT_MAX_QUEUES_PER_DEV? If so, looks like the change
in the DLB PMD to increase that to 255 makes this comment unnecessary.

> +struct dlb2_dir_resource_list {
> +	int entries;
> +	uint32_t id[DLB2_MAX_NUM_DIR_PORTS];
> +};

Doesn't look this structure is used elsewhere in this series.

[...]

> +struct dlb2_eventdev_queue {
> +	struct dlb2_queue qm_queue;
> +	struct rte_event_queue_conf conf; /* User config */
> +	int depth_threshold; /* use default if 0 */
> +	uint32_t id;
> +	bool setup_done;
> +	uint8_t num_links;
> +};
> +
> +struct dlb2_device_stats {
> +	/* Device specific */
> +};

Any need for this, since it's empty?

> +
> +enum dlb2_run_state {
> +	DLB2_RUN_STATE_STOPPED = 0,
> +	DLB2_RUN_STATE_STOPPING,
> +	DLB2_RUN_STATE_STARTING,
> +	DLB2_RUN_STATE_STARTED
> +};
> +
> +struct dlb2_eventdev {
> +	struct dlb2_eventdev_port ev_ports[DLB2_MAX_NUM_PORTS];
> +	struct dlb2_eventdev_queue ev_queues[DLB2_MAX_NUM_QUEUES];
> +	uint8_t qm_ldb_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
> +	uint8_t qm_dir_to_ev_queue_id[DLB2_MAX_NUM_QUEUES];
> +	/* store num stats and offset of the stats for each queue */
> +	uint16_t xstats_count_per_qid[DLB2_MAX_NUM_QUEUES];
> +	uint16_t xstats_offset_for_qid[DLB2_MAX_NUM_QUEUES];
> +	/* store num stats and offset of the stats for each port */
> +	uint16_t xstats_count_per_port[DLB2_MAX_NUM_PORTS];
> +	uint16_t xstats_offset_for_port[DLB2_MAX_NUM_PORTS];
> +	struct dlb2_get_num_resources_args hw_rsrc_query_results;
> +	uint32_t xstats_count_mode_queue;
> +	struct dlb2_hw_dev qm_instance; /* strictly hw related */
> +	uint64_t global_dequeue_wait_ticks;
> +	struct dlb2_xstats_entry *xstats;
> +	struct rte_eventdev *event_dev; /* backlink to dev */
> +	uint32_t xstats_count_mode_dev;
> +	uint32_t xstats_count_mode_port;
> +	uint32_t xstats_count;
> +	uint32_t inflights; /* use __atomic builtins */
> +	uint32_t new_event_limit;
> +	int max_num_events_override;
> +	int num_dir_credits_override;
> +	volatile enum dlb2_run_state run_state;
> +	uint16_t num_dir_queues; /* total num of evdev dir queues requested
> */
> +	uint16_t num_dir_credits;
> +	uint16_t num_ldb_credits;
> +	uint16_t num_queues; /* total queues */
> +	uint16_t num_ldb_queues; /* total num of evdev ldb queues requested
> */
> +	uint16_t num_ports; /* total num of evdev ports requested */
> +	uint16_t num_ldb_ports; /* total num of ldb ports requested */
> +	uint16_t num_dir_ports; /* total num of dir ports requested */
> +	bool umwait_allowed;
> +	bool global_dequeue_wait; /* Not using per dequeue wait if true */
> +	bool defer_sched;
> +	enum dlb2_cq_poll_modes poll_mode;
> +	uint8_t revision;
> +	bool configured;
> +	uint16_t max_ldb_credits;
> +	uint16_t max_dir_credits;
> +
> +	/* force hw credit pool counters into exclusive cache lines */
> +
> +	/* use __atomic builtins */ /* shared hw cred */
> +	uint32_t ldb_credit_pool __rte_cache_aligned;
> +	/* use __atomic builtins */ /* shared hw cred */
> +	uint32_t dir_credit_pool __rte_cache_aligned;
> +
> +	/* Device stats */
> +	struct dlb2_device_stats stats __rte_cache_aligned;
> +};
> +
> +/* used for collecting and passing around the dev args */
> +struct dlb2_qid_depth_thresholds {
> +	int val[DLB2_MAX_NUM_QUEUES];
> +};

Nit: whitespace between the struct definitions, to match the rest of the file.

> +struct dlb2_devargs {
> +	int socket_id;
> +	int max_num_events;
> +	int num_dir_credits_override;
> +	int dev_id;
> +	int defer_sched;
> +	struct dlb2_qid_depth_thresholds qid_depth_thresholds;
> +	enum dlb2_cos cos_id;
> +};
> +
> +/* End Eventdev related defines and structs */
> +
> +/* Forwards for non-inlined functions */
> +
> +int dlb2_uninit(const char *name);

This function is never defined

> +
> +void dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f);
> +
> +int dlb2_xstats_init(struct dlb2_eventdev *dlb2);
> +
> +void dlb2_xstats_uninit(struct dlb2_eventdev *dlb2);
> +
> +int dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
> +		enum rte_event_dev_xstats_mode mode, uint8_t
> queue_port_id,
> +		const unsigned int ids[], uint64_t values[], unsigned int n);
> +
> +int dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
> +		enum rte_event_dev_xstats_mode mode, uint8_t
> queue_port_id,
> +		struct rte_event_dev_xstats_name *xstat_names,
> +		unsigned int *ids, unsigned int size);
> +
> +uint64_t dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
> +					  const char *name, unsigned int *id);
> +
> +int dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
> +		enum rte_event_dev_xstats_mode mode,
> +		int16_t queue_port_id,
> +		const uint32_t ids[],
> +		uint32_t nb_ids);
> +
> +int test_dlb2_eventdev(void);
> +
> +int dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
> +				const char *name,
> +				struct dlb2_devargs *dlb2_args);
> +
> +int dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
> +				  const char *name);
> +
> +uint32_t dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
> +		     struct dlb2_eventdev_queue *queue);
> +
> +/* arg parsing helper functions */
> +int dlb2_parse_params(const char *params,
> +		      const char *name,
> +		      struct dlb2_devargs *dlb2_args);
> +
> +int dlb2_string_to_int(int *result, const char *str);

Looks like this is only used in dlb2.c, is the forward declaration needed?

Thanks,
Gage

Re: [dpdk-dev] [PATCH 03/22] event/dlb2: add private data structures and constants

[Eads, Gage]

> +/*!
> + * Configure the token pop mode for a DLB2 port. By default, all ports use
> + * AUTO_POP. This function must be called before calling
> rte_event_port_setup()
> + * for the port, but after calling rte_event_dev_configure().
> + *
> + * @param dev_id
> + *    The identifier of the event device.
> + * @param port_id
> + *    The identifier of the event port.
> + * @param mode
> + *    The token pop mode.
> + *
> + * @return
> + * - 0: Success
> + * - EINVAL: Invalid dev_id, port_id, or mode
> + * - EINVAL: The DLB2 is not configured, is already running, or the port is
> + *   already setup
> + */
> +
> +int
> +rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
> +				uint8_t port_id,
> +				enum dlb2_token_pop_mode mode);

When I build the full series (plus dependencies), I get this error:

"
rte_pmd_dlb2_set_token_pop_mode is not flagged as experimental
but is listed in version map
Please add __rte_experimental to the definition of rte_pmd_dlb2_set_token_pop_mode
"
(https://doc.dpdk.org/guides/contributing/abi_policy.html#experimental-apis)

Thanks,
Gage

[dpdk-dev] [PATCH 04/22] event/dlb2: add definitions shared with LKM or shared code

[Timothy McDaniel]

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_user.h | 883 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 883 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_user.h

diff --git a/drivers/event/dlb2/dlb2_user.h b/drivers/event/dlb2/dlb2_user.h
new file mode 100644
index 0000000..1bfec7f
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_user.h
@@ -0,0 +1,883 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#define DLB2_MAX_NAME_LEN 64
+
+#include <linux/types.h>
+
+enum dlb2_error {
+	DLB2_ST_SUCCESS = 0,
+	DLB2_ST_NAME_EXISTS,
+	DLB2_ST_DOMAIN_UNAVAILABLE,
+	DLB2_ST_LDB_PORTS_UNAVAILABLE,
+	DLB2_ST_DIR_PORTS_UNAVAILABLE,
+	DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB2_ST_INVALID_DOMAIN_ID,
+	DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB2_ST_INVALID_LDB_QUEUE_ID,
+	DLB2_ST_INVALID_CQ_DEPTH,
+	DLB2_ST_INVALID_CQ_VIRT_ADDR,
+	DLB2_ST_INVALID_PORT_ID,
+	DLB2_ST_INVALID_QID,
+	DLB2_ST_INVALID_PRIORITY,
+	DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+	DLB2_ST_INVALID_DIR_QUEUE_ID,
+	DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB2_ST_DOMAIN_NOT_CONFIGURED,
+	DLB2_ST_INTERNAL_ERROR,
+	DLB2_ST_DOMAIN_IN_USE,
+	DLB2_ST_DOMAIN_NOT_FOUND,
+	DLB2_ST_QUEUE_NOT_FOUND,
+	DLB2_ST_DOMAIN_STARTED,
+	DLB2_ST_DOMAIN_NOT_STARTED,
+	DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB2_ST_DOMAIN_RESET_FAILED,
+	DLB2_ST_MBOX_ERROR,
+	DLB2_ST_INVALID_HIST_LIST_DEPTH,
+	DLB2_ST_NO_MEMORY,
+	DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB2_ST_INVALID_COS_ID,
+};
+
+static const char dlb2_error_strings[][128] = {
+	"DLB2_ST_SUCCESS",
+	"DLB2_ST_NAME_EXISTS",
+	"DLB2_ST_DOMAIN_UNAVAILABLE",
+	"DLB2_ST_LDB_PORTS_UNAVAILABLE",
+	"DLB2_ST_DIR_PORTS_UNAVAILABLE",
+	"DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+	"DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+	"DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+	"DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+	"DLB2_ST_INVALID_DOMAIN_ID",
+	"DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+	"DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+	"DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+	"DLB2_ST_INVALID_LDB_QUEUE_ID",
+	"DLB2_ST_INVALID_CQ_DEPTH",
+	"DLB2_ST_INVALID_CQ_VIRT_ADDR",
+	"DLB2_ST_INVALID_PORT_ID",
+	"DLB2_ST_INVALID_QID",
+	"DLB2_ST_INVALID_PRIORITY",
+	"DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+	"DLB2_ST_INVALID_DIR_QUEUE_ID",
+	"DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+	"DLB2_ST_DOMAIN_NOT_CONFIGURED",
+	"DLB2_ST_INTERNAL_ERROR",
+	"DLB2_ST_DOMAIN_IN_USE",
+	"DLB2_ST_DOMAIN_NOT_FOUND",
+	"DLB2_ST_QUEUE_NOT_FOUND",
+	"DLB2_ST_DOMAIN_STARTED",
+	"DLB2_ST_DOMAIN_NOT_STARTED",
+	"DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+	"DLB2_ST_DOMAIN_RESET_FAILED",
+	"DLB2_ST_MBOX_ERROR",
+	"DLB2_ST_INVALID_HIST_LIST_DEPTH",
+	"DLB2_ST_NO_MEMORY",
+	"DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+	"DLB2_ST_INVALID_COS_ID",
+};
+
+struct dlb2_cmd_response {
+	__u32 status; /* Interpret using enum dlb2_error */
+	__u32 id;
+};
+
+/********************************/
+/* 'dlb2' device file commands */
+/********************************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+	DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	This ioctl interface is the same in all driver versions and is always
+ *	the first ioctl.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	from any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+	/* Output parameters */
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ *	load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ *		percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ *	load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 cos_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ *   ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 group;
+	__u32 padding0;
+};
+
+enum dlb2_cq_poll_modes {
+	DLB2_CQ_POLL_MODE_STD,
+	DLB2_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB2_CQ_POLL_MODE must be last */
+	NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+enum dlb2_user_interface_commands {
+	DLB2_CMD_GET_DEVICE_VERSION,
+	DLB2_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_CMD_GET_NUM_RESOURCES,
+	DLB2_CMD_SET_SN_ALLOCATION,
+	DLB2_CMD_GET_SN_ALLOCATION,
+	DLB2_CMD_SET_COS_BW,
+	DLB2_CMD_GET_COS_BW,
+	DLB2_CMD_GET_SN_OCCUPANCY,
+	DLB2_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_DLB2_CMD must be last */
+	NUM_DLB2_CMD,
+};
+
+/*******************************/
+/* 'domain' device file alerts */
+/*******************************/
+
+/*
+ * Scheduling domain device files can be read to receive domain-specific
+ * notifications, for alerts such as hardware errors or device reset.
+ *
+ * Each alert is encoded in a 16B message. The first 8B contains the alert ID,
+ * and the second 8B is optional and contains additional information.
+ * Applications should cast read data to a struct dlb2_domain_alert, and
+ * interpret the struct's alert_id according to dlb2_domain_alert_id. The read
+ * length must be 16B, or the function will return -EINVAL.
+ *
+ * Reads are destructive, and in the case of multiple file descriptors for the
+ * same domain device file, an alert will be read by only one of the file
+ * descriptors.
+ *
+ * The driver stores alerts in a fixed-size alert ring until they are read. If
+ * the alert ring fills completely, subsequent alerts will be dropped. It is
+ * recommended that DLB2 applications dedicate a thread to perform blocking
+ * reads on the device file.
+ */
+enum dlb2_domain_alert_id {
+	/*
+	 * Software issued an illegal enqueue for a port in this domain. An
+	 * illegal enqueue could be:
+	 * - Illegal (excess) completion
+	 * - Illegal fragment
+	 * - Insufficient credits
+	 * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+	 * contains a flag indicating whether the port is load-balanced (1) or
+	 * directed (0).
+	 */
+	DLB2_DOMAIN_ALERT_PP_ILLEGAL_ENQ,
+	/*
+	 * Software issued excess CQ token pops for a port in this domain.
+	 * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+	 * contains a flag indicating whether the port is load-balanced (1) or
+	 * directed (0).
+	 */
+	DLB2_DOMAIN_ALERT_PP_EXCESS_TOKEN_POPS,
+	/*
+	 * A enqueue contained either an invalid command encoding or a REL,
+	 * REL_T, RLS, FWD, FWD_T, FRAG, or FRAG_T from a directed port.
+	 *
+	 * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+	 * contains a flag indicating whether the port is load-balanced (1) or
+	 * directed (0).
+	 */
+	DLB2_DOMAIN_ALERT_ILLEGAL_HCW,
+	/*
+	 * The QID must be valid and less than 128.
+	 *
+	 * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+	 * contains a flag indicating whether the port is load-balanced (1) or
+	 * directed (0).
+	 */
+	DLB2_DOMAIN_ALERT_ILLEGAL_QID,
+	/*
+	 * An enqueue went to a disabled QID.
+	 *
+	 * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+	 * contains a flag indicating whether the port is load-balanced (1) or
+	 * directed (0).
+	 */
+	DLB2_DOMAIN_ALERT_DISABLED_QID,
+	/*
+	 * The device containing this domain was reset. All applications using
+	 * the device need to exit for the driver to complete the reset
+	 * procedure.
+	 *
+	 * aux_alert_data doesn't contain any information for this alert.
+	 */
+	DLB2_DOMAIN_ALERT_DEVICE_RESET,
+	/*
+	 * User-space has enqueued an alert.
+	 *
+	 * aux_alert_data contains user-provided data.
+	 */
+	DLB2_DOMAIN_ALERT_USER,
+	/*
+	 * The watchdog timer fired for the specified port. This occurs if its
+	 * CQ was not serviced for a large amount of time, likely indicating a
+	 * hung thread.
+	 * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+	 * contains a flag indicating whether the port is load-balanced (1) or
+	 * directed (0).
+	 */
+	DLB2_DOMAIN_ALERT_CQ_WATCHDOG_TIMEOUT,
+
+	/* Number of DLB2 domain alerts */
+	NUM_DLB2_DOMAIN_ALERTS
+};
+
+static const char dlb2_domain_alert_strings[][128] = {
+	"DLB2_DOMAIN_ALERT_PP_ILLEGAL_ENQ",
+	"DLB2_DOMAIN_ALERT_PP_EXCESS_TOKEN_POPS",
+	"DLB2_DOMAIN_ALERT_ILLEGAL_HCW",
+	"DLB2_DOMAIN_ALERT_ILLEGAL_QID",
+	"DLB2_DOMAIN_ALERT_DISABLED_QID",
+	"DLB2_DOMAIN_ALERT_DEVICE_RESET",
+	"DLB2_DOMAIN_ALERT_USER",
+	"DLB2_DOMAIN_ALERT_CQ_WATCHDOG_TIMEOUT",
+};
+
+struct dlb2_domain_alert {
+	__u64 alert_id;
+	__u64 aux_alert_data;
+};
+
+/*********************************/
+/* 'domain' device file commands */
+/*********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_start_domain_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ *	port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENQUEUE_DOMAIN_ALERT: Enqueue a domain alert that will be
+ *	read by one reader thread.
+ *
+ * Input parameters:
+ * - aux_alert_data: user-defined auxiliary data.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enqueue_domain_alert_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u64 aux_alert_data;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_LDB_PORT_PP_FD: Get file descriptor to mmap a load-balanced
+ *	port's producer port (PP).
+ * DLB2_CMD_GET_LDB_PORT_CQ_FD: Get file descriptor to mmap a load-balanced
+ *	port's consumer queue (CQ).
+ *
+ *	The load-balanced port must have been previously created with the ioctl
+ *	DLB2_CMD_CREATE_LDB_PORT. The fd is used to mmap the PP/CQ region.
+ *
+ * DLB2_CMD_GET_DIR_PORT_PP_FD: Get file descriptor to mmap a directed port's
+ *	producer port (PP).
+ * DLB2_CMD_GET_DIR_PORT_CQ_FD: Get file descriptor to mmap a directed port's
+ *	consumer queue (CQ).
+ *
+ *	The directed port must have been previously created with the ioctl
+ *	DLB2_CMD_CREATE_DIR_PORT. The fd is used to mmap PP/CQ region.
+ *
+ * Input parameters:
+ * - port_id: port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * - response.id: fd.
+ */
+struct dlb2_get_port_fd_args {
+	/* Output parameters */
+	struct dlb2_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
+enum dlb2_domain_user_interface_commands {
+	DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE,
+	DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE,
+	DLB2_DOMAIN_CMD_CREATE_LDB_PORT,
+	DLB2_DOMAIN_CMD_CREATE_DIR_PORT,
+	DLB2_DOMAIN_CMD_START_DOMAIN,
+	DLB2_DOMAIN_CMD_MAP_QID,
+	DLB2_DOMAIN_CMD_UNMAP_QID,
+	DLB2_DOMAIN_CMD_ENABLE_LDB_PORT,
+	DLB2_DOMAIN_CMD_ENABLE_DIR_PORT,
+	DLB2_DOMAIN_CMD_DISABLE_LDB_PORT,
+	DLB2_DOMAIN_CMD_DISABLE_DIR_PORT,
+	DLB2_DOMAIN_CMD_BLOCK_ON_CQ_INTERRUPT,
+	DLB2_DOMAIN_CMD_ENQUEUE_DOMAIN_ALERT,
+	DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS,
+	DLB2_DOMAIN_CMD_GET_LDB_PORT_PP_FD,
+	DLB2_DOMAIN_CMD_GET_LDB_PORT_CQ_FD,
+	DLB2_DOMAIN_CMD_GET_DIR_PORT_PP_FD,
+	DLB2_DOMAIN_CMD_GET_DIR_PORT_CQ_FD,
+
+	/* NUM_DLB2_DOMAIN_CMD must be last */
+	NUM_DLB2_DOMAIN_CMD,
+};
+
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
+#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
+
+
+#endif /* __DLB2_USER_H */
-- 
2.6.4

Re: [dpdk-dev] [PATCH 04/22] event/dlb2: add definitions shared with LKM or shared code

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 04/22] event/dlb2: add definitions shared with LKM or shared
> code
> 
> Add headers containing structs and constants shared between
> the PMD and the shared code.  The term shared code refers to
> the code that implements the hardware interface. The shared code
> is introduced in the probe patch, and then is extended as
> additional eventdev PMD entry points are added to the patchset.
> In the case of the bifurcated PMD (to be introduced in the
> future), the shared code is contained in the Linux kernel
> module itself.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

The checkpatch warnings are codespell false positives; looks good to me.

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH 05/22] event/dlb2: add inline functions

[Timothy McDaniel]

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2_inline_fns.h | 85 ++++++++++++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)
 create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h

diff --git a/drivers/event/dlb2/dlb2_inline_fns.h b/drivers/event/dlb2/dlb2_inline_fns.h
new file mode 100644
index 0000000..f2f0935
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_inline_fns.h
@@ -0,0 +1,85 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_INLINE_FNS_H_
+#define _DLB2_INLINE_FNS_H_
+
+/* Inline functions required in more than one source file.
+ */
+
+static inline struct dlb2_eventdev *
+dlb2_pmd_priv(const struct rte_eventdev *eventdev)
+{
+	return eventdev->data->dev_private;
+}
+
+static inline void
+dlb2_umonitor(volatile void *addr)
+{
+	/* TODO: Change to proper assembly when compiler support available */
+	asm volatile(".byte 0xf3, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (addr));
+}
+
+static inline void
+dlb2_umwait(int state, uint64_t timeout)
+{
+	uint32_t eax = timeout & UINT32_MAX;
+	uint32_t edx = timeout >> 32;
+
+	/* TODO: Change to proper assembly when compiler support available */
+	asm volatile(".byte 0xf2, 0x0f, 0xae, 0xf7\t\n"
+			:
+			: "D" (state),  "a" (eax), "d" (edx));
+}
+
+static inline void
+dlb2_movntdq(void *qe4, void *pp_addr)
+{
+	/* Move entire 64B cache line of QEs, 128 bits (16B) at a time. */
+	long long *_qe  = (long long *)qe4;
+	__v2di src_data0 = (__v2di){_qe[0], _qe[1]};
+	__v2di src_data1 = (__v2di){_qe[2], _qe[3]};
+	__v2di src_data2 = (__v2di){_qe[4], _qe[5]};
+	__v2di src_data3 = (__v2di){_qe[6], _qe[7]};
+
+	__builtin_ia32_movntdq((__v2di *)pp_addr + 0, (__v2di)src_data0);
+	rte_wmb();
+	__builtin_ia32_movntdq((__v2di *)pp_addr + 1, (__v2di)src_data1);
+	rte_wmb();
+	__builtin_ia32_movntdq((__v2di *)pp_addr + 2, (__v2di)src_data2);
+	rte_wmb();
+	__builtin_ia32_movntdq((__v2di *)pp_addr + 3, (__v2di)src_data3);
+	rte_wmb();
+}
+
+static inline void
+dlb2_movntdq_single(void *qe4, void *pp_addr)
+{
+	long long *_qe  = (long long *)qe4;
+	__v2di src_data0 = (__v2di){_qe[0], _qe[1]};
+
+	__builtin_ia32_movntdq((__v2di *)pp_addr, (__v2di)src_data0);
+}
+
+static inline void
+dlb2_cldemote(void *addr)
+{
+	/* Load addr into RSI, then demote the cache line of the address
+	 * contained in that register.
+	 */
+	asm volatile(".byte 0x0f, 0x1c, 0x06" :: "S" (addr));
+}
+
+static inline void
+dlb2_movdir64b(void *qe4, void *pp_addr)
+{
+	/* TODO: Change to proper assembly when compiler support available */
+	asm volatile(".byte 0x66, 0x0f, 0x38, 0xf8, 0x02"
+		     :
+		     : "a" (pp_addr), "d" (qe4));
+}
+
+#endif /* _DLB2_INLINE_FNS_H_ */
-- 
2.6.4

Re: [dpdk-dev] [PATCH 05/22] event/dlb2: add inline functions

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 05/22] event/dlb2: add inline functions
> 
> Add miscellaneous inline functions that may be called
> from multiple files.  These functions include inline
> assembly of new x86 instructions, such as movdir64b,
> since they are not available as builtin functions in
> the minimum supported GCC version.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  drivers/event/dlb2/dlb2_inline_fns.h | 85
> ++++++++++++++++++++++++++++++++++++
>  1 file changed, 85 insertions(+)
>  create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
> 
> diff --git a/drivers/event/dlb2/dlb2_inline_fns.h
> b/drivers/event/dlb2/dlb2_inline_fns.h
> new file mode 100644
> index 0000000..f2f0935
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2_inline_fns.h
> @@ -0,0 +1,85 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef _DLB2_INLINE_FNS_H_
> +#define _DLB2_INLINE_FNS_H_
> +
> +/* Inline functions required in more than one source file.
> + */

Nit: this comment can be a one-liner.

> +
> +static inline struct dlb2_eventdev *
> +dlb2_pmd_priv(const struct rte_eventdev *eventdev)
> +{
> +	return eventdev->data->dev_private;
> +}
> +
> +static inline void
> +dlb2_umonitor(volatile void *addr)
> +{
> +	/* TODO: Change to proper assembly when compiler support available */

I would drop/reword the 'TODO', here and elsewhere in the file, so it doesn't give the
wrong impression that this code is incomplete.

> +	asm volatile(".byte 0xf3, 0x0f, 0xae, 0xf7\t\n"
> +			:
> +			: "D" (addr));
> +}

Thanks,
Gage

[dpdk-dev] [PATCH 06/22] event/dlb2: add probe

[Timothy McDaniel]

The DLB2 hardware is a PCI device. This commit adds
support for probe and other initialization. The
dlb2_iface.[ch] files implement a flexible interface
that supports both the PF PMD and the bifurcated PMD.
The bifurcated PMD will be released in a future
patch set. Note that the flexible interface is only
used for configuration, and is not used in the data
path. The shared code is added in pf/base.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                      |  557 ++++++
 drivers/event/dlb2/dlb2_iface.c                |   42 +
 drivers/event/dlb2/dlb2_iface.h                |   29 +
 drivers/event/dlb2/dlb2_priv.h                 |    5 +
 drivers/event/dlb2/meson.build                 |    6 +-
 drivers/event/dlb2/pf/base/dlb2_hw_types.h     |  367 ++++
 drivers/event/dlb2/pf/base/dlb2_mbox.h         |  596 ++++++
 drivers/event/dlb2/pf/base/dlb2_osdep.h        |  248 +++
 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h |  447 +++++
 drivers/event/dlb2/pf/base/dlb2_osdep_list.h   |  131 ++
 drivers/event/dlb2/pf/base/dlb2_osdep_types.h  |   31 +
 drivers/event/dlb2/pf/base/dlb2_regs.h         | 2527 ++++++++++++++++++++++++
 drivers/event/dlb2/pf/base/dlb2_resource.c     |  274 +++
 drivers/event/dlb2/pf/base/dlb2_resource.h     | 1913 ++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c              |  620 ++++++
 drivers/event/dlb2/pf/dlb2_main.h              |  107 +
 drivers/event/dlb2/pf/dlb2_pf.c                |  251 +++
 17 files changed, 8150 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.c
 create mode 100644 drivers/event/dlb2/dlb2_iface.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
 create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
 create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
 create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
new file mode 100644
index 0000000..7ff7dac
--- /dev/null
+++ b/drivers/event/dlb2/dlb2.c
@@ -0,0 +1,557 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <assert.h>
+#include <errno.h>
+#include <nmmintrin.h>
+#include <pthread.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+
+#include <rte_common.h>
+#include <rte_config.h>
+#include <rte_cycles.h>
+#include <rte_debug.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_kvargs.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_ring.h>
+#include <rte_string_fns.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_iface.h"
+#include "dlb2_inline_fns.h"
+
+#if !defined RTE_ARCH_X86_64
+#error "This implementation only supports RTE_ARCH_X86_64 architecture."
+#endif
+
+/*
+ * Resources exposed to eventdev. Some values overridden at runtime using
+ * values returned by the DLB kernel driver.
+ */
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
+#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues"
+#endif
+static struct rte_event_dev_info evdev_dlb2_default_info = {
+	.driver_name = "", /* probe will set */
+	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
+	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
+#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
+	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
+#else
+	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
+#endif
+	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
+	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_priority_levels = DLB2_QID_PRIORITIES,
+	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
+	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
+	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
+	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
+	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+	.max_single_link_event_port_queue_pairs = DLB2_MAX_NUM_DIR_PORTS,
+	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
+			  RTE_EVENT_DEV_CAP_EVENT_QOS |
+			  RTE_EVENT_DEV_CAP_BURST_MODE |
+			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
+			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
+			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
+};
+
+/* These functions will vary based on processor capabilities */
+static struct dlb2_port_low_level_io_functions qm_mmio_fns;
+
+struct process_local_port_data
+dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
+
+/* override defaults with value(s) provided on command line */
+static void
+dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
+				 int *qid_depth_thresholds)
+{
+	int q;
+
+	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
+		if (qid_depth_thresholds[q] != 0)
+			dlb2->ev_queues[q].depth_threshold =
+				qid_depth_thresholds[q];
+	}
+}
+
+static int
+dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
+	int ret;
+
+	/* Query driver resources provisioned for this VF */
+
+	ret = dlb2_iface_get_num_resources(handle,
+					   &dlb2->hw_rsrc_query_results);
+	if (ret) {
+		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n",
+			     ret);
+		return ret;
+	}
+
+	/* Complete filling in device resource info returned to evdev app,
+	 * overriding any default values.
+	 * The capabilities (CAPs) were set at compile time.
+	 */
+
+	evdev_dlb2_default_info.max_event_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	evdev_dlb2_default_info.max_event_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	evdev_dlb2_default_info.max_num_events =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	/* Save off values used when creating the scheduling domain. */
+
+	handle->info.num_sched_domains =
+		dlb2->hw_rsrc_query_results.num_sched_domains;
+
+	handle->info.hw_rsrc_max.nb_events_limit =
+		dlb2->hw_rsrc_query_results.num_ldb_credits;
+
+	handle->info.hw_rsrc_max.num_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues +
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.num_ldb_queues =
+		dlb2->hw_rsrc_query_results.num_ldb_queues;
+
+	handle->info.hw_rsrc_max.num_ldb_ports =
+		dlb2->hw_rsrc_query_results.num_ldb_ports;
+
+	handle->info.hw_rsrc_max.num_dir_ports =
+		dlb2->hw_rsrc_query_results.num_dir_ports;
+
+	handle->info.hw_rsrc_max.reorder_window_size =
+		dlb2->hw_rsrc_query_results.num_hist_list_entries;
+
+	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max, sizeof(*dlb2_info));
+
+	return 0;
+}
+
+static void
+dlb2_qm_mmio_fn_init(void)
+{
+	/* Process-local function pointers for performing low level port i/o */
+
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIR64B))
+		qm_mmio_fns.pp_enqueue_four = dlb2_movdir64b;
+	else
+		qm_mmio_fns.pp_enqueue_four = dlb2_movntdq;
+}
+
+#define RTE_BASE_10 10
+
+int dlb2_string_to_int(int *result, const char *str)
+{
+	long ret;
+
+	if (str == NULL || result == NULL)
+		return -EINVAL;
+
+	errno = 0;
+	ret = strtol(str, NULL, RTE_BASE_10);
+	if (errno)
+		return -errno;
+
+	/* long int and int may be different width for some architectures */
+	if (ret < INT_MIN || ret > INT_MAX)
+		return -EINVAL;
+
+	*result = ret;
+	return 0;
+}
+
+static int
+set_numa_node(const char *key __rte_unused, const char *value, void *opaque)
+{
+	int *socket_id = opaque;
+	int ret;
+
+	ret = dlb2_string_to_int(socket_id, value);
+	if (ret < 0)
+		return ret;
+
+	if (*socket_id > RTE_MAX_NUMA_NODES)
+		return -EINVAL;
+	return 0;
+}
+
+static int
+set_max_num_events(const char *key __rte_unused,
+		   const char *value,
+		   void *opaque)
+{
+	int *max_num_events = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(max_num_events, value);
+	if (ret < 0)
+		return ret;
+
+
+	if (*max_num_events < 0 || *max_num_events >
+			DLB2_MAX_NUM_LDB_CREDITS) {
+		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and %d\n",
+			     DLB2_MAX_NUM_LDB_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_num_dir_credits(const char *key __rte_unused,
+		    const char *value,
+		    void *opaque)
+{
+	int *num_dir_credits = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(num_dir_credits, value);
+	if (ret < 0)
+		return ret;
+
+	if (*num_dir_credits < 0 ||
+	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
+		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and %d\n",
+			     DLB2_MAX_NUM_DIR_CREDITS);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int
+set_dev_id(const char *key __rte_unused,
+	   const char *value,
+	   void *opaque)
+{
+	int *dev_id = opaque;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(dev_id, value);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int
+set_cos(const char *key __rte_unused,
+	const char *value,
+	void *opaque)
+{
+	enum dlb2_cos *cos_id = opaque;
+	int x = 0;
+	int ret;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	ret = dlb2_string_to_int(&x, value);
+	if (ret < 0)
+		return ret;
+
+	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
+		DLB2_LOG_ERR("COS %d out of range, must be 0-3\n", x);
+		return -EINVAL;
+	}
+
+	*cos_id = x;
+
+	return 0;
+}
+
+
+static int
+set_qid_depth_thresh(const char *key __rte_unused,
+		     const char *value,
+		     void *opaque)
+{
+	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
+	int first, last, thresh, i;
+
+	if (value == NULL || opaque == NULL) {
+		DLB2_LOG_ERR("NULL pointer\n");
+		return -EINVAL;
+	}
+
+	/* command line override may take one of the following 3 forms:
+	 * qid_depth_thresh=all:<threshold_value> ... all queues
+	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
+	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
+	 */
+	if (sscanf(value, "all:%d", &thresh) == 1) {
+		first = 0;
+		last = DLB2_MAX_NUM_QUEUES - 1;
+	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
+		/* we have everything we need */
+	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
+		last = first;
+	} else {
+		DLB2_LOG_ERR("Error parsing qid depth vdev arg. Should be all:val, qid-qid:val, or qid:val\n");
+		return -EINVAL;
+	}
+
+	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
+		DLB2_LOG_ERR("Error parsing qid depth vdev arg, invalid qid value\n");
+		return -EINVAL;
+	}
+
+	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
+		DLB2_LOG_ERR("Error parsing qid depth vdev arg, threshold > %d\n",
+			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
+		return -EINVAL;
+	}
+
+	for (i = first; i <= last; i++)
+		qid_thresh->val[i] = thresh; /* indexed by qid */
+
+	return 0;
+}
+
+static void
+dlb2_entry_points_init(struct rte_eventdev *dev)
+{
+	RTE_SET_USED(dev);
+
+	/* Eventdev PMD entry points */
+}
+
+int
+dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
+			    const char *name,
+			    struct dlb2_devargs *dlb2_args)
+{
+	struct dlb2_eventdev *dlb2;
+	int err, i;
+
+	dlb2 = dev->data->dev_private;
+
+	dlb2->event_dev = dev; /* backlink */
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	dlb2->max_num_events_override = dlb2_args->max_num_events;
+	dlb2->num_dir_credits_override = dlb2_args->num_dir_credits_override;
+	dlb2->qm_instance.device_path_id = dlb2_args->dev_id;
+	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
+
+	/* Open the interface.
+	 * For vdev mode, this means open the dlb2 kernel module.
+	 */
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
+					    &dlb2->revision);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the device version, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_iface_hardware_init(&dlb2->qm_instance);
+
+	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2->poll_mode);
+	if (err < 0) {
+		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
+			     err);
+		return err;
+	}
+
+	/* Initialize each port's token pop mode */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
+		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
+
+	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
+
+	dlb2_qm_mmio_fn_init();
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	dlb2_init_queue_depth_thresholds(dlb2,
+					 dlb2_args->qid_depth_thresholds.val);
+
+	return 0;
+}
+
+int
+dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
+			      const char *name)
+{
+	struct dlb2_eventdev *dlb2;
+	int err;
+
+	dlb2 = dev->data->dev_private;
+
+	evdev_dlb2_default_info.driver_name = name;
+
+	err = dlb2_iface_open(&dlb2->qm_instance, name);
+	if (err < 0) {
+		DLB2_LOG_ERR("could not open event hardware device, err=%d\n",
+			     err);
+		return err;
+	}
+
+	err = dlb2_hw_query_resources(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("get resources err=%d for %s\n",
+			     err, name);
+		return err;
+	}
+
+	dlb2_qm_mmio_fn_init();
+
+	dlb2_iface_low_level_io_init();
+
+	dlb2_entry_points_init(dev);
+
+	return 0;
+}
+
+int
+dlb2_parse_params(const char *params,
+		  const char *name,
+		  struct dlb2_devargs *dlb2_args)
+{
+	int ret = 0;
+	static const char * const args[] = { NUMA_NODE_ARG,
+				      DLB2_MAX_NUM_EVENTS,
+				      DLB2_NUM_DIR_CREDITS,
+				      DEV_ID_ARG,
+				      DLB2_QID_DEPTH_THRESH_ARG,
+				      DLB2_COS_ARG,
+				      NULL };
+
+	if (params != NULL && params[0] != '\0') {
+		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
+
+		if (!kvlist) {
+			RTE_LOG(INFO, PMD,
+				"Ignoring unsupported parameters when creating device '%s'\n",
+				name);
+		} else {
+			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
+						     set_numa_node,
+						     &dlb2_args->socket_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing numa node parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_MAX_NUM_EVENTS,
+						 set_max_num_events,
+						 &dlb2_args->max_num_events);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing max_num_events parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist,
+					DLB2_NUM_DIR_CREDITS,
+					set_num_dir_credits,
+					&dlb2_args->num_dir_credits_override);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing num_dir_credits parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
+						 set_dev_id,
+						 &dlb2_args->dev_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing dev_id parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(
+					kvlist,
+					DLB2_QID_DEPTH_THRESH_ARG,
+					set_qid_depth_thresh,
+					&dlb2_args->qid_depth_thresholds);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing qid_depth_thresh parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
+						 set_cos,
+						 &dlb2_args->cos_id);
+			if (ret != 0) {
+				DLB2_LOG_ERR("%s: Error parsing cos parameter",
+					     name);
+				rte_kvargs_free(kvlist);
+				return ret;
+			}
+
+			rte_kvargs_free(kvlist);
+		}
+	}
+	return ret;
+}
+RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
+
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
new file mode 100644
index 0000000..fefdf78
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdbool.h>
+#include <stdint.h>
+#include <rte_debug.h>
+#include <rte_bus_pci.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "dlb2_priv.h"
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+void (*dlb2_iface_low_level_io_init)(void);
+
+int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+				     uint8_t *revision);
+
+void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+				   enum dlb2_cq_poll_modes *mode);
+
+int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_num_resources_args *rsrcs);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
new file mode 100644
index 0000000..4fb416e
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef _DLB2_IFACE_H_
+#define _DLB2_IFACE_H_
+
+/* DLB2 PMD Internal interface function pointers.
+ * If VDEV (bifurcated PMD),  these will resolve to functions that issue ioctls
+ * serviced by DLB kernel module.
+ * If PCI (PF PMD),  these will be implemented locally in user mode.
+ */
+
+extern void (*dlb2_iface_low_level_io_init)(void);
+
+extern int (*dlb2_iface_open)(struct dlb2_hw_dev *handle, const char *name);
+
+extern int (*dlb2_iface_get_device_version)(struct dlb2_hw_dev *handle,
+					    uint8_t *revision);
+
+extern void (*dlb2_iface_hardware_init)(struct dlb2_hw_dev *handle);
+
+extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
+					  enum dlb2_cq_poll_modes *mode);
+
+extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_num_resources_args *rsrcs);
+
+#endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 7bec835..68cab92 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -611,4 +611,9 @@ int dlb2_parse_params(const char *params,
 		      struct dlb2_devargs *dlb2_args);
 
 int dlb2_string_to_int(int *result, const char *str);
+
+/* Extern globals */
+
+extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
+
 #endif	/* _DLB2_PRIV_H_ */
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index d4fd39f..557e3b4 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,7 +1,11 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
-sources = files(
+sources = files('dlb2.c',
+		'dlb2_iface.c',
+		'pf/dlb2_main.c',
+		'pf/dlb2_pf.c',
+		'pf/base/dlb2_resource.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'bus_vdev', 'pci', 'bus_pci']
diff --git a/drivers/event/dlb2/pf/base/dlb2_hw_types.h b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
new file mode 100644
index 0000000..428a5e8
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_hw_types.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_osdep_list.h"
+#include "dlb2_osdep_types.h"
+
+#define DLB2_MAX_NUM_VDEVS			16
+#define DLB2_MAX_NUM_DOMAINS			32
+#define DLB2_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS			64
+#define DLB2_MAX_NUM_DIR_PORTS			64
+#define DLB2_MAX_NUM_LDB_CREDITS		(8 * 1024)
+#define DLB2_MAX_NUM_DIR_CREDITS		(2 * 1024)
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB2_MAX_NUM_AQED_ENTRIES		2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB2_QID_PRIORITIES			8
+#define DLB2_NUM_ARB_WEIGHTS			8
+#define DLB2_MAX_WEIGHT				255
+#define DLB2_NUM_COS_DOMAINS			4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#ifdef FPGA
+#define DLB2_HZ					2000000
+#else
+#define DLB2_HZ					800000000
+#endif
+
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
+
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS     64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+	(DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+	DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+	DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID		3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID		4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW		5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ	1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS     31
+#define DLB2_VF_BASE_CQ_VECTOR_ID	     0
+#define DLB2_VF_LAST_CQ_VECTOR_ID	     30
+#define DLB2_VF_MBOX_VECTOR_ID		     31
+#define DLB2_VF_TOTAL_NUM_INTERRUPT_VECTORS \
+	(DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+	 DLB2_VF_NUM_CQ_INTERRUPT_VECTORS)
+
+#define DLB2_VDEV_MAX_NUM_INTERRUPT_VECTORS (DLB2_MAX_NUM_LDB_PORTS + \
+					     DLB2_MAX_NUM_DIR_PORTS + 1)
+
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE   0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND  (DLB2_DRV_LDB_PP_BASE + \
+				DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE   0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND  (DLB2_DRV_DIR_PP_BASE + \
+				DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE       0x2100000
+#define DLB2_LDB_PP_STRIDE     0x1000
+#define DLB2_LDB_PP_BOUND      (DLB2_LDB_PP_BASE + \
+				DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id)   (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE       0x2000000
+#define DLB2_DIR_PP_STRIDE     0x1000
+#define DLB2_DIR_PP_BOUND      (DLB2_DIR_PP_BASE + \
+				DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id)   (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
+struct dlb2_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb2_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb2_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
+struct dlb2_ldb_queue {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB2_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB2_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB2_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB2_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+	enum dlb2_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+	struct dlb2_list_entry domain_list;
+	struct dlb2_list_entry func_list;
+	struct dlb2_resource_id id;
+	struct dlb2_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb2_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+	u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+	u32 bound[6] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & (1 << i))) {
+			group->slot_use_bitmap |= 1 << i;
+			return i;
+		}
+	}
+
+	return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+	int i, cnt = 0;
+
+	for (i = 0; i < 32; i++)
+		cnt += !!(group->slot_use_bitmap & (1 << i));
+
+	return cnt;
+}
+
+struct dlb2_hw_domain {
+	struct dlb2_function_resources *parent_func;
+	struct dlb2_list_entry func_list;
+	struct dlb2_list_head used_ldb_queues;
+	struct dlb2_list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head used_dir_pq_pairs;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb2_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb2_bitmap;
+
+struct dlb2_function_resources {
+	struct dlb2_list_head avail_domains;
+	struct dlb2_list_head used_domains;
+	struct dlb2_list_head avail_ldb_queues;
+	struct dlb2_list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	struct dlb2_list_head avail_dir_pq_pairs;
+	struct dlb2_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+	struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+	struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+	struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
+struct dlb2_mbox {
+	u32 *mbox;
+	u32 *isr_in_progress;
+};
+
+struct dlb2_sw_mbox {
+	struct dlb2_mbox vdev_to_pf;
+	struct dlb2_mbox pf_to_vdev;
+	void (*pf_to_vdev_inject)(void *arg);
+	void *pf_to_vdev_inject_arg;
+};
+
+struct dlb2_hw {
+	/* BAR 0 address */
+	void  *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void  *func_kva;
+	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb2_hw_resources rsrcs;
+	struct dlb2_function_resources pf;
+	struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+	struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+	/* Virtualization */
+	int virt_mode;
+	struct dlb2_sw_mbox mbox[DLB2_MAX_NUM_VDEVS];
+	unsigned int pasid[DLB2_MAX_NUM_VDEVS];
+};
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_mbox.h b/drivers/event/dlb2/pf/base/dlb2_mbox.h
new file mode 100644
index 0000000..ce462c0
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_mbox.h
@@ -0,0 +1,596 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_BASE_DLB2_MBOX_H
+#define __DLB2_BASE_DLB2_MBOX_H
+
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+
+#define DLB2_MBOX_INTERFACE_VERSION 1
+
+/*
+ * The PF uses its PF->VF mailbox to send responses to VF requests, as well as
+ * to send requests of its own (e.g. notifying a VF of an impending FLR).
+ * To avoid communication race conditions, e.g. the PF sends a response and then
+ * sends a request before the VF reads the response, the PF->VF mailbox is
+ * divided into two sections:
+ * - Bytes 0-47: PF responses
+ * - Bytes 48-63: PF requests
+ *
+ * Partitioning the PF->VF mailbox allows responses and requests to occupy the
+ * mailbox simultaneously.
+ */
+#define DLB2_PF2VF_RESP_BYTES	  48
+#define DLB2_PF2VF_RESP_BASE	  0
+#define DLB2_PF2VF_RESP_BASE_WORD (DLB2_PF2VF_RESP_BASE / 4)
+
+#define DLB2_PF2VF_REQ_BYTES	  16
+#define DLB2_PF2VF_REQ_BASE	  (DLB2_PF2VF_RESP_BASE + DLB2_PF2VF_RESP_BYTES)
+#define DLB2_PF2VF_REQ_BASE_WORD  (DLB2_PF2VF_REQ_BASE / 4)
+
+/*
+ * Similarly, the VF->PF mailbox is divided into two sections:
+ * - Bytes 0-239: VF requests
+ * -- (Bytes 0-3 are unused due to a hardware errata)
+ * - Bytes 240-255: VF responses
+ */
+#define DLB2_VF2PF_REQ_BYTES	 236
+#define DLB2_VF2PF_REQ_BASE	 4
+#define DLB2_VF2PF_REQ_BASE_WORD (DLB2_VF2PF_REQ_BASE / 4)
+
+#define DLB2_VF2PF_RESP_BYTES	  16
+#define DLB2_VF2PF_RESP_BASE	  (DLB2_VF2PF_REQ_BASE + DLB2_VF2PF_REQ_BYTES)
+#define DLB2_VF2PF_RESP_BASE_WORD (DLB2_VF2PF_RESP_BASE / 4)
+
+/* VF-initiated commands */
+enum dlb2_mbox_cmd_type {
+	DLB2_MBOX_CMD_REGISTER,
+	DLB2_MBOX_CMD_UNREGISTER,
+	DLB2_MBOX_CMD_GET_NUM_RESOURCES,
+	DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_RESET_SCHED_DOMAIN,
+	DLB2_MBOX_CMD_CREATE_LDB_QUEUE,
+	DLB2_MBOX_CMD_CREATE_DIR_QUEUE,
+	DLB2_MBOX_CMD_CREATE_LDB_PORT,
+	DLB2_MBOX_CMD_CREATE_DIR_PORT,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT,
+	DLB2_MBOX_CMD_DISABLE_LDB_PORT,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT,
+	DLB2_MBOX_CMD_DISABLE_DIR_PORT,
+	DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN,
+	DLB2_MBOX_CMD_MAP_QID,
+	DLB2_MBOX_CMD_UNMAP_QID,
+	DLB2_MBOX_CMD_START_DOMAIN,
+	DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR,
+	DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR,
+	DLB2_MBOX_CMD_ARM_CQ_INTR,
+	DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES,
+	DLB2_MBOX_CMD_GET_SN_ALLOCATION,
+	DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB2_MBOX_CMD_PENDING_PORT_UNMAPS,
+	DLB2_MBOX_CMD_GET_COS_BW,
+	DLB2_MBOX_CMD_GET_SN_OCCUPANCY,
+	DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE,
+
+	/* NUM_QE_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_CMD_TYPES,
+};
+
+static const char dlb2_mbox_cmd_type_strings[][128] = {
+	"DLB2_MBOX_CMD_REGISTER",
+	"DLB2_MBOX_CMD_UNREGISTER",
+	"DLB2_MBOX_CMD_GET_NUM_RESOURCES",
+	"DLB2_MBOX_CMD_CREATE_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_RESET_SCHED_DOMAIN",
+	"DLB2_MBOX_CMD_CREATE_LDB_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_DIR_QUEUE",
+	"DLB2_MBOX_CMD_CREATE_LDB_PORT",
+	"DLB2_MBOX_CMD_CREATE_DIR_PORT",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_DISABLE_LDB_PORT",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_DISABLE_DIR_PORT",
+	"DLB2_MBOX_CMD_LDB_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_DIR_PORT_OWNED_BY_DOMAIN",
+	"DLB2_MBOX_CMD_MAP_QID",
+	"DLB2_MBOX_CMD_UNMAP_QID",
+	"DLB2_MBOX_CMD_START_DOMAIN",
+	"DLB2_MBOX_CMD_ENABLE_LDB_PORT_INTR",
+	"DLB2_MBOX_CMD_ENABLE_DIR_PORT_INTR",
+	"DLB2_MBOX_CMD_ARM_CQ_INTR",
+	"DLB2_MBOX_CMD_GET_NUM_USED_RESOURCES",
+	"DLB2_MBOX_CMD_GET_SN_ALLOCATION",
+	"DLB2_MBOX_CMD_GET_LDB_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_GET_DIR_QUEUE_DEPTH",
+	"DLB2_MBOX_CMD_PENDING_PORT_UNMAPS",
+	"DLB2_MBOX_CMD_GET_COS_BW",
+	"DLB2_MBOX_CMD_GET_SN_OCCUPANCY",
+	"DLB2_MBOX_CMD_QUERY_CQ_POLL_MODE",
+};
+
+/* PF-initiated commands */
+enum dlb2_mbox_vf_cmd_type {
+	DLB2_MBOX_VF_CMD_DOMAIN_ALERT,
+	DLB2_MBOX_VF_CMD_NOTIFICATION,
+	DLB2_MBOX_VF_CMD_IN_USE,
+
+	/* NUM_DLB2_MBOX_VF_CMD_TYPES must be last */
+	NUM_DLB2_MBOX_VF_CMD_TYPES,
+};
+
+static const char dlb2_mbox_vf_cmd_type_strings[][128] = {
+	"DLB2_MBOX_VF_CMD_DOMAIN_ALERT",
+	"DLB2_MBOX_VF_CMD_NOTIFICATION",
+	"DLB2_MBOX_VF_CMD_IN_USE",
+};
+
+#define DLB2_MBOX_CMD_TYPE(hdr) \
+	(((struct dlb2_mbox_req_hdr *)hdr)->type)
+#define DLB2_MBOX_CMD_STRING(hdr) \
+	dlb2_mbox_cmd_type_strings[DLB2_MBOX_CMD_TYPE(hdr)]
+
+enum dlb2_mbox_status_type {
+	DLB2_MBOX_ST_SUCCESS,
+	DLB2_MBOX_ST_INVALID_CMD_TYPE,
+	DLB2_MBOX_ST_VERSION_MISMATCH,
+	DLB2_MBOX_ST_INVALID_OWNER_VF,
+};
+
+static const char dlb2_mbox_status_type_strings[][128] = {
+	"DLB2_MBOX_ST_SUCCESS",
+	"DLB2_MBOX_ST_INVALID_CMD_TYPE",
+	"DLB2_MBOX_ST_VERSION_MISMATCH",
+	"DLB2_MBOX_ST_INVALID_OWNER_VF",
+};
+
+#define DLB2_MBOX_ST_TYPE(hdr) \
+	(((struct dlb2_mbox_resp_hdr *)hdr)->status)
+#define DLB2_MBOX_ST_STRING(hdr) \
+	dlb2_mbox_status_type_strings[DLB2_MBOX_ST_TYPE(hdr)]
+
+/* This structure is always the first field in a request structure */
+struct dlb2_mbox_req_hdr {
+	u32 type;
+};
+
+/* This structure is always the first field in a response structure */
+struct dlb2_mbox_resp_hdr {
+	u32 status;
+};
+
+struct dlb2_mbox_register_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 min_interface_version;
+	u16 max_interface_version;
+};
+
+struct dlb2_mbox_register_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 interface_version;
+	u8 pf_id;
+	u8 vf_id;
+	u8 is_auxiliary_vf;
+	u8 primary_vf_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_unregister_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_num_resources_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u16 num_sched_domains;
+	u16 num_ldb_queues;
+	u16 num_ldb_ports;
+	u16 num_cos_ldb_ports[4];
+	u16 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 max_contiguous_hist_list_entries;
+	u16 num_ldb_credits;
+	u16 num_dir_credits;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 num_ldb_queues;
+	u32 num_ldb_ports;
+	u32 num_cos_ldb_ports[4];
+	u32 num_dir_ports;
+	u32 num_atomic_inflights;
+	u32 num_hist_list_entries;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u8 cos_strict;
+	u8 padding0[3];
+	u32 padding1;
+};
+
+struct dlb2_mbox_create_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 id;
+};
+
+struct dlb2_mbox_reset_sched_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 num_sequence_numbers;
+	u32 num_qid_inflights;
+	u32 num_atomic_inflights;
+	u32 lock_id_comp_level;
+	u32 depth_threshold;
+	u32 padding;
+};
+
+struct dlb2_mbox_create_ldb_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 depth_threshold;
+};
+
+struct dlb2_mbox_create_dir_queue_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u16 cq_depth;
+	u16 cq_history_list_size;
+	u8 cos_id;
+	u8 cos_strict;
+	u16 padding1;
+	u64 cq_base_address;
+};
+
+struct dlb2_mbox_create_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u64 cq_base_address;
+	u16 cq_depth;
+	u16 padding0;
+	s32 queue_id;
+};
+
+struct dlb2_mbox_create_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_ldb_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_disable_dir_port_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	s32 owned;
+};
+
+struct dlb2_mbox_map_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+	u32 priority;
+	u32 padding0;
+};
+
+struct dlb2_mbox_map_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 id;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 qid;
+};
+
+struct dlb2_mbox_unmap_qid_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_start_domain_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+};
+
+struct dlb2_mbox_start_domain_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u16 port_id;
+	u16 thresh;
+	u16 vector;
+	u16 owner_vf;
+	u16 reserved[2];
+};
+
+struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 is_ldb;
+};
+
+struct dlb2_mbox_arm_cq_intr_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 padding0;
+};
+
+/*
+ * The alert_id and aux_alert_data follows the format of the alerts defined in
+ * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
+ * the aux_alert_data value varies depending on the alert.
+ */
+struct dlb2_mbox_vf_alert_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 alert_id;
+	u32 aux_alert_data;
+};
+
+enum dlb2_mbox_vf_notification_type {
+	DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
+	DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
+
+	/* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
+	NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
+};
+
+struct dlb2_mbox_vf_notification_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 notification;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_vf_in_use_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 in_use;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_allocation_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 queue_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 depth;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 domain_id;
+	u32 port_id;
+	u32 padding;
+};
+
+struct dlb2_mbox_pending_port_unmaps_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 num;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 cos_id;
+};
+
+struct dlb2_mbox_get_cos_bw_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 group_id;
+};
+
+struct dlb2_mbox_get_sn_occupancy_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 num;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_req {
+	struct dlb2_mbox_req_hdr hdr;
+	u32 padding;
+};
+
+struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
+	struct dlb2_mbox_resp_hdr hdr;
+	u32 error_code;
+	u32 status;
+	u32 mode;
+};
+
+#endif /* __DLB2_BASE_DLB2_MBOX_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
new file mode 100644
index 0000000..c8d8d5b
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
@@ -0,0 +1,248 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_H
+#define __DLB2_OSDEP_H
+
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include <rte_string_fns.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include "../dlb2_main.h"
+#include "dlb2_resource.h"
+#include "../../dlb2_log.h"
+#include "../../dlb2_user.h"
+
+
+#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
+#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
+#define DLB2_CSR_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
+#define DLB2_FUNC_RD(hw, reg) \
+	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
+/* Map to PMDs logging interface */
+#define DLB2_ERR(dev, fmt, args...) \
+	DLB2_LOG_ERR(fmt, ## args)
+
+#define DLB2_INFO(dev, fmt, args...) \
+	DLB2_LOG_INFO(fmt, ## args)
+
+#define DLB2_DEBUG(dev, fmt, args...) \
+	DLB2_LOG_DBG(fmt, ## args)
+
+/**
+ * os_udelay() - busy-wait for a number of microseconds
+ * @usecs: delay duration.
+ */
+static inline void os_udelay(int usecs)
+{
+	rte_delay_us(usecs);
+}
+
+/**
+ * os_msleep() - sleep for a number of milliseconds
+ * @usecs: delay duration.
+ */
+static inline void os_msleep(int msecs)
+{
+	rte_delay_ms(msecs);
+}
+
+#define DLB2_PP_BASE(__is_ldb) \
+	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
+
+/**
+ * os_map_producer_port() - map a producer port into the caller's address space
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ *
+ * This function maps the requested producer port memory into the caller's
+ * address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void *os_map_producer_port(struct dlb2_hw *hw,
+					 u8 port_id,
+					 bool is_ldb)
+{
+	uint64_t addr;
+	uint64_t pp_dma_base;
+
+
+	pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
+	addr = (pp_dma_base + (PAGE_SIZE * port_id));
+
+	return (void *)(uintptr_t)addr;
+}
+
+/**
+ * os_unmap_producer_port() - unmap a producer port
+ * @addr: mapped producer port address
+ *
+ * This function undoes os_map_producer_port() by unmapping the producer port
+ * memory from the caller's address space.
+ *
+ * Return:
+ * Returns the base address at which the PP memory was mapped, else NULL.
+ */
+static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
+{
+	RTE_SET_USED(hw);
+	RTE_SET_USED(addr);
+}
+
+/**
+ * os_fence_hcw() - fence an HCW to ensure it arrives at the device
+ * @hw: dlb2_hw handle for a particular device.
+ * @pp_addr: producer port address
+ */
+static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
+{
+	RTE_SET_USED(hw);
+
+	/* To ensure outstanding HCWs reach the device, read the PP address. IA
+	 * memory ordering prevents reads from passing older writes, and the
+	 * mfence also ensures this.
+	 */
+	rte_mb();
+
+	*(volatile u64 *)pp_addr;
+}
+
+/**
+ * os_enqueue_four_hcws() - enqueue four HCWs to DLB
+ * @hw: dlb2_hw handle for a particular device.
+ * @hcw: pointer to the 64B-aligned contiguous HCW memory
+ * @addr: producer port address
+ */
+static inline void os_enqueue_four_hcws(struct dlb2_hw *hw,
+					struct dlb2_hcw *hcw,
+					void *addr)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	dlb2_dev->enqueue_four(hcw, addr);
+}
+
+/**
+ * DLB2_HW_ERR() - log an error message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_ERR(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_ERR(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/**
+ * DLB2_HW_DBG() - log an info message
+ * @dlb2: dlb2_hw handle for a particular device.
+ * @...: variable string args.
+ */
+#define DLB2_HW_DBG(dlb2, ...) do {	\
+	RTE_SET_USED(dlb2);		\
+	DLB2_DEBUG(dlb2, __VA_ARGS__);	\
+} while (0)
+
+/* The callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void *dlb2_complete_queue_map_unmap(void *__args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
+	int ret;
+
+	while (1) {
+		rte_spinlock_lock(&dlb2_dev->resource_mutex);
+
+		ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+		ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+		if (ret != 0) {
+			rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+			/* Relinquish the CPU so the application can process
+			 * its CQs, so this function doesn't deadlock.
+			 */
+			sched_yield();
+		} else {
+			break;
+		}
+	}
+
+	dlb2_dev->worker_launched = false;
+
+	rte_spinlock_unlock(&dlb2_dev->resource_mutex);
+
+	return NULL;
+}
+
+
+/**
+ * os_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static inline void os_schedule_work(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+	pthread_t complete_queue_map_unmap_thread;
+	int ret;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
+				     "dlb_queue_unmap_waiter",
+				     NULL,
+				     dlb2_complete_queue_map_unmap,
+				     dlb2_dev);
+	if (ret)
+		DLB2_ERR(dlb2_dev,
+			 "Could not create queue complete map/unmap thread, err=%d\n",
+			 ret);
+	else
+		dlb2_dev->worker_launched = true;
+}
+
+/**
+ * os_worker_active() - query whether the map/unmap worker thread is active
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether a thread (launched by
+ * os_schedule_work()) is active. This function is used to determine
+ * whether or not to launch a worker thread.
+ */
+static inline bool os_worker_active(struct dlb2_hw *hw)
+{
+	struct dlb2_dev *dlb2_dev;
+
+	dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+	return dlb2_dev->worker_launched;
+}
+
+#endif /*  __DLB2_OSDEP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
new file mode 100644
index 0000000..7e48878
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
@@ -0,0 +1,447 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <rte_bitmap.h>
+#include <rte_string_fns.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include "../dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+	struct rte_bitmap *map;
+	unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+				    unsigned int len)
+{
+	struct dlb2_bitmap *bm;
+	void *mem;
+	uint32_t alloc_size;
+	uint32_t nbits = (uint32_t)len;
+
+	if (!bitmap || nbits == 0)
+		return -EINVAL;
+
+	/* Allocate DLB2 bitmap control struct */
+	bm = rte_malloc("DLB2_PF",
+			sizeof(struct dlb2_bitmap),
+			RTE_CACHE_LINE_SIZE);
+
+	if (!bm)
+		return -ENOMEM;
+
+	/* Allocate bitmap memory */
+	alloc_size = rte_bitmap_get_memory_footprint(nbits);
+	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
+	if (!mem) {
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->map = rte_bitmap_init(len, mem, alloc_size);
+	if (!bm->map) {
+		rte_free(mem);
+		rte_free(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+	if (!bitmap)
+		return;
+
+	rte_free(bitmap->map);
+	rte_free(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (!bitmap || !bitmap->map)
+		return -EINVAL;
+
+	/* TODO - optimize */
+	for (i = 0; i != bitmap->len; i++)
+		rte_bitmap_set(bitmap->map, i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+	if (!bitmap || !bitmap->map)
+		return -EINVAL;
+
+	rte_bitmap_reset(bitmap->map);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set() - set a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
+				  unsigned int bit)
+{
+	if (!bitmap || !bitmap->map)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_set(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+					unsigned int bit,
+					unsigned int len)
+{
+	unsigned int i;
+
+	if (!bitmap || !bitmap->map)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	/* TODO - optimize */
+	for (i = 0; i != len; i++)
+		rte_bitmap_set(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear() - clear a bitmap entry
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: bit index.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
+ *	    bitmap length.
+ */
+static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
+				    unsigned int bit)
+{
+	if (!bitmap || !bitmap->map)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	rte_bitmap_clear(bitmap->map, bit);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+					  unsigned int bit,
+					  unsigned int len)
+{
+	unsigned int i;
+
+	if (!bitmap || !bitmap->map)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	/* TODO - optimize */
+	for (i = 0; i != len; i++)
+		rte_bitmap_clear(bitmap->map, bit + i);
+
+	return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+						 unsigned int len)
+{
+	unsigned int i, j = 0;
+
+	if (!bitmap || !bitmap->map || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	/* TODO - optimize */
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			if (++j == len)
+				return i - j + 1;
+		} else {
+			j = 0;
+		}
+	}
+
+	/* No set bit range of length len? */
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
+{
+	unsigned int i;
+
+	if (!bitmap)
+		return -EINVAL;
+
+	if (!bitmap->map)
+		return -EINVAL;
+
+	/* TODO - optimize */
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			return i;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+	int weight = 0;
+	unsigned int i;
+
+	if (!bitmap)
+		return -EINVAL;
+
+	if (!bitmap->map)
+		return -EINVAL;
+
+	/* TODO - optimize */
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i))
+			weight++;
+	}
+	return weight;
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				      bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+	int max_len = 0, len = 0;
+	unsigned int i;
+
+	if (!bitmap)
+		return -EINVAL;
+
+	if (!bitmap->map)
+		return -EINVAL;
+
+	/* TODO - optimize */
+	for (i = 0; i != bitmap->len; i++) {
+		if  (rte_bitmap_get(bitmap->map, i)) {
+			len++;
+		} else {
+			if (len > max_len)
+				max_len = len;
+			len = 0;
+		}
+	}
+
+	if (len > max_len)
+		max_len = len;
+
+	return max_len;
+}
+
+/**
+ * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
+ * @dest: pointer to dlb2_bitmap structure, which will contain the results of
+ *	  the 'or' of src1 and src2.
+ * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
+ * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
+ *
+ * This function 'or's two bitmaps together and stores the result in a third
+ * bitmap. The source and destination bitmaps can be the same.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - One of the bitmaps is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
+				 struct dlb2_bitmap *src1,
+				 struct dlb2_bitmap *src2)
+{
+	unsigned int i, min;
+	int numset = 0;
+
+	if (!dest || !dest->map ||
+	    !src1 || !src1->map ||
+	    !src2 || !src2->map)
+		return -EINVAL;
+
+	min = dest->len;
+	min = (min > src1->len) ? src1->len : min;
+	min = (min > src2->len) ? src2->len : min;
+
+	for (i = 0; i != min; i++) {
+		if  (rte_bitmap_get(src1->map, i) ||
+		     rte_bitmap_get(src2->map, i)) {
+			rte_bitmap_set(dest->map, i);
+			numset++;
+		} else {
+			rte_bitmap_clear(dest->map, i);
+		}
+	}
+
+	return numset;
+}
+
+#endif /*  __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
new file mode 100644
index 0000000..5531739
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
@@ -0,0 +1,131 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_LIST_H
+#define __DLB2_OSDEP_LIST_H
+
+#include <rte_tailq.h>
+
+struct dlb2_list_entry {
+	TAILQ_ENTRY(dlb2_list_entry) node;
+};
+
+/* Dummy - just a struct definition */
+TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
+
+/* =================
+ * TAILQ Supplements
+ * =================
+ */
+
+#ifndef TAILQ_FOREACH_ENTRY
+#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)		\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter)						\
+		&& (ptr = container_of(iter, typeof(*(ptr)), name)); \
+	    (iter) = TAILQ_NEXT((iter), node))
+#endif
+
+#ifndef TAILQ_FOREACH_ENTRY_SAFE
+#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)	\
+	for ((iter) = TAILQ_FIRST(&head);			\
+	    (iter) &&						\
+		(ptr = container_of(iter, typeof(*(ptr)), name)) &&\
+		((tvar) = TAILQ_NEXT((iter), node), 1);	\
+	    (iter) = (tvar))
+#endif
+
+/***********************/
+/*** List operations ***/
+/***********************/
+
+/**
+ * dlb2_list_init_head() - initialize the head of a list
+ * @head: list head
+ */
+static inline void dlb2_list_init_head(struct dlb2_list_head *head)
+{
+	TAILQ_INIT(head);
+}
+
+/**
+ * dlb2_list_add() - add an entry to a list
+ * @head: list head
+ * @entry: new list entry
+ */
+static inline void
+dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
+{
+	TAILQ_INSERT_TAIL(head, entry, node);
+}
+
+/**
+ * dlb2_list_del() - delete an entry from a list
+ * @entry: list entry
+ * @head: list head
+ */
+static inline void dlb2_list_del(struct dlb2_list_head *head,
+				 struct dlb2_list_entry *entry)
+{
+	TAILQ_REMOVE(head, entry, node);
+}
+
+/**
+ * dlb2_list_empty() - check if a list is empty
+ * @head: list head
+ *
+ * Return:
+ * Returns 1 if empty, 0 if not.
+ */
+static inline int dlb2_list_empty(struct dlb2_list_head *head)
+{
+	return TAILQ_EMPTY(head);
+}
+
+/**
+ * dlb2_list_splice() - splice a list
+ * @src_head: list to be added
+ * @ head: where src_head will be inserted
+ */
+static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
+				    struct dlb2_list_head *head)
+{
+	TAILQ_CONCAT(head, src_head, node);
+}
+
+/**
+ * DLB2_LIST_HEAD() - retrieve the head of the list
+ * @head: list head
+ * @type: type of the list variable
+ * @name: name of the list field within the containing struct
+ */
+#define DLB2_LIST_HEAD(head, type, name)                       \
+	(TAILQ_FIRST(&head) ?					\
+		container_of(TAILQ_FIRST(&head), type, name) :	\
+		NULL)
+
+/**
+ * DLB2_LIST_FOR_EACH() - iterate over a list
+ * @head: list head
+ * @ptr: pointer to struct containing a struct list
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ */
+#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
+	TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
+
+/**
+ * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
+ * an element is removed from the list while processing it.
+ * @ptr: pointer to struct containing a struct list
+ * @ptr_tmp: pointer to struct containing a struct list (temporary)
+ * @head: list head
+ * @name: name of the list field within the containing struct
+ * @iter: iterator variable
+ * @iter_tmp: iterator variable (temporary)
+ */
+#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
+	TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
+
+#endif /*  __DLB2_OSDEP_LIST_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
new file mode 100644
index 0000000..0a48f7e
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_TYPES_H
+#define __DLB2_OSDEP_TYPES_H
+
+#include <linux/types.h>
+
+#include <inttypes.h>
+#include <ctype.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Types for user mode PF PMD */
+typedef uint8_t         u8;
+typedef int8_t          s8;
+typedef uint16_t        u16;
+typedef int16_t         s16;
+typedef uint32_t        u32;
+typedef int32_t         s32;
+typedef uint64_t        u64;
+
+#define __iomem
+
+/* END types for user mode PF PMD */
+
+#endif /* __DLB2_OSDEP_TYPES_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
new file mode 100644
index 0000000..43ecad4
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
@@ -0,0 +1,2527 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "dlb2_osdep_types.h"
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+	struct {
+		u32 isr_pend : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+	struct {
+		u32 vf0_isr : 1;
+		u32 vf1_isr : 1;
+		u32 vf2_isr : 1;
+		u32 vf3_isr : 1;
+		u32 vf4_isr : 1;
+		u32 vf5_isr : 1;
+		u32 vf6_isr : 1;
+		u32 vf7_isr : 1;
+		u32 vf8_isr : 1;
+		u32 vf9_isr : 1;
+		u32 vf10_isr : 1;
+		u32 vf11_isr : 1;
+		u32 vf12_isr : 1;
+		u32 vf13_isr : 1;
+		u32 vf14_isr : 1;
+		u32 vf15_isr : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+	struct {
+		u32 vf0_reset_in_progress : 1;
+		u32 vf1_reset_in_progress : 1;
+		u32 vf2_reset_in_progress : 1;
+		u32 vf3_reset_in_progress : 1;
+		u32 vf4_reset_in_progress : 1;
+		u32 vf5_reset_in_progress : 1;
+		u32 vf6_reset_in_progress : 1;
+		u32 vf7_reset_in_progress : 1;
+		u32 vf8_reset_in_progress : 1;
+		u32 vf9_reset_in_progress : 1;
+		u32 vf10_reset_in_progress : 1;
+		u32 vf11_reset_in_progress : 1;
+		u32 vf12_reset_in_progress : 1;
+		u32 vf13_reset_in_progress : 1;
+		u32 vf14_reset_in_progress : 1;
+		u32 vf15_reset_in_progress : 1;
+		u32 rsvd0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+	struct {
+		u32 vec_mask : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+	struct {
+		u32 func_vf_bar_dis : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+	struct {
+		u32 total_vas : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+	struct {
+		u32 total_dir_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+	struct {
+		u32 total_ldb_ports : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+	struct {
+		u32 total_dir_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+	struct {
+		u32 total_ldb_qid : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+	struct {
+		u32 total_dir_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+	struct {
+		u32 total_ldb_credits : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 meas : 1;
+		u32 debug : 7;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 cq_int_rearm : 1;
+		u32 dsi_error : 1;
+		u32 rsvd0 : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 rsvd0 : 3;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+	struct {
+		u32 vpp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+	struct {
+		u32 pp : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+	struct {
+		u32 qid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+	struct {
+		u32 vqid : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+	struct {
+		u32 vqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+	struct {
+		u32 qid : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+	struct {
+		u32 vasqid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+	struct {
+		u32 lock_id : 16;
+		u32 debug : 8;
+		u32 cq_pop : 1;
+		u32 qe_uhl : 1;
+		u32 qe_orsp : 1;
+		u32 qe_valid : 1;
+		u32 isz : 1;
+		u32 dsi_error : 1;
+		u32 dlbrsvd : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+	struct {
+		u32 dsi : 16;
+		u32 qid : 8;
+		u32 qtype : 2;
+		u32 qpri : 3;
+		u32 msg_type : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 vf_synd0_parity : 1;
+		u32 vf_synd1_parity : 1;
+		u32 vf_synd2_parity : 1;
+		u32 is_ldb : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+	struct {
+		u32 sn_cfg_v : 1;
+		u32 fid_cfg_v : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+	struct {
+		u32 qid_its : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+	struct {
+		u32 qid_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS  0
+#define DLB2_CQ_ISR_MODE_MSI  1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI  3
+union dlb2_sys_ldb_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+	struct {
+		u32 keep_pf_ppid : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+	struct {
+		u32 cq_ai_data : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+	struct {
+		u32 rsvd1 : 2;
+		u32 cq_ai_addr : 18;
+		u32 rsvd0 : 12;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+	struct {
+		u32 pasid : 20;
+		u32 exe_req : 1;
+		u32 priv_req : 1;
+		u32 fmt2 : 1;
+		u32 rsvd0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+	struct {
+		u32 cq_at : 2;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+	struct {
+		u32 vector : 6;
+		u32 vf : 4;
+		u32 en_code : 2;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+	struct {
+		u32 vf : 4;
+		u32 is_pf : 1;
+		u32 ro : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+	struct {
+		u32 pp_v : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+	struct {
+		u32 vdev : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+	struct {
+		u32 vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+	struct {
+		u32 addr_u : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+	struct {
+		u32 rsvd0 : 6;
+		u32 addr_l : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+	struct {
+		u32 illegal_hcw : 1;
+		u32 illegal_pp : 1;
+		u32 illegal_pasid : 1;
+		u32 illegal_qid : 1;
+		u32 disabled_qid : 1;
+		u32 illegal_ldb_qid_cfg : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+	struct {
+		u32 msix_0_ack : 1;
+		u32 msix_1_ack : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+	struct {
+		u32 msix_0_passthru : 1;
+		u32 msix_1_passthru : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED     0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+	struct {
+		u32 mode : 1;
+		u32 poll_mode : 1;
+		u32 poll_mask : 1;
+		u32 poll_lock : 1;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+	struct {
+		u32 cq_0_occ_int : 1;
+		u32 cq_1_occ_int : 1;
+		u32 cq_2_occ_int : 1;
+		u32 cq_3_occ_int : 1;
+		u32 cq_4_occ_int : 1;
+		u32 cq_5_occ_int : 1;
+		u32 cq_6_occ_int : 1;
+		u32 cq_7_occ_int : 1;
+		u32 cq_8_occ_int : 1;
+		u32 cq_9_occ_int : 1;
+		u32 cq_10_occ_int : 1;
+		u32 cq_11_occ_int : 1;
+		u32 cq_12_occ_int : 1;
+		u32 cq_13_occ_int : 1;
+		u32 cq_14_occ_int : 1;
+		u32 cq_15_occ_int : 1;
+		u32 cq_16_occ_int : 1;
+		u32 cq_17_occ_int : 1;
+		u32 cq_18_occ_int : 1;
+		u32 cq_19_occ_int : 1;
+		u32 cq_20_occ_int : 1;
+		u32 cq_21_occ_int : 1;
+		u32 cq_22_occ_int : 1;
+		u32 cq_23_occ_int : 1;
+		u32 cq_24_occ_int : 1;
+		u32 cq_25_occ_int : 1;
+		u32 cq_26_occ_int : 1;
+		u32 cq_27_occ_int : 1;
+		u32 cq_28_occ_int : 1;
+		u32 cq_29_occ_int : 1;
+		u32 cq_30_occ_int : 1;
+		u32 cq_31_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+	struct {
+		u32 cq_32_occ_int : 1;
+		u32 cq_33_occ_int : 1;
+		u32 cq_34_occ_int : 1;
+		u32 cq_35_occ_int : 1;
+		u32 cq_36_occ_int : 1;
+		u32 cq_37_occ_int : 1;
+		u32 cq_38_occ_int : 1;
+		u32 cq_39_occ_int : 1;
+		u32 cq_40_occ_int : 1;
+		u32 cq_41_occ_int : 1;
+		u32 cq_42_occ_int : 1;
+		u32 cq_43_occ_int : 1;
+		u32 cq_44_occ_int : 1;
+		u32 cq_45_occ_int : 1;
+		u32 cq_46_occ_int : 1;
+		u32 cq_47_occ_int : 1;
+		u32 cq_48_occ_int : 1;
+		u32 cq_49_occ_int : 1;
+		u32 cq_50_occ_int : 1;
+		u32 cq_51_occ_int : 1;
+		u32 cq_52_occ_int : 1;
+		u32 cq_53_occ_int : 1;
+		u32 cq_54_occ_int : 1;
+		u32 cq_55_occ_int : 1;
+		u32 cq_56_occ_int : 1;
+		u32 cq_57_occ_int : 1;
+		u32 cq_58_occ_int : 1;
+		u32 cq_59_occ_int : 1;
+		u32 cq_60_occ_int : 1;
+		u32 cq_61_occ_int : 1;
+		u32 cq_62_occ_int : 1;
+		u32 cq_63_occ_int : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+	struct {
+		u32 cq : 6;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+	struct {
+		u32 syndrome : 8;
+		u32 rtype : 2;
+		u32 alarm : 1;
+		u32 cwd : 1;
+		u32 vf_pf_mb : 1;
+		u32 rsvd0 : 1;
+		u32 cls : 2;
+		u32 aid : 6;
+		u32 unit : 4;
+		u32 source : 4;
+		u32 more : 1;
+		u32 valid : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+	struct {
+		u32 qid_fid_limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+	struct {
+		u32 compress_code : 3;
+		u32 rsvd0 : 29;
+	} field;
+	u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+	(DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+	struct {
+		u32 bin0 : 8;
+		u32 bin1 : 8;
+		u32 bin2 : 8;
+		u32 bin3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+	struct {
+		u32 count : 15;
+		u32 rsvd0 : 17;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+	struct {
+		u32 sn : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+	struct {
+		u32 mode : 3;
+		u32 slot : 4;
+		u32 rsvz0 : 1;
+		u32 grp : 1;
+		u32 rsvz1 : 1;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+	struct {
+		u32 en : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+	struct {
+		u32 write_pointer : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+	struct {
+		u32 base : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+	struct {
+		u32 limit : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+	struct {
+		u32 pop_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+	struct {
+		u32 push_ptr : 13;
+		u32 generation : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+	struct {
+		u32 depth : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+	struct {
+		u32 depth_threshold : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+	struct {
+		u32 en_tim : 1;
+		u32 en_depth : 1;
+		u32 rsvd0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+	struct {
+		u32 thrsh_0 : 1;
+		u32 thrsh_13_1 : 13;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 rsvd0 : 28;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+	struct {
+		u32 wd_enable : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+	struct {
+		u32 write_pointer : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+	struct {
+		u32 cq2vas : 5;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 dlb_cor_alarm_enable : 1;
+		u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+		u32 cfg_64bytes_qe_dir_cq_mode : 1;
+		u32 pad_write_ldb : 1;
+		u32 pad_write_dir : 1;
+		u32 pad_first_write_ldb : 1;
+		u32 pad_first_write_dir : 1;
+		u32 rsvz0 : 9;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+	struct {
+		u32 armed : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+	struct {
+		u32 sample_interval : 8;
+		u32 enb : 1;
+		u32 rsvz0 : 23;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+	struct {
+		u32 wdto : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+	struct {
+		u32 wd_disable : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+	struct {
+		u32 sample_interval : 28;
+		u32 enb : 1;
+		u32 rsvz0 : 3;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+	struct {
+		u32 wd_threshold : 8;
+		u32 rsvz0 : 24;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+	struct {
+		u32 egress_credit_status_empty : 1;
+		u32 egress_credit_status_afull : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+		u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+		u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+		u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+		u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+		u32 chp_rop_pipe_credit_status_empty : 1;
+		u32 chp_rop_pipe_credit_status_afull : 1;
+		u32 qed_to_cq_pipe_credit_status_empty : 1;
+		u32 qed_to_cq_pipe_credit_status_afull : 1;
+		u32 egress_lsp_token_credit_status_empty : 1;
+		u32 egress_lsp_token_credit_status_afull : 1;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+	struct {
+		u32 int_cor_alarm_dis : 1;
+		u32 int_cor_synd_dis : 1;
+		u32 int_uncr_alarm_dis : 1;
+		u32 int_unc_synd_dis : 1;
+		u32 int_inf0_alarm_dis : 1;
+		u32 int_inf0_synd_dis : 1;
+		u32 int_inf1_alarm_dis : 1;
+		u32 int_inf1_synd_dis : 1;
+		u32 int_inf2_alarm_dis : 1;
+		u32 int_inf2_synd_dis : 1;
+		u32 int_inf3_alarm_dis : 1;
+		u32 int_inf3_synd_dis : 1;
+		u32 int_inf4_alarm_dis : 1;
+		u32 int_inf4_synd_dis : 1;
+		u32 int_inf5_alarm_dis : 1;
+		u32 int_inf5_synd_dis : 1;
+		u32 rsvz0 : 16;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+	struct {
+		u32 pri0 : 8;
+		u32 pri1 : 8;
+		u32 pri2 : 8;
+		u32 pri3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+	struct {
+		u32 change : 10;
+		u32 rsvd0 : 22;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+	struct {
+		u32 sn_mode_0 : 3;
+		u32 rszv0 : 5;
+		u32 sn_mode_1 : 3;
+		u32 rszv1 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+	struct {
+		u32 unit_single_step_mode : 1;
+		u32 rr_en : 1;
+		u32 rszv0 : 30;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+	struct {
+		u32 prio : 24;
+		u32 v : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+	struct {
+		u32 qid_p0 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p1 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p2 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p3 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+	struct {
+		u32 qid_p4 : 7;
+		u32 rsvd3 : 1;
+		u32 qid_p5 : 7;
+		u32 rsvd2 : 1;
+		u32 qid_p6 : 7;
+		u32 rsvd1 : 1;
+		u32 qid_p7 : 7;
+		u32 rsvd0 : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+	struct {
+		u32 token_depth_select : 4;
+		u32 disable_wb_opt : 1;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 26;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+	struct {
+		u32 disabled : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+	struct {
+		u32 token_count : 11;
+		u32 rsvd0 : 21;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+	struct {
+		u32 token_depth_select : 4;
+		u32 ignore_depth : 1;
+		u32 rsvd0 : 27;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+	struct {
+		u32 depth : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+	struct {
+		u32 count : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+	struct {
+		u32 thresh : 13;
+		u32 rsvd0 : 19;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+	struct {
+		u32 count : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+	struct {
+		u32 limit : 12;
+		u32 rsvd0 : 20;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+	(DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+	(DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+	struct {
+		u32 cq_p0 : 8;
+		u32 cq_p1 : 8;
+		u32 cq_p2 : 8;
+		u32 cq_p3 : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+	struct {
+		u32 depth : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+	struct {
+		u32 thresh : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+	struct {
+		u32 count : 14;
+		u32 rsvd0 : 18;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+	struct {
+		u32 pri0_weight : 8;
+		u32 pri1_weight : 8;
+		u32 pri2_weight : 8;
+		u32 pri3_weight : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+	struct {
+		u32 rsvz0 : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+	struct {
+		u32 cq : 8;
+		u32 qidix : 3;
+		u32 value : 1;
+		u32 nalb_haswork_v : 1;
+		u32 rlist_haswork_v : 1;
+		u32 slist_haswork_v : 1;
+		u32 inflight_ok_v : 1;
+		u32 aqed_nfull_v : 1;
+		u32 rsvz0 : 15;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+	struct {
+		u32 count : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+	struct {
+		u32 transfer : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+	struct {
+		u32 bw_range : 9;
+		u32 rsvz0 : 22;
+		u32 no_extra_credit : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+	struct {
+		u32 disab_atq_empty_arb : 1;
+		u32 inc_tok_unit_idle : 1;
+		u32 disab_rlist_pri : 1;
+		u32 inc_cmp_unit_idle : 1;
+		u32 rsvz0 : 2;
+		u32 dir_single_op : 1;
+		u32 dir_half_bw : 1;
+		u32 dir_single_out : 1;
+		u32 dir_disab_multi : 1;
+		u32 atq_single_op : 1;
+		u32 atq_half_bw : 1;
+		u32 atq_single_out : 1;
+		u32 atq_disab_multi : 1;
+		u32 dirrpl_single_op : 1;
+		u32 dirrpl_half_bw : 1;
+		u32 dirrpl_single_out : 1;
+		u32 lbrpl_single_op : 1;
+		u32 lbrpl_half_bw : 1;
+		u32 lbrpl_single_out : 1;
+		u32 ldb_single_op : 1;
+		u32 ldb_half_bw : 1;
+		u32 ldb_disab_multi : 1;
+		u32 atm_single_sch : 1;
+		u32 atm_single_cmp : 1;
+		u32 ldb_ce_tog_arb : 1;
+		u32 rsvz1 : 1;
+		u32 smon0_valid_sel : 2;
+		u32 smon0_value_sel : 1;
+		u32 smon0_compare_sel : 2;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+	struct {
+		u32 chp_pf_reset_done : 1;
+		u32 rop_pf_reset_done : 1;
+		u32 lsp_pf_reset_done : 1;
+		u32 nalb_pf_reset_done : 1;
+		u32 ap_pf_reset_done : 1;
+		u32 dp_pf_reset_done : 1;
+		u32 qed_pf_reset_done : 1;
+		u32 dqed_pf_reset_done : 1;
+		u32 aqed_pf_reset_done : 1;
+		u32 sys_pf_reset_done : 1;
+		u32 pf_reset_active : 1;
+		u32 flrsm_state : 7;
+		u32 rsvd0 : 13;
+		u32 dlb_proc_reset_done : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+	struct {
+		u32 chp_pipeidle : 1;
+		u32 rop_pipeidle : 1;
+		u32 lsp_pipeidle : 1;
+		u32 nalb_pipeidle : 1;
+		u32 ap_pipeidle : 1;
+		u32 dp_pipeidle : 1;
+		u32 qed_pipeidle : 1;
+		u32 dqed_pipeidle : 1;
+		u32 aqed_pipeidle : 1;
+		u32 sys_pipeidle : 1;
+		u32 chp_unit_idle : 1;
+		u32 rop_unit_idle : 1;
+		u32 lsp_unit_idle : 1;
+		u32 nalb_unit_idle : 1;
+		u32 ap_unit_idle : 1;
+		u32 dp_unit_idle : 1;
+		u32 qed_unit_idle : 1;
+		u32 dqed_unit_idle : 1;
+		u32 aqed_unit_idle : 1;
+		u32 sys_unit_idle : 1;
+		u32 rsvd1 : 4;
+		u32 mstr_cfg_ring_idle : 1;
+		u32 mstr_cfg_mstr_idle : 1;
+		u32 mstr_flr_clkreq_b : 1;
+		u32 mstr_proc_idle : 1;
+		u32 mstr_proc_idle_masked : 1;
+		u32 rsvd0 : 2;
+		u32 dlb_func_idle : 1;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+	struct {
+		u32 prochot : 1;
+		u32 pgcb_dlb_idle : 1;
+		u32 pgcb_dlb_pg_rdy_ack_b : 1;
+		u32 pmsm_pgcb_req_b : 1;
+		u32 pgbc_pmc_pg_req_b : 1;
+		u32 pmc_pgcb_pg_ack_b : 1;
+		u32 pmc_pgcb_fet_en_b : 1;
+		u32 pgcb_fet_en_b : 1;
+		u32 rsvz0 : 1;
+		u32 rsvz1 : 1;
+		u32 fuse_force_on : 1;
+		u32 fuse_proc_disable : 1;
+		u32 rsvz2 : 1;
+		u32 rsvz3 : 1;
+		u32 pm_fsm_d0tod3_ok : 1;
+		u32 pm_fsm_d3tod0_ok : 1;
+		u32 dlb_in_d3 : 1;
+		u32 rsvz4 : 7;
+		u32 pmsm : 8;
+	} field;
+	u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+	struct {
+		u32 disable : 1;
+		u32 rsvz0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+	struct {
+		u32 isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+	struct {
+		u32 msg : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+	struct {
+		u32 pf_isr : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+	struct {
+		u32 isr_pend : 32;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+	struct {
+		u32 reset_in_progress : 1;
+		u32 rsvd0 : 31;
+	} field;
+	u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+	struct {
+		u32 vf_msi_isr : 32;
+	} field;
+	u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
new file mode 100644
index 0000000..6de8b95
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_mbox.h"
+#include "dlb2_osdep.h"
+#include "dlb2_osdep_bitmap.h"
+#include "dlb2_osdep_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+	int i;
+
+	dlb2_list_init_head(&domain->used_ldb_queues);
+	dlb2_list_init_head(&domain->used_dir_pq_pairs);
+	dlb2_list_init_head(&domain->avail_ldb_queues);
+	dlb2_list_init_head(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&domain->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+	int i;
+
+	dlb2_list_init_head(&rsrc->avail_domains);
+	dlb2_list_init_head(&rsrc->used_domains);
+	dlb2_list_init_head(&rsrc->avail_ldb_queues);
+	dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
+	arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
+	arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
+	arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb2_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+	union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+	r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+	struct dlb2_list_entry *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. This is application
+	 * dependent, but the driver interleaves port IDs as much as possible
+	 * to reduce the likelihood of this. This initial allocation maximizes
+	 * the average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	/* Zero-out resource tracking data structures */
+	memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+	memset(&hw->pf, 0, sizeof(hw->pf));
+
+	dlb2_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+		dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+		dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_domains, list);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_ldb_queues, list);
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+		struct dlb2_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
+			      &port->func_list);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+				DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+	if (ret)
+		goto unwind;
+
+	for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+		ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+					DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+		if (ret)
+			goto unwind;
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb2_resource_free(hw);
+
+	return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+	union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+	r0.field.disable = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
new file mode 100644
index 0000000..503fdf3
--- /dev/null
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
@@ -0,0 +1,1913 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include "dlb2_user.h"
+
+#include "dlb2_hw_types.h"
+#include "dlb2_osdep_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_start_domain() - start a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: start domain arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int dlb2_hw_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp,
+			 bool vdev_request,
+			 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_request,
+		    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding unmap procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_finish_map_qid_procedures() - finish any pending map procedures
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function attempts to finish any outstanding map procedures.
+ * This function should be called by the kernel thread responsible for
+ * finishing map/unmap procedures.
+ *
+ * Return:
+ * Returns the number of procedures that weren't completed.
+ */
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_request,
+			    unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_request,
+			     unsigned int vdev_id);
+
+/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ *					interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ *	    else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ *	virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ *	greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+				    int port_id,
+				    int vector,
+				    int mode,
+				    unsigned int vf,
+				    unsigned int owner_vf,
+				    u16 threshold);
+
+/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+			  int port_id,
+			  bool is_ldb,
+			  bool vdev_request,
+			  unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+					 u32 *ldb_interrupts,
+					 u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+				 u32 *ldb_interrupts,
+				 u32 *dir_interrupts);
+
+/**
+ * dlb2_read_vf_intr_status() - read the VF interrupt status register
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function can be called from a VF's interrupt handler to determine
+ * which interrupts have fired. The first 31 bits correspond to CQ interrupt
+ * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
+ *
+ * Return:
+ * Returns a bit vector indicating which interrupt vectors are active.
+ */
+u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_intr_status() - ack VF interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function ACKs a VF's interrupts. Its interrupts argument should be the
+ * value returned by dlb2_read_vf_intr_status().
+ */
+void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @interrupts: 32-bit bitmap
+ *
+ * This function clears the VF's MSI interrupt pending register. Its interrupts
+ * argument should be contain the MSI vectors to ACK. For example, if MSI MME
+ * is in mode 0, then one bit 0 should ever be set.
+ */
+void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
+
+/**
+ * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When done processing the PF mailbox request, this function unsets
+ * the PF's mailbox ISR register.
+ */
+void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
+
+/**
+ * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
+ *					vdevs
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the vdev->PF ISR fires, this function can be called to determine which
+ * vdev(s) are requesting service. This bitvector must be passed to
+ * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
+ * vdevs.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested service.
+ */
+u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ *
+ * When done processing all VF mailbox requests, this function unsets the VF's
+ * mailbox ISR register.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
+ *				    FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * When the VF FLR ISR fires, this function can be called to determine which
+ * VF(s) are requesting FLRs. This bitvector must passed to
+ * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
+ *
+ * Return:
+ * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
+ */
+u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
+
+/**
+ * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF FLR requests, this function unsets the VF's FLR
+ * ISR register.
+ */
+void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
+
+/**
+ * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
+ * @hw: dlb2_hw handle for a particular device.
+ * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
+ * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
+ *
+ * When done processing all VF requests, this function communicates to the
+ * hardware that processing is complete.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
+			     u32 mbox_bitvec,
+			     u32 flr_bitvec);
+
+/**
+ * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the watchdog timer interrupt cause registers to
+ * determine which port(s) had a watchdog timeout, and notifies the
+ * application(s) that own the port(s).
+ */
+void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads and logs the alarm syndrome, then acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_request,
+		      unsigned int vdev_id);
+
+/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: indicates whether this request came from a VF.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  u32 port_id,
+				  bool vdev_request,
+				  unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+			      struct dlb2_get_num_resources_args *arg,
+			      bool vdev_request,
+			      unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of resources in use by the PF or a VF. It
+ * fills in the fields that args points to, except the following:
+ * - max_contiguous_atomic_inflights
+ * - max_contiguous_hist_list_entries
+ * - max_contiguous_ldb_credits
+ * - max_contiguous_dir_credits
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
+				   struct dlb2_get_num_resources_args *arg,
+				   bool vdev_request,
+				   unsigned int vdev_id);
+
+/**
+ * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
+ *				       the PF
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function sends a VF->PF mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the PF has processed
+ * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
+ * when the PF has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
+ *				 request
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * processing a VF->PF mailbox request. It should only be called after sending
+ * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_vf_flr_complete() - check the status of a VF FLR
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function returns a boolean indicating whether the PF has finished
+ * executing the VF FLR. It should only be called after setting the VF's FLR
+ * bit.
+ */
+bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
+
+/**
+ * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
+ *					vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function sends a PF->vdev mailbox message. It is asynchronous, so it
+ * returns once the message is sent but potentially before the vdev has
+ * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
+ * determine when the vdev has finished processing the request.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
+ *			       request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function returns a boolean indicating whether the vdev has finished
+ * processing a PF->vdev mailbox request. It should only be called after
+ * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
+			     unsigned int vf_id,
+			     void *data,
+			     int len);
+
+/**
+ * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies one of the PF's VF->PF mailboxes into the array pointed
+ * to by data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
+			       unsigned int vf_id,
+			       void *data,
+			       int len);
+
+/**
+ * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the PF's VF->PF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
+			      unsigned int vf_id,
+			      void *data,
+			      int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the VF's PF->VF mailbox into the array pointed to by
+ * data.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
+ */
+int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
+ * @hw: dlb2_hw handle for a particular device.
+ * @data: pointer to message data.
+ * @len: size, in bytes, of the data array.
+ *
+ * This function copies the user-provided message data into of the VF's PF->VF
+ * mailboxes.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
+ */
+int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
+
+/**
+ * dlb2_reset_vdev() - reset the hardware owned by a virtual device
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function resets the hardware owned by a vdev, by resetting the vdev's
+ * domains one by one.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function returns whether or not the vdev's resource assignments are
+ * locked. If locked, no resources can be added to or subtracted from the
+ * group.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_lock_vdev() - lock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sets a flag indicating that the vdev is using its resources.
+ * When the vdev is locked, its resource assignment cannot be changed.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_unlock_vdev() - unlock the vdev's resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function unlocks the vdev's resource assignment, allowing it to be
+ * modified.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ */
+void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of scheduling domains to assign to this vdev
+ *
+ * This function assigns num scheduling domains to the specified vdev. If the
+ * vdev already has domains assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB queues to assign to this vdev
+ *
+ * This function assigns num LDB queues to the specified vdev. If the vdev
+ * already has LDB queues assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports to the specified vdev. If the vdev
+ * already has LDB ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @cos: class-of-service ID
+ * @num: number of LDB ports to assign to this vdev
+ *
+ * This function assigns num LDB ports from class-of-service cos to the
+ * specified vdev. If the vdev already has LDB ports from this class-of-service
+ * assigned, this existing assignment is adjusted accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
+				   u32 id,
+				   u32 cos,
+				   u32 num);
+
+/**
+ * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR ports to assign to this vdev
+ *
+ * This function assigns num DIR ports to the specified vdev. If the vdev
+ * already has DIR ports assigned, this existing assignment is adjusted
+ * accordingly.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of LDB credit credits to assign to this vdev
+ *
+ * This function assigns num LDB credit to the specified vdev. If the vdev
+ * already has LDB credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of DIR credits to assign to this vdev
+ *
+ * This function assigns num DIR credit to the specified vdev. If the vdev
+ * already has DIR credits assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of credits, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of history list entries to assign to this vdev
+ *
+ * This function assigns num history list entries to the specified vdev. If the
+ * vdev already has history list entries assigned, this existing assignment is
+ * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
+ * this function may fail if a sufficiently large contiguous chunk is not
+ * available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ * @num: number of atomic inflights to assign to this vdev
+ *
+ * This function assigns num atomic inflights to the specified vdev. If the vdev
+ * already has atomic inflights assigned, this existing assignment is adjusted
+ * accordingly. vdevs are assigned a contiguous chunk of entries, so this
+ * function may fail if a sufficiently large contiguous chunk is not available.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid, or the requested number of resources are
+ *	    unavailable.
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
+
+/**
+ * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function takes any resources currently assigned to the vdev and
+ * reassigns them to the PF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - id is invalid
+ * EPERM  - The vdev's resource assignment is locked and cannot be changed.
+ */
+int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_notify_vf() - send an alarm to a VF
+ * @hw: dlb2_hw handle for a particular device.
+ * @vf_id: VF ID
+ * @notification: notification
+ *
+ * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
+ *
+ * Return:
+ * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
+ */
+int dlb2_notify_vf(struct dlb2_hw *hw,
+		   unsigned int vf_id,
+		   u32 notification);
+
+/**
+ * dlb2_vdev_in_use() - query whether a virtual device is in use
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual device ID
+ *
+ * This function sends a mailbox request to the vdev to query whether the vdev
+ * is in use.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
+ * an internal error occurs.
+ */
+int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_request,
+				unsigned int vdev_id);
+
+enum dlb2_virt_mode {
+	DLB2_VIRT_NONE,
+	DLB2_VIRT_SRIOV,
+	DLB2_VIRT_SIOV,
+
+	/* NUM_DLB2_VIRT_MODES must be last */
+	NUM_DLB2_VIRT_MODES,
+};
+
+/**
+ * dlb2_hw_set_virt_mode() - set the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: either none, SR-IOV, or Scalable IOV.
+ *
+ * This function sets the virtualization mode of the device. This controls
+ * whether the device uses a software or hardware mailbox.
+ *
+ * This should be called by the PF driver when either SR-IOV or Scalable IOV is
+ * selected as the virtualization mechanism, and by the VF/VDEV driver during
+ * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
+ *
+ * Errors:
+ * EINVAL - Invalid mode.
+ */
+int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
+
+/**
+ * dlb2_hw_get_virt_mode() - get the device's virtualization mode
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function gets the virtualization mode of the device.
+ */
+enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: virtual port ID.
+ * @vdev_id: vdev ID.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
+				 u32 id,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_register_sw_mbox() - register a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
+ * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
+ * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
+ * @inject_arg: user argument for pf2vdev_inject callback.
+ *
+ * When Scalable IOV is enabled, the VDCM must register a software mailbox for
+ * every virtual device during vdev creation.
+ *
+ * This function notifies the driver to use a software mailbox using the
+ * provided pointers, instead of the device's hardware mailbox. When the driver
+ * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
+ * go to the software mailbox instead of the hardware one. This is used in
+ * Scalable IOV virtualization.
+ */
+void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
+			      unsigned int vdev_id,
+			      u32 *vdev2pf_mbox,
+			      u32 *pf2vdev_mbox,
+			      void (*pf2vdev_inject)(void *),
+			      void *inject_arg);
+
+/**
+ * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ *
+ * This function notifies the driver to stop using a previously registered
+ * software mailbox.
+ */
+void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
+
+/**
+ * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ * @addr_lo: least-significant 32 bits of address.
+ * @data: 32 data bits.
+ *
+ * This sets up the CQ's IMS entry with the provided address and data values.
+ * This function should only be called if the device is configured for Scalable
+ * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
+ * not needed.
+ */
+void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb,
+				u32 addr_lo,
+				u32 data);
+
+/**
+ * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @virt_cq_id: virtual CQ ID.
+ * @is_ldb: CQ is load-balanced.
+ *
+ * This clears the CQ's IMS entry, reverting it to its reset state.
+ */
+void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
+				unsigned int vdev_id,
+				u32 virt_cq_id,
+				bool is_ldb);
+
+/**
+ * dlb2_hw_register_pasid() - register a vdev's PASID
+ * @hw: dlb2_hw handle for a particular device.
+ * @vdev_id: vdev ID.
+ * @pasid: the vdev's PASID.
+ *
+ * This function stores the user-supplied PASID, and uses it when configuring
+ * the vdev's CQs.
+ *
+ * Return:
+ * Returns >= 0 upon success, -1 otherwise.
+ */
+int dlb2_hw_register_pasid(struct dlb2_hw *hw,
+			   unsigned int vdev_id,
+			   unsigned int pasid);
+
+/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vf_request,
+				unsigned int vf_id);
+
+/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ *	to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ *	port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ *	    cause the total bandwidth across all classes of service to exceed
+ *	    100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+enum dlb2_wd_tmo {
+	/* 40s watchdog timeout */
+	DLB2_WD_TMO_40S,
+	/* 10s watchdog timeout */
+	DLB2_WD_TMO_10S,
+	/* 1s watchdog timeout */
+	DLB2_WD_TMO_1S,
+
+	/* Must be last */
+	NUM_DLB2_WD_TMOS,
+};
+
+/**
+ * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
+ *	caller-specified timeout.
+ * @hw: dlb2_hw handle for a particular device.
+ * @tmo: watchdog timeout.
+ *
+ * This function should be called during device initialization and after reset.
+ * The watchdog timer interrupt must also be enabled per-CQ, using either
+ * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid timeout.
+ */
+int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
+
+/**
+ * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid directed port ID.
+ */
+int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
+ *	individual CQ.
+ * @hw: dlb2_hw handle for a particular device.
+ * @id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid load-balanced port ID.
+ */
+int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
+				 u32 id,
+				 bool vdev_req,
+				 unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *	ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
+ * @hw: dlb2_hw handle for a particular device.
+ * @weight: 8-entry array of arbiter weights.
+ *
+ * weight[N] programs priority N's weight. In cases where the 8 priorities are
+ * reduced to 4 bins, the mapping is:
+ * - weight[1] programs bin 0
+ * - weight[3] programs bin 1
+ * - weight[5] programs bin 2
+ * - weight[7] programs bin 3
+ */
+void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * load-balanced CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_ldb_cq_interrupt().
+ */
+void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical load-balanced port ID.
+ *
+ * This function returns whether the load-balanced CQ interrupt is enabled.
+ */
+int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
+
+/**
+ * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: physical directed port ID.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
+ *
+ * This function can be used to disable (MODE_DIS) and re-enable the
+ * directed CQ's interrupt. It should only be called after the interrupt
+ * has been configured with dlb2_configure_dir_cq_interrupt().
+ */
+void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
+				       int port_id,
+				       int mode);
+
+#endif /* __DLB2_RESOURCE_H */
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
new file mode 100644
index 0000000..1c275ff
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -0,0 +1,620 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "base/dlb2_resource.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_regs.h"
+#include "dlb2_main.h"
+#include "../dlb2_user.h"
+#include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
+#include "../dlb2_inline_fns.h"
+
+#define PF_ID_ZERO 0	/* PF ONLY! */
+#define NO_OWNER_VF 0	/* PF ONLY! */
+#define NOT_VF_REQ false /* PF ONLY! */
+
+#define DLB2_PCI_CFG_SPACE_SIZE 256
+#define DLB2_PCI_CAP_POINTER 0x34
+#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
+#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
+#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
+#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
+#define DLB2_PCI_EXT_CAP_ID_ERR 1
+#define DLB2_PCI_ERR_UNCOR_MASK 8
+#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
+
+#define DLB2_PCI_EXP_DEVCTL 8
+#define DLB2_PCI_LNKCTL 16
+#define DLB2_PCI_SLTCTL 24
+#define DLB2_PCI_RTCTL 28
+#define DLB2_PCI_EXP_DEVCTL2 40
+#define DLB2_PCI_LNKCTL2 48
+#define DLB2_PCI_SLTCTL2 56
+#define DLB2_PCI_CMD 4
+#define DLB2_PCI_X_CMD 2
+#define DLB2_PCI_EXP_DEVSTA 10
+#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
+#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
+
+#define DLB2_PCI_CAP_ID_EXP       0x10
+#define DLB2_PCI_CAP_ID_MSIX      0x11
+#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
+#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
+#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
+
+#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
+#define DLB2_PCI_PRI_ALLOC_REQ           0xC
+#define DLB2_PCI_PRI_CTRL                0x4
+#define DLB2_PCI_MSIX_FLAGS              0x2
+#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
+#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
+#define DLB2_PCI_ERR_ROOT_STATUS         0x30
+#define DLB2_PCI_ERR_COR_STATUS          0x10
+#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
+#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
+#define DLB2_PCI_ACS_CAP                 0x4
+#define DLB2_PCI_ACS_CTRL                0x6
+#define DLB2_PCI_ACS_SV                  0x1
+#define DLB2_PCI_ACS_RR                  0x4
+#define DLB2_PCI_ACS_CR                  0x8
+#define DLB2_PCI_ACS_UF                  0x10
+#define DLB2_PCI_ACS_EC                  0x20
+
+static int
+dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint32_t hdr;
+	size_t sz;
+	int pos;
+
+	pos = DLB2_PCI_CFG_SPACE_SIZE;
+	sz = sizeof(hdr);
+
+	while (pos > 0xFF) {
+		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
+			return -1;
+
+		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
+			return pos;
+
+		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
+{
+	uint8_t pos;
+	int ret;
+	uint16_t hdr;
+
+	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
+	pos &= 0xFC;
+
+	if (ret != 1)
+		return -1;
+
+	while (pos > 0x3F) {
+		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
+		if (ret != 2)
+			return -1;
+
+		if (DLB2_PCI_CAP_ID(hdr) == id)
+			return pos;
+
+		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
+			return -1;
+
+		pos = DLB2_PCI_CAP_NEXT(hdr);
+	}
+
+	return -1;
+}
+
+static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+	int i;
+
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIR64B))
+		dlb2_dev->enqueue_four = dlb2_movdir64b;
+	else
+		dlb2_dev->enqueue_four = dlb2_movntdq;
+
+	/* Initialize software state */
+	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++)
+		dlb2_list_init_head(&dlb2_dev->ldb_port_pages[i].list);
+
+	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++)
+		dlb2_list_init_head(&dlb2_dev->dir_port_pages[i].list);
+
+	rte_spinlock_init(&dlb2_dev->resource_mutex);
+	rte_spinlock_init(&dlb2_dev->measurement_lock);
+
+	return 0;
+}
+
+static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
+{
+	u32 retries = 0;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
+		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+		union dlb2_cfg_mstr_cfg_pm_status pm_st;
+		u32 addr;
+
+		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+			break;
+
+		rte_delay_ms(1);
+	};
+
+	if (retries == DLB2_READY_RETRY_LIMIT) {
+		printf("[%s()] wait for device ready timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	return 0;
+}
+
+struct dlb2_dev *
+dlb2_probe(struct rte_pci_device *pdev)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret = 0;
+
+	DLB2_INFO(dlb2_dev, "probe\n");
+
+	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
+			      RTE_CACHE_LINE_SIZE);
+
+	if (!dlb2_dev) {
+		ret = -ENOMEM;
+		goto dlb2_dev_malloc_fail;
+	}
+
+	/* PCI Bus driver has already mapped bar space into process.
+	 * Save off our IO register and FUNC addresses.
+	 */
+
+	/* BAR 0 */
+	if (pdev->mem_resource[0].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
+	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.func_kva,
+		  (void *)dlb2_dev->hw.func_phys_addr,
+		  (void *)(pdev->mem_resource[0].len));
+
+	/* BAR 2 */
+	if (pdev->mem_resource[2].addr == NULL) {
+		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
+		ret = -EINVAL;
+		goto pci_mmap_bad_addr;
+	}
+	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
+	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
+
+	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
+		  (void *)dlb2_dev->hw.csr_kva,
+		  (void *)dlb2_dev->hw.csr_phys_addr,
+		  (void *)(pdev->mem_resource[2].len));
+
+	dlb2_dev->pdev = pdev;
+
+	/* PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb2_pf_enable_pm(dlb2_dev);
+
+	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb2_pf_reset(dlb2_dev);
+	if (ret)
+		goto dlb2_reset_fail;
+
+	ret = dlb2_pf_init_driver_state(dlb2_dev);
+	if (ret)
+		goto init_driver_state_fail;
+
+	ret = dlb2_resource_init(&dlb2_dev->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	return dlb2_dev;
+
+resource_init_fail:
+	dlb2_resource_free(&dlb2_dev->hw);
+init_driver_state_fail:
+dlb2_reset_fail:
+pci_mmap_bad_addr:
+wait_for_device_ready_fail:
+	rte_free(dlb2_dev);
+dlb2_dev_malloc_fail:
+	rte_errno = ret;
+	return NULL;
+}
+
+int
+dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
+{
+	int ret = 0;
+	int i = 0;
+	uint32_t dword[16];
+	uint16_t cmd;
+	off_t off;
+
+	uint16_t dev_ctl_word;
+	uint16_t dev_ctl2_word;
+	uint16_t lnk_word;
+	uint16_t lnk_word2;
+	uint16_t slt_word;
+	uint16_t slt_word2;
+	uint16_t rt_ctl_word;
+	uint32_t pri_reqs_dword;
+	uint16_t pri_ctrl_word;
+
+	int pcie_cap_offset;
+	int pri_cap_offset;
+	int msix_cap_offset;
+	int err_cap_offset;
+	int acs_cap_offset;
+	int wait_count;
+
+	uint16_t devsta_busy_word;
+	uint16_t devctl_word;
+
+	struct rte_pci_device *pdev = dlb2_dev->pdev;
+
+	/* Save PCI config state */
+
+	for (i = 0; i < 16; i++) {
+		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
+			return ret;
+	}
+
+	pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
+
+	if (pcie_cap_offset < 0) {
+		printf("[%s()] failed to find the pcie capability\n",
+		       __func__);
+		return pcie_cap_offset;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
+		dev_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
+		lnk_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
+		slt_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_RTCTL;
+	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
+		rt_ctl_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
+		dev_ctl2_word = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
+		lnk_word2 = 0;
+
+	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
+		slt_word2 = 0;
+
+	off = DLB2_PCI_EXT_CAP_ID_PRI;
+	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (pri_cap_offset >= 0) {
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
+			pri_reqs_dword = 0;
+	}
+
+	/* clear the PCI command register before issuing the FLR */
+
+	off = DLB2_PCI_CMD;
+	cmd = 0;
+	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+		printf("[%s()] failed to write the pci command\n",
+		       __func__);
+		return ret;
+	}
+
+	/* issue the FLR */
+	for (wait_count = 0; wait_count < 4; wait_count++) {
+		int sleep_time;
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
+		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to read the pci device status\n",
+			       __func__);
+			return ret;
+		}
+
+		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
+			break;
+
+		sleep_time = (1 << (wait_count)) * 100;
+		rte_delay_ms(sleep_time);
+	}
+
+	if (wait_count == 4) {
+		printf("[%s()] wait for pci pending transactions timed out\n",
+		       __func__);
+		return -1;
+	}
+
+	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to read the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
+
+	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
+	if (ret != 2) {
+		printf("[%s()] failed to write the pcie device control\n",
+		       __func__);
+		return ret;
+	}
+
+	rte_delay_ms(100);
+
+	/* Restore PCI config state */
+
+	if (pcie_cap_offset >= 0) {
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
+		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie device control at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
+		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
+		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_RTCTL;
+		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
+		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
+		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
+		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	if (pri_cap_offset >= 0) {
+		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
+
+		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
+		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
+		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ERR;
+	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (err_cap_offset >= 0) {
+		uint32_t tmp;
+
+		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
+		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
+			tmp = 0;
+
+		ret = rte_pci_write_config(pdev, &tmp, 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	for (i = 16; i > 0; i--) {
+		off = (i - 1) * 4;
+		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
+		if (ret != 4) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	off = DLB2_PCI_CMD;
+	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
+		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+			printf("[%s()] failed to write the pci command\n",
+			       __func__);
+			return ret;
+		}
+	}
+
+	msix_cap_offset = dlb2_pci_find_capability(pdev,
+						   DLB2_PCI_CAP_ID_MSIX);
+	if (msix_cap_offset >= 0) {
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
+			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+
+		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
+		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
+			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
+			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
+				printf("[%s()] failed to write msix flags\n",
+				       __func__);
+				return ret;
+			}
+		}
+	}
+
+	off = DLB2_PCI_EXT_CAP_ID_ACS;
+	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
+
+	if (acs_cap_offset >= 0) {
+		uint16_t acs_cap, acs_ctrl, acs_mask;
+		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
+		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
+			acs_cap = 0;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
+		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
+		acs_ctrl |= (acs_cap & acs_mask);
+
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
+			acs_ctrl = 0;
+
+		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
+		acs_mask |= DLB2_PCI_ACS_EC;
+		acs_ctrl &= ~acs_mask;
+
+		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
+		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
+		if (ret != 2) {
+			printf("[%s()] failed to write the pcie config space at offset %d\n",
+				__func__, (int)off);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+/**********************************/
+/****** Device configuration ******/
+/**********************************/
+
diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
new file mode 100644
index 0000000..a914077
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_main.h
@@ -0,0 +1,107 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_spinlock.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#ifndef PAGE_SIZE
+#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
+#endif
+
+#include "base/dlb2_hw_types.h"
+#include "../dlb2_user.h"
+
+#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
+
+struct dlb2_dev;
+
+struct dlb2_port_memory {
+	struct dlb2_list_head list;
+	void *cq_base;
+	bool valid;
+};
+
+struct dlb2_dev {
+	struct rte_pci_device *pdev;
+	struct dlb2_hw hw;
+	/* struct list_head list; */
+	struct device *dlb2_device;
+	struct dlb2_port_memory ldb_port_pages[DLB2_MAX_NUM_LDB_PORTS];
+	struct dlb2_port_memory dir_port_pages[DLB2_MAX_NUM_DIR_PORTS];
+	/* The enqueue_four function enqueues four HCWs (one cache-line worth)
+	 * to the DLB2, using whichever mechanism is supported by the platform
+	 * on which this driver is running.
+	 */
+	void (*enqueue_four)(void *qe4, void *pp_addr);
+
+	bool domain_reset_failed;
+	/* The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	rte_spinlock_t resource_mutex;
+	rte_spinlock_t measurement_lock;
+	bool worker_launched;
+	u8 revision;
+};
+
+struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+/* The following functions were pf_ops in kernel driver implementation */
+int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
+int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_start_domain(struct dlb2_hw *hw,
+			 u32 domain_id,
+			 struct dlb2_start_domain_args *args,
+			 struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_ldb_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_ldb_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_enable_dir_port_args *args,
+			    struct dlb2_cmd_response *resp);
+int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_disable_dir_port_args *args,
+			     struct dlb2_cmd_response *resp);
+int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
+int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+				     u32 domain_id,
+				     u32 port_id);
+
+#endif /* __DLB2_MAIN_H */
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
new file mode 100644
index 0000000..8c5ec20
--- /dev/null
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -0,0 +1,251 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <sys/time.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_devargs.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_pci.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "../dlb2_priv.h"
+#include "../dlb2_iface.h"
+#include "../dlb2_inline_fns.h"
+#include "dlb2_main.h"
+#include "base/dlb2_hw_types.h"
+#include "base/dlb2_osdep.h"
+#include "base/dlb2_resource.h"
+
+extern struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
+
+#if !defined RTE_ARCH_X86_64
+#error "This implementation only supports RTE_ARCH_X86_64 architecture."
+#endif
+
+static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
+
+static void
+dlb2_pf_low_level_io_init(void)
+{
+	int i;
+	/* Addresses will be initialized at port create */
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		/* First directed ports */
+		dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
+
+		/* Now load balanced ports */
+		dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
+		dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
+	}
+}
+
+static int
+dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
+{
+	RTE_SET_USED(handle);
+	RTE_SET_USED(name);
+
+	return 0;
+}
+
+static int
+dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
+			   uint8_t *revision)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	*revision = dlb2_dev->revision;
+
+	return 0;
+}
+
+static void
+dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+	dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_num_resources_args *rsrcs)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+
+	return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
+}
+
+static int
+dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
+			 enum dlb2_cq_poll_modes *mode)
+{
+	RTE_SET_USED(handle);
+
+	*mode = DLB2_CQ_POLL_MODE_SPARSE;
+
+	return 0;
+}
+
+static void
+dlb2_pf_iface_fn_ptrs_init(void)
+{
+
+	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
+	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
+	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
+	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
+	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+}
+
+/* PCI DEV HOOKS */
+static int
+dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
+{
+	int ret = 0;
+	struct rte_pci_device *pci_dev;
+	struct dlb2_devargs dlb2_args = {
+		.socket_id = rte_socket_id(),
+		.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
+		.num_dir_credits_override = -1,
+		.qid_depth_thresholds = { {0} },
+		.cos_id = DLB2_COS_DEFAULT
+	};
+	struct dlb2_eventdev *dlb2;
+
+	DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
+		     eventdev->data->dev_id, eventdev->data->socket_id);
+
+	dlb2_pf_iface_fn_ptrs_init();
+
+	pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
+
+		/* Probe the DLB2 PF layer */
+		dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
+
+		if (dlb2->qm_instance.pf_dev == NULL) {
+			DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
+				     rte_errno);
+			ret = -rte_errno;
+			goto dlb2_probe_failed;
+		}
+
+		/* Were we invoked with runtime parameters? */
+		if (pci_dev->device.devargs) {
+			ret = dlb2_parse_params(pci_dev->device.devargs->args,
+						pci_dev->device.devargs->name,
+						&dlb2_args);
+			if (ret) {
+				DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
+					     ret, rte_errno);
+				goto dlb2_probe_failed;
+			}
+		}
+
+		ret = dlb2_primary_eventdev_probe(eventdev,
+						  event_dlb2_pf_name,
+						  &dlb2_args);
+	} else {
+		ret = dlb2_secondary_eventdev_probe(eventdev,
+						    event_dlb2_pf_name);
+	}
+	if (ret)
+		goto dlb2_probe_failed;
+
+	DLB2_LOG_INFO("DLB2 PF Probe success\n");
+
+	return 0;
+
+dlb2_probe_failed:
+
+	DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
+
+	return ret;
+}
+
+#define EVENTDEV_INTEL_VENDOR_ID 0x8086
+
+static const struct rte_pci_id pci_id_dlb2_map[] = {
+	{
+		RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
+			       PCI_DEVICE_ID_INTEL_DLB2_PF)
+	},
+	{
+		.vendor_id = 0,
+	},
+};
+
+static int
+event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
+		     struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
+					     sizeof(struct dlb2_eventdev),
+					     dlb2_eventdev_pci_init,
+					     event_dlb2_pf_name);
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+}
+
+static int
+event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
+{
+	int ret;
+
+	ret = rte_event_pmd_pci_remove(pci_dev, NULL);
+
+	if (ret) {
+		DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
+				"ret=%d\n", ret);
+	}
+
+	return ret;
+
+}
+
+static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
+	.id_table = pci_id_dlb2_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe = event_dlb2_pci_probe,
+	.remove = event_dlb2_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
-- 
2.6.4

Re: [dpdk-dev] [PATCH 06/22] event/dlb2: add probe

[Eads, Gage]

> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> new file mode 100644
> index 0000000..7ff7dac
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2.c
> @@ -0,0 +1,557 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <assert.h>
> +#include <errno.h>
> +#include <nmmintrin.h>
> +#include <pthread.h>
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <sys/mman.h>
> +#include <sys/fcntl.h>
> +
> +#include <rte_common.h>
> +#include <rte_config.h>
> +#include <rte_cycles.h>
> +#include <rte_debug.h>
> +#include <rte_dev.h>
> +#include <rte_errno.h>
> +#include <rte_eventdev.h>
> +#include <rte_eventdev_pmd.h>
> +#include <rte_kvargs.h>
> +#include <rte_io.h>

Nit: flip the order of rte_kvargs.h and rte_io.h

> +#include <rte_log.h>
> +#include <rte_malloc.h>
> +#include <rte_mbuf.h>
> +#include <rte_prefetch.h>
> +#include <rte_ring.h>
> +#include <rte_string_fns.h>
> +
> +#include "dlb2_priv.h"
> +#include "dlb2_iface.h"
> +#include "dlb2_inline_fns.h"
> +
> +#if !defined RTE_ARCH_X86_64
> +#error "This implementation only supports RTE_ARCH_X86_64 architecture."
> +#endif

In patch #1, meson.build allows RTE_ARCH_X86 -- should that be removed?

> +
> +/*
> + * Resources exposed to eventdev. Some values overridden at runtime using
> + * values returned by the DLB kernel driver.
> + */
> +#if (RTE_EVENT_MAX_QUEUES_PER_DEV > UINT8_MAX)
> +#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member
> max_event_queues"
> +#endif
> +static struct rte_event_dev_info evdev_dlb2_default_info = {
> +	.driver_name = "", /* probe will set */
> +	.min_dequeue_timeout_ns = DLB2_MIN_DEQUEUE_TIMEOUT_NS,
> +	.max_dequeue_timeout_ns = DLB2_MAX_DEQUEUE_TIMEOUT_NS,
> +#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB2_MAX_NUM_LDB_QUEUES)
> +	.max_event_queues = RTE_EVENT_MAX_QUEUES_PER_DEV,
> +#else
> +	.max_event_queues = DLB2_MAX_NUM_LDB_QUEUES,
> +#endif
> +	.max_event_queue_flows = DLB2_MAX_NUM_FLOWS,
> +	.max_event_queue_priority_levels = DLB2_QID_PRIORITIES,
> +	.max_event_priority_levels = DLB2_QID_PRIORITIES,
> +	.max_event_ports = DLB2_MAX_NUM_LDB_PORTS,
> +	.max_event_port_dequeue_depth = DLB2_MAX_CQ_DEPTH,
> +	.max_event_port_enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH,
> +	.max_event_port_links = DLB2_MAX_NUM_QIDS_PER_LDB_CQ,
> +	.max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
> +	.max_single_link_event_port_queue_pairs =
> DLB2_MAX_NUM_DIR_PORTS,
> +	.event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS |
> +			  RTE_EVENT_DEV_CAP_EVENT_QOS |
> +			  RTE_EVENT_DEV_CAP_BURST_MODE |
> +			  RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED |
> +			  RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE |
> +			  RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES),
> +};
> +
> +/* These functions will vary based on processor capabilities */
> +static struct dlb2_port_low_level_io_functions qm_mmio_fns;
> +
> +struct process_local_port_data
> +dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
> +
> +/* override defaults with value(s) provided on command line */
> +static void
> +dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
> +				 int *qid_depth_thresholds)
> +{
> +	int q;
> +
> +	for (q = 0; q < DLB2_MAX_NUM_QUEUES; q++) {
> +		if (qid_depth_thresholds[q] != 0)
> +			dlb2->ev_queues[q].depth_threshold =
> +				qid_depth_thresholds[q];
> +	}
> +}
> +
> +static int
> +dlb2_hw_query_resources(struct dlb2_eventdev *dlb2)
> +{
> +	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
> +	struct dlb2_hw_resource_info *dlb2_info = &handle->info;
> +	int ret;
> +
> +	/* Query driver resources provisioned for this VF */

"VF" -> "device"?

> +
> +	ret = dlb2_iface_get_num_resources(handle,
> +					   &dlb2->hw_rsrc_query_results);
> +	if (ret) {
> +		DLB2_LOG_ERR("ioctl get dlb2 num resources, err=%d\n",
> +			     ret);

Nit: this could be a one-liner

> +		return ret;
> +	}
> +
> +	/* Complete filling in device resource info returned to evdev app,
> +	 * overriding any default values.
> +	 * The capabilities (CAPs) were set at compile time.
> +	 */
> +
> +	evdev_dlb2_default_info.max_event_queues =
> +		dlb2->hw_rsrc_query_results.num_ldb_queues;
> +
> +	evdev_dlb2_default_info.max_event_ports =
> +		dlb2->hw_rsrc_query_results.num_ldb_ports;
> +
> +	evdev_dlb2_default_info.max_num_events =
> +		dlb2->hw_rsrc_query_results.num_ldb_credits;
> +
> +	/* Save off values used when creating the scheduling domain. */
> +
> +	handle->info.num_sched_domains =
> +		dlb2->hw_rsrc_query_results.num_sched_domains;
> +
> +	handle->info.hw_rsrc_max.nb_events_limit =
> +		dlb2->hw_rsrc_query_results.num_ldb_credits;
> +
> +	handle->info.hw_rsrc_max.num_queues =
> +		dlb2->hw_rsrc_query_results.num_ldb_queues +
> +		dlb2->hw_rsrc_query_results.num_dir_ports;
> +
> +	handle->info.hw_rsrc_max.num_ldb_queues =
> +		dlb2->hw_rsrc_query_results.num_ldb_queues;
> +
> +	handle->info.hw_rsrc_max.num_ldb_ports =
> +		dlb2->hw_rsrc_query_results.num_ldb_ports;
> +
> +	handle->info.hw_rsrc_max.num_dir_ports =
> +		dlb2->hw_rsrc_query_results.num_dir_ports;
> +
> +	handle->info.hw_rsrc_max.reorder_window_size =
> +		dlb2->hw_rsrc_query_results.num_hist_list_entries;
> +
> +	rte_memcpy(dlb2_info, &handle->info.hw_rsrc_max,
> sizeof(*dlb2_info));
> +
> +	return 0;
> +}
> +
> +static void
> +dlb2_qm_mmio_fn_init(void)
> +{
> +	/* Process-local function pointers for performing low level port i/o */
> +
> +	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIR64B))
> +		qm_mmio_fns.pp_enqueue_four = dlb2_movdir64b;
> +	else
> +		qm_mmio_fns.pp_enqueue_four = dlb2_movntdq;
> +}
> +
> +#define RTE_BASE_10 10
> +
> +int dlb2_string_to_int(int *result, const char *str)
> +{
> +	long ret;
> +
> +	if (str == NULL || result == NULL)
> +		return -EINVAL;
> +
> +	errno = 0;
> +	ret = strtol(str, NULL, RTE_BASE_10);
> +	if (errno)
> +		return -errno;

Looks like strtol will return 0 if 'str' doesn't contain valid numeric characters,
but won't set errno in this case. See the "No digits were found" case in the
man page's example at the bottom: http://man7.org/linux/man-pages/man3/strtol.3.html

> +
> +	/* long int and int may be different width for some architectures */
> +	if (ret < INT_MIN || ret > INT_MAX)
> +		return -EINVAL;
> +
> +	*result = ret;
> +	return 0;
> +}
> +
> +static int
> +set_numa_node(const char *key __rte_unused, const char *value, void
> *opaque)
> +{
> +	int *socket_id = opaque;
> +	int ret;
> +
> +	ret = dlb2_string_to_int(socket_id, value);
> +	if (ret < 0)
> +		return ret;
> +
> +	if (*socket_id > RTE_MAX_NUMA_NODES)
> +		return -EINVAL;
> +	return 0;
> +}
> +
> +static int
> +set_max_num_events(const char *key __rte_unused,
> +		   const char *value,
> +		   void *opaque)
> +{
> +	int *max_num_events = opaque;
> +	int ret;
> +
> +	if (value == NULL || opaque == NULL) {
> +		DLB2_LOG_ERR("NULL pointer\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = dlb2_string_to_int(max_num_events, value);
> +	if (ret < 0)
> +		return ret;
> +
> +

Nit: extra newline

> +	if (*max_num_events < 0 || *max_num_events >
> +			DLB2_MAX_NUM_LDB_CREDITS) {
> +		DLB2_LOG_ERR("dlb2: max_num_events must be between 0 and
> %d\n",
> +			     DLB2_MAX_NUM_LDB_CREDITS);
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +set_num_dir_credits(const char *key __rte_unused,
> +		    const char *value,
> +		    void *opaque)
> +{
> +	int *num_dir_credits = opaque;
> +	int ret;
> +
> +	if (value == NULL || opaque == NULL) {
> +		DLB2_LOG_ERR("NULL pointer\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = dlb2_string_to_int(num_dir_credits, value);
> +	if (ret < 0)
> +		return ret;
> +
> +	if (*num_dir_credits < 0 ||
> +	    *num_dir_credits > DLB2_MAX_NUM_DIR_CREDITS) {
> +		DLB2_LOG_ERR("dlb2: num_dir_credits must be between 0 and
> %d\n",
> +			     DLB2_MAX_NUM_DIR_CREDITS);
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +set_dev_id(const char *key __rte_unused,
> +	   const char *value,
> +	   void *opaque)
> +{
> +	int *dev_id = opaque;
> +	int ret;
> +
> +	if (value == NULL || opaque == NULL) {
> +		DLB2_LOG_ERR("NULL pointer\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = dlb2_string_to_int(dev_id, value);
> +	if (ret < 0)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static int
> +set_cos(const char *key __rte_unused,
> +	const char *value,
> +	void *opaque)
> +{
> +	enum dlb2_cos *cos_id = opaque;
> +	int x = 0;
> +	int ret;
> +
> +	if (value == NULL || opaque == NULL) {
> +		DLB2_LOG_ERR("NULL pointer\n");
> +		return -EINVAL;
> +	}
> +
> +	ret = dlb2_string_to_int(&x, value);
> +	if (ret < 0)
> +		return ret;
> +
> +	if (x != DLB2_COS_DEFAULT && (x < DLB2_COS_0 || x > DLB2_COS_3)) {
> +		DLB2_LOG_ERR("COS %d out of range, must be 0-3\n", x);

If DLB2_COS_DEFAULT (-1) is a valid value, should this printf reflect that?

> +		return -EINVAL;
> +	}
> +
> +	*cos_id = x;
> +
> +	return 0;
> +}
> +
> +
> +static int
> +set_qid_depth_thresh(const char *key __rte_unused,
> +		     const char *value,
> +		     void *opaque)
> +{
> +	struct dlb2_qid_depth_thresholds *qid_thresh = opaque;
> +	int first, last, thresh, i;
> +
> +	if (value == NULL || opaque == NULL) {
> +		DLB2_LOG_ERR("NULL pointer\n");
> +		return -EINVAL;
> +	}
> +
> +	/* command line override may take one of the following 3 forms:
> +	 * qid_depth_thresh=all:<threshold_value> ... all queues
> +	 * qid_depth_thresh=qidA-qidB:<threshold_value> ... a range of queues
> +	 * qid_depth_thresh=qid:<threshold_value> ... just one queue
> +	 */
> +	if (sscanf(value, "all:%d", &thresh) == 1) {
> +		first = 0;
> +		last = DLB2_MAX_NUM_QUEUES - 1;
> +	} else if (sscanf(value, "%d-%d:%d", &first, &last, &thresh) == 3) {
> +		/* we have everything we need */
> +	} else if (sscanf(value, "%d:%d", &first, &thresh) == 2) {
> +		last = first;
> +	} else {
> +		DLB2_LOG_ERR("Error parsing qid depth vdev arg. Should be
> all:val, qid-qid:val, or qid:val\n");

"vdev arg" -> "devarg"

> +		return -EINVAL;
> +	}
> +
> +	if (first > last || first < 0 || last >= DLB2_MAX_NUM_QUEUES) {
> +		DLB2_LOG_ERR("Error parsing qid depth vdev arg, invalid qid
> value\n");

Ditto

> +		return -EINVAL;
> +	}
> +
> +	if (thresh < 0 || thresh > DLB2_MAX_QUEUE_DEPTH_THRESHOLD) {
> +		DLB2_LOG_ERR("Error parsing qid depth vdev arg, threshold >
> %d\n",

Ditto

> +			     DLB2_MAX_QUEUE_DEPTH_THRESHOLD);
> +		return -EINVAL;
> +	}
> +
> +	for (i = first; i <= last; i++)
> +		qid_thresh->val[i] = thresh; /* indexed by qid */
> +
> +	return 0;
> +}
> +
> +static void
> +dlb2_entry_points_init(struct rte_eventdev *dev)
> +{
> +	RTE_SET_USED(dev);
> +
> +	/* Eventdev PMD entry points */
> +}
> +
> +int
> +dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
> +			    const char *name,
> +			    struct dlb2_devargs *dlb2_args)
> +{
> +	struct dlb2_eventdev *dlb2;
> +	int err, i;
> +
> +	dlb2 = dev->data->dev_private;
> +
> +	dlb2->event_dev = dev; /* backlink */
> +
> +	evdev_dlb2_default_info.driver_name = name;
> +
> +	dlb2->max_num_events_override = dlb2_args->max_num_events;
> +	dlb2->num_dir_credits_override = dlb2_args-
> >num_dir_credits_override;
> +	dlb2->qm_instance.device_path_id = dlb2_args->dev_id;
> +	dlb2->qm_instance.cos_id = dlb2_args->cos_id;
> +
> +	/* Open the interface.
> +	 * For vdev mode, this means open the dlb2 kernel module.
> +	 */
> +	err = dlb2_iface_open(&dlb2->qm_instance, name);
> +	if (err < 0) {
> +		DLB2_LOG_ERR("could not open event hardware device,
> err=%d\n",
> +			     err);
> +		return err;
> +	}
> +
> +	err = dlb2_iface_get_device_version(&dlb2->qm_instance,
> +					    &dlb2->revision);
> +	if (err < 0) {
> +		DLB2_LOG_ERR("dlb2: failed to get the device version,
> err=%d\n",
> +			     err);
> +		return err;
> +	}
> +
> +	err = dlb2_hw_query_resources(dlb2);
> +	if (err) {
> +		DLB2_LOG_ERR("get resources err=%d for %s\n",
> +			     err, name);
> +		return err;
> +	}
> +
> +	dlb2_iface_hardware_init(&dlb2->qm_instance);
> +
> +	err = dlb2_iface_get_cq_poll_mode(&dlb2->qm_instance, &dlb2-
> >poll_mode);
> +	if (err < 0) {
> +		DLB2_LOG_ERR("dlb2: failed to get the poll mode, err=%d\n",
> +			     err);
> +		return err;
> +	}
> +
> +	/* Initialize each port's token pop mode */
> +	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
> +		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
> +
> +	rte_spinlock_init(&dlb2->qm_instance.resource_lock);
> +
> +	dlb2_qm_mmio_fn_init();
> +
> +	dlb2_iface_low_level_io_init();
> +
> +	dlb2_entry_points_init(dev);
> +
> +	dlb2_init_queue_depth_thresholds(dlb2,
> +					 dlb2_args->qid_depth_thresholds.val);
> +
> +	return 0;
> +}
> +
> +int
> +dlb2_secondary_eventdev_probe(struct rte_eventdev *dev,
> +			      const char *name)
> +{
> +	struct dlb2_eventdev *dlb2;
> +	int err;
> +
> +	dlb2 = dev->data->dev_private;
> +
> +	evdev_dlb2_default_info.driver_name = name;
> +
> +	err = dlb2_iface_open(&dlb2->qm_instance, name);
> +	if (err < 0) {
> +		DLB2_LOG_ERR("could not open event hardware device,
> err=%d\n",
> +			     err);
> +		return err;
> +	}
> +
> +	err = dlb2_hw_query_resources(dlb2);
> +	if (err) {
> +		DLB2_LOG_ERR("get resources err=%d for %s\n",
> +			     err, name);
> +		return err;
> +	}
> +
> +	dlb2_qm_mmio_fn_init();
> +
> +	dlb2_iface_low_level_io_init();
> +
> +	dlb2_entry_points_init(dev);
> +
> +	return 0;
> +}
> +
> +int
> +dlb2_parse_params(const char *params,
> +		  const char *name,
> +		  struct dlb2_devargs *dlb2_args)
> +{
> +	int ret = 0;
> +	static const char * const args[] = { NUMA_NODE_ARG,
> +				      DLB2_MAX_NUM_EVENTS,
> +				      DLB2_NUM_DIR_CREDITS,
> +				      DEV_ID_ARG,
> +				      DLB2_QID_DEPTH_THRESH_ARG,
> +				      DLB2_COS_ARG,
> +				      NULL };

Nit: const_args entries alignment

> +
> +	if (params != NULL && params[0] != '\0') {
> +		struct rte_kvargs *kvlist = rte_kvargs_parse(params, args);
> +
> +		if (!kvlist) {
> +			RTE_LOG(INFO, PMD,
> +				"Ignoring unsupported parameters when creating
> device '%s'\n",
> +				name);
> +		} else {
> +			int ret = rte_kvargs_process(kvlist, NUMA_NODE_ARG,
> +						     set_numa_node,
> +						     &dlb2_args->socket_id);
> +			if (ret != 0) {
> +				DLB2_LOG_ERR("%s: Error parsing numa node
> parameter",
> +					     name);
> +				rte_kvargs_free(kvlist);
> +				return ret;
> +			}
> +
> +			ret = rte_kvargs_process(kvlist,
> DLB2_MAX_NUM_EVENTS,
> +						 set_max_num_events,
> +						 &dlb2_args->max_num_events);
> +			if (ret != 0) {
> +				DLB2_LOG_ERR("%s: Error parsing
> max_num_events parameter",
> +					     name);
> +				rte_kvargs_free(kvlist);
> +				return ret;
> +			}
> +
> +			ret = rte_kvargs_process(kvlist,
> +					DLB2_NUM_DIR_CREDITS,
> +					set_num_dir_credits,
> +					&dlb2_args->num_dir_credits_override);
> +			if (ret != 0) {
> +				DLB2_LOG_ERR("%s: Error parsing
> num_dir_credits parameter",
> +					     name);
> +				rte_kvargs_free(kvlist);
> +				return ret;
> +			}
> +
> +			ret = rte_kvargs_process(kvlist, DEV_ID_ARG,
> +						 set_dev_id,
> +						 &dlb2_args->dev_id);
> +			if (ret != 0) {
> +				DLB2_LOG_ERR("%s: Error parsing dev_id
> parameter",
> +					     name);
> +				rte_kvargs_free(kvlist);
> +				return ret;
> +			}
> +
> +			ret = rte_kvargs_process(
> +					kvlist,
> +					DLB2_QID_DEPTH_THRESH_ARG,
> +					set_qid_depth_thresh,
> +					&dlb2_args->qid_depth_thresholds);
> +			if (ret != 0) {
> +				DLB2_LOG_ERR("%s: Error parsing
> qid_depth_thresh parameter",
> +					     name);
> +				rte_kvargs_free(kvlist);
> +				return ret;
> +			}
> +
> +			ret = rte_kvargs_process(kvlist, DLB2_COS_ARG,
> +						 set_cos,
> +						 &dlb2_args->cos_id);
> +			if (ret != 0) {
> +				DLB2_LOG_ERR("%s: Error parsing cos
> parameter",
> +					     name);
> +				rte_kvargs_free(kvlist);
> +				return ret;
> +			}
> +
> +			rte_kvargs_free(kvlist);
> +		}
> +	}
> +	return ret;
> +}
> +RTE_LOG_REGISTER(eventdev_dlb2_log_level, pmd.event.dlb2, NOTICE);
> +

Nit: whitespace at the end of the file

> diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
> new file mode 100644
> index 0000000..fefdf78
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2_iface.c
> @@ -0,0 +1,42 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <stdbool.h>
> +#include <stdint.h>
> +#include <rte_debug.h>
> +#include <rte_bus_pci.h>
> +#include <rte_log.h>
> +#include <rte_dev.h>
> +#include <rte_mbuf.h>
> +#include <rte_ring.h>
> +#include <rte_errno.h>
> +#include <rte_kvargs.h>
> +#include <rte_malloc.h>
> +#include <rte_cycles.h>
> +#include <rte_io.h>
> +#include <rte_eventdev.h>
> +#include <rte_eventdev_pmd.h>

Doesn't look stdbool.h and the rte_*.h includes are needed here -- I was able to build
the full series with them commented out.

[...]

> diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h
> b/drivers/event/dlb2/pf/base/dlb2_osdep.h
> new file mode 100644
> index 0000000..c8d8d5b
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
> @@ -0,0 +1,248 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_OSDEP_H
> +#define __DLB2_OSDEP_H
> +
> +#include <string.h>
> +#include <time.h>
> +#include <unistd.h>
> +#include <pthread.h>
> +
> +#include <rte_string_fns.h>
> +#include <rte_cycles.h>
> +#include <rte_io.h>
> +#include <rte_log.h>
> +#include <rte_spinlock.h>
> +#include "../dlb2_main.h"
> +#include "dlb2_resource.h"
> +#include "../../dlb2_log.h"
> +#include "../../dlb2_user.h"
> +
> +
> +#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
> +#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
> +
> +/* Read/write register 'reg' in the CSR BAR space */
> +#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
> +#define DLB2_CSR_RD(hw, reg) \
> +	DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
> +#define DLB2_CSR_WR(hw, reg, value) \
> +	DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
> +
> +/* Read/write register 'reg' in the func BAR space */
> +#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva +
> (reg)))
> +#define DLB2_FUNC_RD(hw, reg) \
> +	DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
> +#define DLB2_FUNC_WR(hw, reg, value) \
> +	DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
> +
> +/* Map to PMDs logging interface */
> +#define DLB2_ERR(dev, fmt, args...) \
> +	DLB2_LOG_ERR(fmt, ## args)
> +
> +#define DLB2_INFO(dev, fmt, args...) \
> +	DLB2_LOG_INFO(fmt, ## args)
> +
> +#define DLB2_DEBUG(dev, fmt, args...) \
> +	DLB2_LOG_DBG(fmt, ## args)
> +
> +/**
> + * os_udelay() - busy-wait for a number of microseconds
> + * @usecs: delay duration.
> + */
> +static inline void os_udelay(int usecs)
> +{
> +	rte_delay_us(usecs);
> +}
> +
> +/**
> + * os_msleep() - sleep for a number of milliseconds
> + * @usecs: delay duration.
> + */
> +static inline void os_msleep(int msecs)
> +{
> +	rte_delay_ms(msecs);
> +}
> +
> +#define DLB2_PP_BASE(__is_ldb) \
> +	((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
> +
> +/**
> + * os_map_producer_port() - map a producer port into the caller's address
> space
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: port ID
> + * @is_ldb: true for load-balanced port, false for a directed port
> + *
> + * This function maps the requested producer port memory into the caller's
> + * address space.
> + *
> + * Return:
> + * Returns the base address at which the PP memory was mapped, else NULL.
> + */
> +static inline void *os_map_producer_port(struct dlb2_hw *hw,
> +					 u8 port_id,
> +					 bool is_ldb)
> +{
> +	uint64_t addr;
> +	uint64_t pp_dma_base;
> +
> +

Nit: extra whitespace

[...]

> diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
> b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
> new file mode 100644
> index 0000000..7e48878
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
> @@ -0,0 +1,447 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_OSDEP_BITMAP_H
> +#define __DLB2_OSDEP_BITMAP_H
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <unistd.h>
> +#include <rte_bitmap.h>
> +#include <rte_string_fns.h>
> +#include <rte_malloc.h>
> +#include <rte_errno.h>
> +#include "../dlb2_main.h"
> +
> +/*************************/
> +/*** Bitmap operations ***/
> +/*************************/
> +struct dlb2_bitmap {
> +	struct rte_bitmap *map;
> +	unsigned int len;
> +};
> +
> +/**
> + * dlb2_bitmap_alloc() - alloc a bitmap data structure
> + * @bitmap: pointer to dlb2_bitmap structure pointer.
> + * @len: number of entries in the bitmap.
> + *
> + * This function allocates a bitmap and initializes it with length @len. All
> + * entries are initially zero.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or len is 0.
> + * ENOMEM - could not allocate memory for the bitmap data structure.
> + */
> +static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
> +				    unsigned int len)
> +{
> +	struct dlb2_bitmap *bm;
> +	void *mem;
> +	uint32_t alloc_size;
> +	uint32_t nbits = (uint32_t)len;
> +
> +	if (!bitmap || nbits == 0)
> +		return -EINVAL;
> +
> +	/* Allocate DLB2 bitmap control struct */
> +	bm = rte_malloc("DLB2_PF",
> +			sizeof(struct dlb2_bitmap),
> +			RTE_CACHE_LINE_SIZE);
> +
> +	if (!bm)
> +		return -ENOMEM;
> +
> +	/* Allocate bitmap memory */
> +	alloc_size = rte_bitmap_get_memory_footprint(nbits);
> +	mem = rte_malloc("DLB2_PF_BITMAP", alloc_size,
> RTE_CACHE_LINE_SIZE);
> +	if (!mem) {
> +		rte_free(bm);
> +		return -ENOMEM;
> +	}
> +
> +	bm->map = rte_bitmap_init(len, mem, alloc_size);
> +	if (!bm->map) {
> +		rte_free(mem);
> +		rte_free(bm);
> +		return -ENOMEM;
> +	}
> +
> +	bm->len = len;
> +
> +	*bitmap = bm;
> +
> +	return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_free() - free a previously allocated bitmap data structure
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
> + */
> +static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
> +{
> +	if (!bitmap)
> +		return;
> +
> +	rte_free(bitmap->map);
> +	rte_free(bitmap);
> +}
> +
> +/**
> + * dlb2_bitmap_fill() - fill a bitmap with all 1s
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function sets all bitmap values to 1.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
> +{
> +	unsigned int i;
> +
> +	if (!bitmap || !bitmap->map)
> +		return -EINVAL;
> +
> +	/* TODO - optimize */
> +	for (i = 0; i != bitmap->len; i++)
> +		rte_bitmap_set(bitmap->map, i);
> +
> +	return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_fill() - fill a bitmap with all 0s
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function sets all bitmap values to 0.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
> +{
> +	if (!bitmap || !bitmap->map)
> +		return -EINVAL;
> +
> +	rte_bitmap_reset(bitmap->map);
> +
> +	return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_set() - set a bitmap entry
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: bit index.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
> + *	    bitmap length.
> + */
> +static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
> +				  unsigned int bit)
> +{
> +	if (!bitmap || !bitmap->map)
> +		return -EINVAL;
> +
> +	if (bitmap->len <= bit)
> +		return -EINVAL;
> +
> +	rte_bitmap_set(bitmap->map, bit);
> +
> +	return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_set_range() - set a range of bitmap entries
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: starting bit index.
> + * @len: length of the range.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
> + *	    length.
> + */
> +static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
> +					unsigned int bit,
> +					unsigned int len)
> +{
> +	unsigned int i;
> +
> +	if (!bitmap || !bitmap->map)
> +		return -EINVAL;
> +
> +	if (bitmap->len <= bit)
> +		return -EINVAL;
> +
> +	/* TODO - optimize */

Leftover TODO

> +	for (i = 0; i != len; i++)
> +		rte_bitmap_set(bitmap->map, bit + i);
> +
> +	return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_clear() - clear a bitmap entry
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: bit index.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
> + *	    bitmap length.
> + */
> +static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
> +				    unsigned int bit)
> +{
> +	if (!bitmap || !bitmap->map)
> +		return -EINVAL;
> +
> +	if (bitmap->len <= bit)
> +		return -EINVAL;
> +
> +	rte_bitmap_clear(bitmap->map, bit);
> +
> +	return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_clear_range() - clear a range of bitmap entries
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: starting bit index.
> + * @len: length of the range.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
> + *	    length.
> + */
> +static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
> +					  unsigned int bit,
> +					  unsigned int len)
> +{
> +	unsigned int i;
> +
> +	if (!bitmap || !bitmap->map)
> +		return -EINVAL;
> +
> +	if (bitmap->len <= bit)
> +		return -EINVAL;
> +
> +	/* TODO - optimize */

Leftover TODO

> +	for (i = 0; i != len; i++)
> +		rte_bitmap_clear(bitmap->map, bit + i);
> +
> +	return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_find_set_bit_range() - find an range of set bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @len: length of the range.
> + *
> + * This function looks for a range of set bits of length @len.
> + *
> + * Return:
> + * Returns the base bit index upon success, < 0 otherwise.
> + *
> + * Errors:
> + * ENOENT - unable to find a length *len* range of set bits.
> + * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
> + */
> +static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
> +						 unsigned int len)
> +{
> +	unsigned int i, j = 0;
> +
> +	if (!bitmap || !bitmap->map || len == 0)
> +		return -EINVAL;
> +
> +	if (bitmap->len < len)
> +		return -ENOENT;
> +
> +	/* TODO - optimize */

Leftover TODO

> +	for (i = 0; i != bitmap->len; i++) {
> +		if  (rte_bitmap_get(bitmap->map, i)) {
> +			if (++j == len)
> +				return i - j + 1;
> +		} else {
> +			j = 0;
> +		}
> +	}
> +
> +	/* No set bit range of length len? */
> +	return -ENOENT;
> +}
> +
> +/**
> + * dlb2_bitmap_find_set_bit() - find an range of set bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function looks for a single set bit.
> + *
> + * Return:
> + * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
> + */
> +static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
> +{
> +	unsigned int i;
> +
> +	if (!bitmap)
> +		return -EINVAL;
> +
> +	if (!bitmap->map)
> +		return -EINVAL;
> +
> +	/* TODO - optimize */

Leftover TODO

> +	for (i = 0; i != bitmap->len; i++) {
> +		if  (rte_bitmap_get(bitmap->map, i))
> +			return i;
> +	}
> +
> +	return -ENOENT;
> +}
> +
> +/**
> + * dlb2_bitmap_count() - returns the number of set bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function looks for a single set bit.
> + *
> + * Return:
> + * Returns the number of set bits upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
> +{
> +	int weight = 0;
> +	unsigned int i;
> +
> +	if (!bitmap)
> +		return -EINVAL;
> +
> +	if (!bitmap->map)
> +		return -EINVAL;
> +
> +	/* TODO - optimize */

Leftover TODO

> +	for (i = 0; i != bitmap->len; i++) {
> +		if  (rte_bitmap_get(bitmap->map, i))
> +			weight++;
> +	}
> +	return weight;
> +}
> +
> +/**
> + * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
> + *				      bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * Return:
> + * Returns the bitmap's longest contiguous range of of set bits upon success,
> + * <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
> +{
> +	int max_len = 0, len = 0;
> +	unsigned int i;
> +
> +	if (!bitmap)
> +		return -EINVAL;
> +
> +	if (!bitmap->map)
> +		return -EINVAL;
> +
> +	/* TODO - optimize */

Leftover TODO

[...]

> diff --git a/drivers/event/dlb2/pf/dlb2_main.c
> b/drivers/event/dlb2/pf/dlb2_main.c
> new file mode 100644
> index 0000000..1c275ff
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/dlb2_main.c
> @@ -0,0 +1,620 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <errno.h>
> +#include <assert.h>
> +#include <unistd.h>
> +#include <string.h>
> +
> +#include <rte_malloc.h>
> +#include <rte_errno.h>
> +
> +#include "base/dlb2_resource.h"
> +#include "base/dlb2_osdep.h"
> +#include "base/dlb2_regs.h"
> +#include "dlb2_main.h"
> +#include "../dlb2_user.h"
> +#include "../dlb2_priv.h"
> +#include "../dlb2_iface.h"
> +#include "../dlb2_inline_fns.h"
> +
> +#define PF_ID_ZERO 0	/* PF ONLY! */
> +#define NO_OWNER_VF 0	/* PF ONLY! */
> +#define NOT_VF_REQ false /* PF ONLY! */
> +
> +#define DLB2_PCI_CFG_SPACE_SIZE 256
> +#define DLB2_PCI_CAP_POINTER 0x34
> +#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
> +#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
> +#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
> +#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
> +#define DLB2_PCI_EXT_CAP_ID_ERR 1
> +#define DLB2_PCI_ERR_UNCOR_MASK 8
> +#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
> +
> +#define DLB2_PCI_EXP_DEVCTL 8
> +#define DLB2_PCI_LNKCTL 16
> +#define DLB2_PCI_SLTCTL 24
> +#define DLB2_PCI_RTCTL 28
> +#define DLB2_PCI_EXP_DEVCTL2 40
> +#define DLB2_PCI_LNKCTL2 48
> +#define DLB2_PCI_SLTCTL2 56
> +#define DLB2_PCI_CMD 4
> +#define DLB2_PCI_X_CMD 2
> +#define DLB2_PCI_EXP_DEVSTA 10
> +#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
> +#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
> +
> +#define DLB2_PCI_CAP_ID_EXP       0x10
> +#define DLB2_PCI_CAP_ID_MSIX      0x11
> +#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
> +#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
> +#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
> +
> +#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
> +#define DLB2_PCI_PRI_ALLOC_REQ           0xC
> +#define DLB2_PCI_PRI_CTRL                0x4
> +#define DLB2_PCI_MSIX_FLAGS              0x2
> +#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
> +#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
> +#define DLB2_PCI_ERR_ROOT_STATUS         0x30
> +#define DLB2_PCI_ERR_COR_STATUS          0x10
> +#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
> +#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
> +#define DLB2_PCI_ACS_CAP                 0x4
> +#define DLB2_PCI_ACS_CTRL                0x6
> +#define DLB2_PCI_ACS_SV                  0x1
> +#define DLB2_PCI_ACS_RR                  0x4
> +#define DLB2_PCI_ACS_CR                  0x8
> +#define DLB2_PCI_ACS_UF                  0x10
> +#define DLB2_PCI_ACS_EC                  0x20
> +
> +static int
> +dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
> +{
> +	uint32_t hdr;
> +	size_t sz;
> +	int pos;
> +
> +	pos = DLB2_PCI_CFG_SPACE_SIZE;
> +	sz = sizeof(hdr);
> +
> +	while (pos > 0xFF) {
> +		if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
> +			return -1;
> +
> +		if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
> +			return pos;
> +
> +		pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
> +	}
> +
> +	return -1;
> +}
> +
> +static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
> +{
> +	uint8_t pos;
> +	int ret;
> +	uint16_t hdr;
> +
> +	ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
> +	pos &= 0xFC;
> +
> +	if (ret != 1)
> +		return -1;
> +
> +	while (pos > 0x3F) {
> +		ret = rte_pci_read_config(pdev, &hdr, 2, pos);
> +		if (ret != 2)
> +			return -1;
> +
> +		if (DLB2_PCI_CAP_ID(hdr) == id)
> +			return pos;
> +
> +		if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
> +			return -1;
> +
> +		pos = DLB2_PCI_CAP_NEXT(hdr);
> +	}
> +
> +	return -1;
> +}
> +
> +static int
> +dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
> +{
> +	int i;
> +
> +	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIR64B))
> +		dlb2_dev->enqueue_four = dlb2_movdir64b;
> +	else
> +		dlb2_dev->enqueue_four = dlb2_movntdq;
> +
> +	/* Initialize software state */
> +	for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++)
> +		dlb2_list_init_head(&dlb2_dev->ldb_port_pages[i].list);
> +
> +	for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++)
> +		dlb2_list_init_head(&dlb2_dev->dir_port_pages[i].list);
> +
> +	rte_spinlock_init(&dlb2_dev->resource_mutex);
> +	rte_spinlock_init(&dlb2_dev->measurement_lock);
> +
> +	return 0;
> +}
> +
> +static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
> +{
> +	dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
> +}
> +
> +#define DLB2_READY_RETRY_LIMIT 1000
> +static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
> +{
> +	u32 retries = 0;
> +
> +	/* Allow at least 1s for the device to become active after power-on */
> +	for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
> +		union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
> +		union dlb2_cfg_mstr_cfg_pm_status pm_st;
> +		u32 addr;
> +
> +		addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
> +		pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
> +		addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
> +		idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
> +		if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
> +			break;
> +
> +		rte_delay_ms(1);
> +	};
> +
> +	if (retries == DLB2_READY_RETRY_LIMIT) {
> +		printf("[%s()] wait for device ready timed out\n",
> +		       __func__);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +struct dlb2_dev *
> +dlb2_probe(struct rte_pci_device *pdev)
> +{
> +	struct dlb2_dev *dlb2_dev;
> +	int ret = 0;
> +
> +	DLB2_INFO(dlb2_dev, "probe\n");
> +
> +	dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
> +			      RTE_CACHE_LINE_SIZE);
> +
> +	if (!dlb2_dev) {
> +		ret = -ENOMEM;
> +		goto dlb2_dev_malloc_fail;
> +	}
> +
> +	/* PCI Bus driver has already mapped bar space into process.
> +	 * Save off our IO register and FUNC addresses.
> +	 */
> +
> +	/* BAR 0 */
> +	if (pdev->mem_resource[0].addr == NULL) {
> +		DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
> +		ret = -EINVAL;
> +		goto pci_mmap_bad_addr;
> +	}
> +	dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev-
> >mem_resource[0].addr;
> +	dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
> +
> +	DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
> +		  (void *)dlb2_dev->hw.func_kva,
> +		  (void *)dlb2_dev->hw.func_phys_addr,
> +		  (void *)(pdev->mem_resource[0].len));
> +
> +	/* BAR 2 */
> +	if (pdev->mem_resource[2].addr == NULL) {
> +		DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
> +		ret = -EINVAL;
> +		goto pci_mmap_bad_addr;
> +	}
> +	dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev-
> >mem_resource[2].addr;
> +	dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
> +
> +	DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
> +		  (void *)dlb2_dev->hw.csr_kva,
> +		  (void *)dlb2_dev->hw.csr_phys_addr,
> +		  (void *)(pdev->mem_resource[2].len));
> +
> +	dlb2_dev->pdev = pdev;
> +
> +	/* PM enable must be done before any other MMIO accesses, and this
> +	 * setting is persistent across device reset.
> +	 */
> +	dlb2_pf_enable_pm(dlb2_dev);
> +
> +	ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
> +	if (ret)
> +		goto wait_for_device_ready_fail;
> +
> +	ret = dlb2_pf_reset(dlb2_dev);
> +	if (ret)
> +		goto dlb2_reset_fail;
> +
> +	ret = dlb2_pf_init_driver_state(dlb2_dev);
> +	if (ret)
> +		goto init_driver_state_fail;
> +
> +	ret = dlb2_resource_init(&dlb2_dev->hw);
> +	if (ret)
> +		goto resource_init_fail;
> +
> +	return dlb2_dev;
> +
> +resource_init_fail:
> +	dlb2_resource_free(&dlb2_dev->hw);
> +init_driver_state_fail:
> +dlb2_reset_fail:
> +pci_mmap_bad_addr:
> +wait_for_device_ready_fail:
> +	rte_free(dlb2_dev);
> +dlb2_dev_malloc_fail:
> +	rte_errno = ret;
> +	return NULL;
> +}
> +
> +int
> +dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
> +{
> +	int ret = 0;
> +	int i = 0;
> +	uint32_t dword[16];
> +	uint16_t cmd;
> +	off_t off;
> +
> +	uint16_t dev_ctl_word;
> +	uint16_t dev_ctl2_word;
> +	uint16_t lnk_word;
> +	uint16_t lnk_word2;
> +	uint16_t slt_word;
> +	uint16_t slt_word2;
> +	uint16_t rt_ctl_word;
> +	uint32_t pri_reqs_dword;
> +	uint16_t pri_ctrl_word;
> +
> +	int pcie_cap_offset;
> +	int pri_cap_offset;
> +	int msix_cap_offset;
> +	int err_cap_offset;
> +	int acs_cap_offset;
> +	int wait_count;
> +
> +	uint16_t devsta_busy_word;
> +	uint16_t devctl_word;
> +
> +	struct rte_pci_device *pdev = dlb2_dev->pdev;
> +
> +	/* Save PCI config state */
> +
> +	for (i = 0; i < 16; i++) {
> +		if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
> +			return ret;
> +	}
> +
> +	pcie_cap_offset = dlb2_pci_find_capability(pdev,
> DLB2_PCI_CAP_ID_EXP);
> +
> +	if (pcie_cap_offset < 0) {
> +		printf("[%s()] failed to find the pcie capability\n",
> +		       __func__);
> +		return pcie_cap_offset;
> +	}
> +
> +	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> +	if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
> +		dev_ctl_word = 0;
> +
> +	off = pcie_cap_offset + DLB2_PCI_LNKCTL;
> +	if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
> +		lnk_word = 0;
> +
> +	off = pcie_cap_offset + DLB2_PCI_SLTCTL;
> +	if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
> +		slt_word = 0;
> +
> +	off = pcie_cap_offset + DLB2_PCI_RTCTL;
> +	if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
> +		rt_ctl_word = 0;
> +
> +	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
> +	if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
> +		dev_ctl2_word = 0;
> +
> +	off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
> +	if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
> +		lnk_word2 = 0;
> +
> +	off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
> +	if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
> +		slt_word2 = 0;
> +
> +	off = DLB2_PCI_EXT_CAP_ID_PRI;
> +	pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> +
> +	if (pri_cap_offset >= 0) {
> +		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
> +		if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
> +			pri_reqs_dword = 0;
> +	}
> +
> +	/* clear the PCI command register before issuing the FLR */
> +
> +	off = DLB2_PCI_CMD;
> +	cmd = 0;
> +	if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +		printf("[%s()] failed to write the pci command\n",
> +		       __func__);
> +		return ret;
> +	}
> +
> +	/* issue the FLR */
> +	for (wait_count = 0; wait_count < 4; wait_count++) {
> +		int sleep_time;
> +
> +		off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
> +		ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
> +		if (ret != 2) {
> +			printf("[%s()] failed to read the pci device status\n",
> +			       __func__);
> +			return ret;
> +		}
> +
> +		if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
> +			break;
> +
> +		sleep_time = (1 << (wait_count)) * 100;
> +		rte_delay_ms(sleep_time);
> +	}
> +
> +	if (wait_count == 4) {
> +		printf("[%s()] wait for pci pending transactions timed out\n",
> +		       __func__);
> +		return -1;
> +	}
> +
> +	off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> +	ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
> +	if (ret != 2) {
> +		printf("[%s()] failed to read the pcie device control\n",
> +		       __func__);
> +		return ret;
> +	}
> +
> +	devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
> +
> +	ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
> +	if (ret != 2) {
> +		printf("[%s()] failed to write the pcie device control\n",
> +		       __func__);
> +		return ret;
> +	}
> +
> +	rte_delay_ms(100);
> +
> +	/* Restore PCI config state */
> +
> +	if (pcie_cap_offset >= 0) {
> +		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> +		ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
> +		if (ret != 2) {
> +			printf("[%s()] failed to write the pcie device control at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +
> +		off = pcie_cap_offset + DLB2_PCI_LNKCTL;
> +		ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
> +		if (ret != 2) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +
> +		off = pcie_cap_offset + DLB2_PCI_SLTCTL;
> +		ret = rte_pci_write_config(pdev, &slt_word, 2, off);
> +		if (ret != 2) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +
> +		off = pcie_cap_offset + DLB2_PCI_RTCTL;
> +		ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
> +		if (ret != 2) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +
> +		off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
> +		ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
> +		if (ret != 2) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +
> +		off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
> +		ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
> +		if (ret != 2) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +
> +		off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
> +		ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
> +		if (ret != 2) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +	}
> +
> +	if (pri_cap_offset >= 0) {
> +		pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
> +
> +		off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
> +		ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
> +		if (ret != 4) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +
> +		off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
> +		ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
> +		if (ret != 2) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +	}
> +
> +	off = DLB2_PCI_EXT_CAP_ID_ERR;
> +	err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> +
> +	if (err_cap_offset >= 0) {
> +		uint32_t tmp;
> +
> +		off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
> +		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> +			tmp = 0;
> +
> +		ret = rte_pci_write_config(pdev, &tmp, 4, off);
> +		if (ret != 4) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +
> +		off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
> +		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> +			tmp = 0;
> +
> +		ret = rte_pci_write_config(pdev, &tmp, 4, off);
> +		if (ret != 4) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +
> +		off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
> +		if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> +			tmp = 0;
> +
> +		ret = rte_pci_write_config(pdev, &tmp, 4, off);
> +		if (ret != 4) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +	}
> +
> +	for (i = 16; i > 0; i--) {
> +		off = (i - 1) * 4;
> +		ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
> +		if (ret != 4) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +	}
> +
> +	off = DLB2_PCI_CMD;
> +	if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> +		cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
> +		if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +			printf("[%s()] failed to write the pci command\n",
> +			       __func__);
> +			return ret;
> +		}
> +	}
> +
> +	msix_cap_offset = dlb2_pci_find_capability(pdev,
> +						   DLB2_PCI_CAP_ID_MSIX);
> +	if (msix_cap_offset >= 0) {
> +		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
> +		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> +			cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
> +			cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
> +			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +				printf("[%s()] failed to write msix flags\n",
> +				       __func__);
> +				return ret;
> +			}
> +		}
> +
> +		off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
> +		if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> +			cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
> +			if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +				printf("[%s()] failed to write msix flags\n",
> +				       __func__);
> +				return ret;
> +			}
> +		}
> +	}
> +
> +	off = DLB2_PCI_EXT_CAP_ID_ACS;
> +	acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> +
> +	if (acs_cap_offset >= 0) {
> +		uint16_t acs_cap, acs_ctrl, acs_mask;
> +		off = acs_cap_offset + DLB2_PCI_ACS_CAP;
> +		if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
> +			acs_cap = 0;
> +
> +		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> +		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
> +			acs_ctrl = 0;
> +
> +		acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
> +		acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
> +		acs_ctrl |= (acs_cap & acs_mask);
> +
> +		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
> +		if (ret != 2) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +
> +		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> +		if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
> +			acs_ctrl = 0;
> +
> +		acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
> +		acs_mask |= DLB2_PCI_ACS_EC;
> +		acs_ctrl &= ~acs_mask;
> +
> +		off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> +		ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
> +		if (ret != 2) {
> +			printf("[%s()] failed to write the pcie config space at
> offset %d\n",
> +				__func__, (int)off);
> +			return ret;
> +		}
> +	}
> +
> +	return 0;
> +}

Since this PCIe function-level reset code is more PCIe-specific than the other functions
in this file, what do you think about putting code this PCIe function-level reset code in
its own file, e.g. dlb2_pci.c? Just a thought.

> +
> +/**********************************/
> +/****** Device configuration ******/
> +/**********************************/
> +

Nit: this comment block would make more sense in patch 8 when the first device
configuration functions are added.

> diff --git a/drivers/event/dlb2/pf/dlb2_main.h
> b/drivers/event/dlb2/pf/dlb2_main.h
> new file mode 100644
> index 0000000..a914077
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/dlb2_main.h
> @@ -0,0 +1,107 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_MAIN_H
> +#define __DLB2_MAIN_H
> +
> +#include <rte_debug.h>
> +#include <rte_log.h>
> +#include <rte_spinlock.h>
> +#include <rte_pci.h>
> +#include <rte_bus_pci.h>
> +
> +#ifndef PAGE_SIZE
> +#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
> +#endif
> +
> +#include "base/dlb2_hw_types.h"
> +#include "../dlb2_user.h"
> +
> +#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
> +
> +struct dlb2_dev;
> +
> +struct dlb2_port_memory {
> +	struct dlb2_list_head list;
> +	void *cq_base;
> +	bool valid;
> +};
> +
> +struct dlb2_dev {
> +	struct rte_pci_device *pdev;
> +	struct dlb2_hw hw;
> +	/* struct list_head list; */
> +	struct device *dlb2_device;
> +	struct dlb2_port_memory
> ldb_port_pages[DLB2_MAX_NUM_LDB_PORTS];
> +	struct dlb2_port_memory
> dir_port_pages[DLB2_MAX_NUM_DIR_PORTS];
> +	/* The enqueue_four function enqueues four HCWs (one cache-line
> worth)
> +	 * to the DLB2, using whichever mechanism is supported by the platform
> +	 * on which this driver is running.
> +	 */
> +	void (*enqueue_four)(void *qe4, void *pp_addr);
> +
> +	bool domain_reset_failed;
> +	/* The resource mutex serializes access to driver data structures and
> +	 * hardware registers.
> +	 */
> +	rte_spinlock_t resource_mutex;
> +	rte_spinlock_t measurement_lock;
> +	bool worker_launched;
> +	u8 revision;
> +};
> +
> +struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
> +
> +/* The following functions were pf_ops in kernel driver implementation */

Probably not important for the DPDK implementation...and better to avoid
mentioning the kernel implementation to avoid any appearance of licensing
impropriety.

> +int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
> +int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
> +				struct dlb2_create_sched_domain_args *args,
> +				struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
> +			     u32 domain_id,
> +			     struct dlb2_create_ldb_queue_args *args,
> +			     struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
> +			     u32 domain_id,
> +			     struct dlb2_create_dir_queue_args *args,
> +			     struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
> +			    u32 domain_id,
> +			    struct dlb2_create_ldb_port_args *args,
> +			    uintptr_t cq_dma_base,
> +			    struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
> +			    u32 domain_id,
> +			    struct dlb2_create_dir_port_args *args,
> +			    uintptr_t cq_dma_base,
> +			    struct dlb2_cmd_response *resp);
> +int dlb2_pf_start_domain(struct dlb2_hw *hw,
> +			 u32 domain_id,
> +			 struct dlb2_start_domain_args *args,
> +			 struct dlb2_cmd_response *resp);
> +int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
> +			    u32 domain_id,
> +			    struct dlb2_enable_ldb_port_args *args,
> +			    struct dlb2_cmd_response *resp);
> +int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
> +			     u32 domain_id,
> +			     struct dlb2_disable_ldb_port_args *args,
> +			     struct dlb2_cmd_response *resp);
> +int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
> +			    u32 domain_id,
> +			    struct dlb2_enable_dir_port_args *args,
> +			    struct dlb2_cmd_response *resp);
> +int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
> +			     u32 domain_id,
> +			     struct dlb2_disable_dir_port_args *args,
> +			     struct dlb2_cmd_response *resp);
> +int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
> +int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
> +				     u32 domain_id,
> +				     u32 port_id);
> +int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
> +				     u32 domain_id,
> +				     u32 port_id);
> +
> +#endif /* __DLB2_MAIN_H */
> diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
> new file mode 100644
> index 0000000..8c5ec20
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/dlb2_pf.c
> @@ -0,0 +1,251 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <sys/mman.h>
> +#include <sys/fcntl.h>
> +#include <sys/time.h>
> +#include <errno.h>
> +#include <assert.h>
> +#include <unistd.h>
> +#include <string.h>
> +#include <rte_debug.h>
> +#include <rte_log.h>
> +#include <rte_dev.h>
> +#include <rte_devargs.h>
> +#include <rte_mbuf.h>
> +#include <rte_ring.h>
> +#include <rte_errno.h>
> +#include <rte_kvargs.h>
> +#include <rte_malloc.h>
> +#include <rte_cycles.h>
> +#include <rte_io.h>
> +#include <rte_pci.h>
> +#include <rte_bus_pci.h>
> +#include <rte_eventdev.h>
> +#include <rte_eventdev_pmd.h>
> +#include <rte_eventdev_pmd_pci.h>
> +#include <rte_memory.h>
> +#include <rte_string_fns.h>
> +
> +#include "../dlb2_priv.h"
> +#include "../dlb2_iface.h"
> +#include "../dlb2_inline_fns.h"
> +#include "dlb2_main.h"
> +#include "base/dlb2_hw_types.h"
> +#include "base/dlb2_osdep.h"
> +#include "base/dlb2_resource.h"
> +
> +extern struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);

Not needed, dlb2_probe is declared in pf/dlb2_main.h

> +
> +#if !defined RTE_ARCH_X86_64
> +#error "This implementation only supports RTE_ARCH_X86_64 architecture."
> +#endif
> +

Seems redundant, this is checked in dlb2.c and meson.build should ensure it's not
built for other archs anyway.

[...]

> +static void
> +dlb2_pf_iface_fn_ptrs_init(void)
> +{
> +

Nit: extra whitespace

Thanks,
Gage

Re: [dpdk-dev] [PATCH 06/22] event/dlb2: add probe

[Jerin Jacob]

On Sat, Sep 12, 2020 at 2:01 AM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> The DLB2 hardware is a PCI device. This commit adds
> support for probe and other initialization. The
> dlb2_iface.[ch] files implement a flexible interface
> that supports both the PF PMD and the bifurcated PMD.
> The bifurcated PMD will be released in a future
> patch set. Note that the flexible interface is only
> used for configuration, and is not used in the data
> path. The shared code is added in pf/base.
>
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

> +static void
> +dlb2_qm_mmio_fn_init(void)
> +{
> +       /* Process-local function pointers for performing low level port i/o */
> +
> +       if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIR64B))
> +               qm_mmio_fns.pp_enqueue_four = dlb2_movdir64b;
> +       else
> +               qm_mmio_fns.pp_enqueue_four = dlb2_movntdq;
> +}
> +
> +#define RTE_BASE_10 10

DONT create RTE_ symbols in drivers as it will create conflict if
someone adds such symbols in the future in EAL.
Either make it an internal symbol or push an EAL patch.

Also please split this patch as at minimum as two as it is a huge patch.



> +
> +int dlb2_string_to_int(int *result, const char *str)
> +{
> +       long ret;
> +
> +       if (str == NULL || result == NULL)
> +               return -EINVAL;
> +
> +       errno = 0;
> +       ret = strtol(str, NULL, RTE_BASE_10);
> +       if (errno)
> +               return -errno;
> +
> +       /* long int and int may be different width for some architectures */
> +       if (ret < INT_MIN || ret > INT_MAX)
> +               return -EINVAL;
> +
> +       *result = ret;
> +       return 0;
> +}
> +
> +       u32 num_atomic_inflights;
> +       u32 num_hist_list_entries;
> +       u32 num_ldb_credits;
> +       u32 num_dir_credits;
> +       u8 cos_strict;
> +       u8 padding0[3];
> +       u32 padding1;
> +};
> +
> +struct dlb2_mbox_create_sched_domain_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_reset_sched_domain_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_reset_sched_domain_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +};
> +
> +struct dlb2_mbox_create_ldb_queue_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 num_sequence_numbers;
> +       u32 num_qid_inflights;
> +       u32 num_atomic_inflights;
> +       u32 lock_id_comp_level;
> +       u32 depth_threshold;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_create_ldb_queue_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_create_dir_queue_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 depth_threshold;
> +};
> +
> +struct dlb2_mbox_create_dir_queue_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_create_ldb_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u16 cq_depth;
> +       u16 cq_history_list_size;
> +       u8 cos_id;
> +       u8 cos_strict;
> +       u16 padding1;
> +       u64 cq_base_address;
> +};
> +
> +struct dlb2_mbox_create_ldb_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_create_dir_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u64 cq_base_address;
> +       u16 cq_depth;
> +       u16 padding0;
> +       s32 queue_id;
> +};
> +
> +struct dlb2_mbox_create_dir_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_enable_ldb_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_enable_ldb_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_disable_ldb_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_disable_ldb_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_enable_dir_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_enable_dir_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_disable_dir_port_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_disable_dir_port_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_ldb_port_owned_by_domain_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_ldb_port_owned_by_domain_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       s32 owned;
> +};
> +
> +struct dlb2_mbox_dir_port_owned_by_domain_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_dir_port_owned_by_domain_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       s32 owned;
> +};
> +
> +struct dlb2_mbox_map_qid_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 qid;
> +       u32 priority;
> +       u32 padding0;
> +};
> +
> +struct dlb2_mbox_map_qid_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 id;
> +};
> +
> +struct dlb2_mbox_unmap_qid_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 qid;
> +};
> +
> +struct dlb2_mbox_unmap_qid_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_start_domain_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +};
> +
> +struct dlb2_mbox_start_domain_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_enable_ldb_port_intr_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u16 port_id;
> +       u16 thresh;
> +       u16 vector;
> +       u16 owner_vf;
> +       u16 reserved[2];
> +};
> +
> +struct dlb2_mbox_enable_ldb_port_intr_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_enable_dir_port_intr_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u16 port_id;
> +       u16 thresh;
> +       u16 vector;
> +       u16 owner_vf;
> +       u16 reserved[2];
> +};
> +
> +struct dlb2_mbox_enable_dir_port_intr_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_arm_cq_intr_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 is_ldb;
> +};
> +
> +struct dlb2_mbox_arm_cq_intr_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 padding0;
> +};
> +
> +/*
> + * The alert_id and aux_alert_data follows the format of the alerts defined in
> + * dlb2_types.h. The alert id contains an enum dlb2_domain_alert_id value, and
> + * the aux_alert_data value varies depending on the alert.
> + */
> +struct dlb2_mbox_vf_alert_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 alert_id;
> +       u32 aux_alert_data;
> +};
> +
> +enum dlb2_mbox_vf_notification_type {
> +       DLB2_MBOX_VF_NOTIFICATION_PRE_RESET,
> +       DLB2_MBOX_VF_NOTIFICATION_POST_RESET,
> +
> +       /* NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES must be last */
> +       NUM_DLB2_MBOX_VF_NOTIFICATION_TYPES,
> +};
> +
> +struct dlb2_mbox_vf_notification_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 notification;
> +};
> +
> +struct dlb2_mbox_vf_in_use_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_vf_in_use_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 in_use;
> +};
> +
> +struct dlb2_mbox_get_sn_allocation_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 group_id;
> +};
> +
> +struct dlb2_mbox_get_sn_allocation_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 num;
> +};
> +
> +struct dlb2_mbox_get_ldb_queue_depth_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 queue_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_get_ldb_queue_depth_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 depth;
> +};
> +
> +struct dlb2_mbox_get_dir_queue_depth_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 queue_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_get_dir_queue_depth_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 depth;
> +};
> +
> +struct dlb2_mbox_pending_port_unmaps_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 domain_id;
> +       u32 port_id;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_pending_port_unmaps_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 num;
> +};
> +
> +struct dlb2_mbox_get_cos_bw_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 cos_id;
> +};
> +
> +struct dlb2_mbox_get_cos_bw_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 num;
> +};
> +
> +struct dlb2_mbox_get_sn_occupancy_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 group_id;
> +};
> +
> +struct dlb2_mbox_get_sn_occupancy_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 num;
> +};
> +
> +struct dlb2_mbox_query_cq_poll_mode_cmd_req {
> +       struct dlb2_mbox_req_hdr hdr;
> +       u32 padding;
> +};
> +
> +struct dlb2_mbox_query_cq_poll_mode_cmd_resp {
> +       struct dlb2_mbox_resp_hdr hdr;
> +       u32 error_code;
> +       u32 status;
> +       u32 mode;
> +};
> +
> +#endif /* __DLB2_BASE_DLB2_MBOX_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep.h b/drivers/event/dlb2/pf/base/dlb2_osdep.h
> new file mode 100644
> index 0000000..c8d8d5b
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_osdep.h
> @@ -0,0 +1,248 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_OSDEP_H
> +#define __DLB2_OSDEP_H
> +
> +#include <string.h>
> +#include <time.h>
> +#include <unistd.h>
> +#include <pthread.h>
> +
> +#include <rte_string_fns.h>
> +#include <rte_cycles.h>
> +#include <rte_io.h>
> +#include <rte_log.h>
> +#include <rte_spinlock.h>
> +#include "../dlb2_main.h"
> +#include "dlb2_resource.h"
> +#include "../../dlb2_log.h"
> +#include "../../dlb2_user.h"
> +
> +
> +#define DLB2_PCI_REG_READ(addr)        rte_read32((void *)addr)
> +#define DLB2_PCI_REG_WRITE(reg, value) rte_write32(value, (void *)reg)
> +
> +/* Read/write register 'reg' in the CSR BAR space */
> +#define DLB2_CSR_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->csr_kva + (reg)))
> +#define DLB2_CSR_RD(hw, reg) \
> +       DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
> +#define DLB2_CSR_WR(hw, reg, value) \
> +       DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
> +
> +/* Read/write register 'reg' in the func BAR space */
> +#define DLB2_FUNC_REG_ADDR(a, reg) ((void *)((uintptr_t)(a)->func_kva + (reg)))
> +#define DLB2_FUNC_RD(hw, reg) \
> +       DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
> +#define DLB2_FUNC_WR(hw, reg, value) \
> +       DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
> +
> +/* Map to PMDs logging interface */
> +#define DLB2_ERR(dev, fmt, args...) \
> +       DLB2_LOG_ERR(fmt, ## args)
> +
> +#define DLB2_INFO(dev, fmt, args...) \
> +       DLB2_LOG_INFO(fmt, ## args)
> +
> +#define DLB2_DEBUG(dev, fmt, args...) \
> +       DLB2_LOG_DBG(fmt, ## args)
> +
> +/**
> + * os_udelay() - busy-wait for a number of microseconds
> + * @usecs: delay duration.
> + */
> +static inline void os_udelay(int usecs)
> +{
> +       rte_delay_us(usecs);
> +}
> +
> +/**
> + * os_msleep() - sleep for a number of milliseconds
> + * @usecs: delay duration.
> + */
> +static inline void os_msleep(int msecs)
> +{
> +       rte_delay_ms(msecs);
> +}
> +
> +#define DLB2_PP_BASE(__is_ldb) \
> +       ((__is_ldb) ? DLB2_LDB_PP_BASE : DLB2_DIR_PP_BASE)
> +
> +/**
> + * os_map_producer_port() - map a producer port into the caller's address space
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: port ID
> + * @is_ldb: true for load-balanced port, false for a directed port
> + *
> + * This function maps the requested producer port memory into the caller's
> + * address space.
> + *
> + * Return:
> + * Returns the base address at which the PP memory was mapped, else NULL.
> + */
> +static inline void *os_map_producer_port(struct dlb2_hw *hw,
> +                                        u8 port_id,
> +                                        bool is_ldb)
> +{
> +       uint64_t addr;
> +       uint64_t pp_dma_base;
> +
> +
> +       pp_dma_base = (uintptr_t)hw->func_kva + DLB2_PP_BASE(is_ldb);
> +       addr = (pp_dma_base + (PAGE_SIZE * port_id));
> +
> +       return (void *)(uintptr_t)addr;
> +}
> +
> +/**
> + * os_unmap_producer_port() - unmap a producer port
> + * @addr: mapped producer port address
> + *
> + * This function undoes os_map_producer_port() by unmapping the producer port
> + * memory from the caller's address space.
> + *
> + * Return:
> + * Returns the base address at which the PP memory was mapped, else NULL.
> + */
> +static inline void os_unmap_producer_port(struct dlb2_hw *hw, void *addr)
> +{
> +       RTE_SET_USED(hw);
> +       RTE_SET_USED(addr);
> +}
> +
> +/**
> + * os_fence_hcw() - fence an HCW to ensure it arrives at the device
> + * @hw: dlb2_hw handle for a particular device.
> + * @pp_addr: producer port address
> + */
> +static inline void os_fence_hcw(struct dlb2_hw *hw, u64 *pp_addr)
> +{
> +       RTE_SET_USED(hw);
> +
> +       /* To ensure outstanding HCWs reach the device, read the PP address. IA
> +        * memory ordering prevents reads from passing older writes, and the
> +        * mfence also ensures this.
> +        */
> +       rte_mb();
> +
> +       *(volatile u64 *)pp_addr;
> +}
> +
> +/**
> + * os_enqueue_four_hcws() - enqueue four HCWs to DLB
> + * @hw: dlb2_hw handle for a particular device.
> + * @hcw: pointer to the 64B-aligned contiguous HCW memory
> + * @addr: producer port address
> + */
> +static inline void os_enqueue_four_hcws(struct dlb2_hw *hw,
> +                                       struct dlb2_hcw *hcw,
> +                                       void *addr)
> +{
> +       struct dlb2_dev *dlb2_dev;
> +
> +       dlb2_dev = container_of(hw, struct dlb2_dev, hw);
> +
> +       dlb2_dev->enqueue_four(hcw, addr);
> +}
> +
> +/**
> + * DLB2_HW_ERR() - log an error message
> + * @dlb2: dlb2_hw handle for a particular device.
> + * @...: variable string args.
> + */
> +#define DLB2_HW_ERR(dlb2, ...) do {    \
> +       RTE_SET_USED(dlb2);             \
> +       DLB2_ERR(dlb2, __VA_ARGS__);    \
> +} while (0)
> +
> +/**
> + * DLB2_HW_DBG() - log an info message
> + * @dlb2: dlb2_hw handle for a particular device.
> + * @...: variable string args.
> + */
> +#define DLB2_HW_DBG(dlb2, ...) do {    \
> +       RTE_SET_USED(dlb2);             \
> +       DLB2_DEBUG(dlb2, __VA_ARGS__);  \
> +} while (0)
> +
> +/* The callback runs until it completes all outstanding QID->CQ
> + * map and unmap requests. To prevent deadlock, this function gives other
> + * threads a chance to grab the resource mutex and configure hardware.
> + */
> +static void *dlb2_complete_queue_map_unmap(void *__args)
> +{
> +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)__args;
> +       int ret;
> +
> +       while (1) {
> +               rte_spinlock_lock(&dlb2_dev->resource_mutex);
> +
> +               ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
> +               ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
> +
> +               if (ret != 0) {
> +                       rte_spinlock_unlock(&dlb2_dev->resource_mutex);
> +                       /* Relinquish the CPU so the application can process
> +                        * its CQs, so this function doesn't deadlock.
> +                        */
> +                       sched_yield();
> +               } else {
> +                       break;
> +               }
> +       }
> +
> +       dlb2_dev->worker_launched = false;
> +
> +       rte_spinlock_unlock(&dlb2_dev->resource_mutex);
> +
> +       return NULL;
> +}
> +
> +
> +/**
> + * os_schedule_work() - launch a thread to process pending map and unmap work
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function launches a kernel thread that will run until all pending
> + * map and unmap procedures are complete.
> + */
> +static inline void os_schedule_work(struct dlb2_hw *hw)
> +{
> +       struct dlb2_dev *dlb2_dev;
> +       pthread_t complete_queue_map_unmap_thread;
> +       int ret;
> +
> +       dlb2_dev = container_of(hw, struct dlb2_dev, hw);
> +
> +       ret = rte_ctrl_thread_create(&complete_queue_map_unmap_thread,
> +                                    "dlb_queue_unmap_waiter",
> +                                    NULL,
> +                                    dlb2_complete_queue_map_unmap,
> +                                    dlb2_dev);
> +       if (ret)
> +               DLB2_ERR(dlb2_dev,
> +                        "Could not create queue complete map/unmap thread, err=%d\n",
> +                        ret);
> +       else
> +               dlb2_dev->worker_launched = true;
> +}
> +
> +/**
> + * os_worker_active() - query whether the map/unmap worker thread is active
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function returns a boolean indicating whether a thread (launched by
> + * os_schedule_work()) is active. This function is used to determine
> + * whether or not to launch a worker thread.
> + */
> +static inline bool os_worker_active(struct dlb2_hw *hw)
> +{
> +       struct dlb2_dev *dlb2_dev;
> +
> +       dlb2_dev = container_of(hw, struct dlb2_dev, hw);
> +
> +       return dlb2_dev->worker_launched;
> +}
> +
> +#endif /*  __DLB2_OSDEP_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
> new file mode 100644
> index 0000000..7e48878
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
> @@ -0,0 +1,447 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_OSDEP_BITMAP_H
> +#define __DLB2_OSDEP_BITMAP_H
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <unistd.h>
> +#include <rte_bitmap.h>
> +#include <rte_string_fns.h>
> +#include <rte_malloc.h>
> +#include <rte_errno.h>
> +#include "../dlb2_main.h"
> +
> +/*************************/
> +/*** Bitmap operations ***/
> +/*************************/
> +struct dlb2_bitmap {
> +       struct rte_bitmap *map;
> +       unsigned int len;
> +};
> +
> +/**
> + * dlb2_bitmap_alloc() - alloc a bitmap data structure
> + * @bitmap: pointer to dlb2_bitmap structure pointer.
> + * @len: number of entries in the bitmap.
> + *
> + * This function allocates a bitmap and initializes it with length @len. All
> + * entries are initially zero.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or len is 0.
> + * ENOMEM - could not allocate memory for the bitmap data structure.
> + */
> +static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
> +                                   unsigned int len)
> +{
> +       struct dlb2_bitmap *bm;
> +       void *mem;
> +       uint32_t alloc_size;
> +       uint32_t nbits = (uint32_t)len;
> +
> +       if (!bitmap || nbits == 0)
> +               return -EINVAL;
> +
> +       /* Allocate DLB2 bitmap control struct */
> +       bm = rte_malloc("DLB2_PF",
> +                       sizeof(struct dlb2_bitmap),
> +                       RTE_CACHE_LINE_SIZE);
> +
> +       if (!bm)
> +               return -ENOMEM;
> +
> +       /* Allocate bitmap memory */
> +       alloc_size = rte_bitmap_get_memory_footprint(nbits);
> +       mem = rte_malloc("DLB2_PF_BITMAP", alloc_size, RTE_CACHE_LINE_SIZE);
> +       if (!mem) {
> +               rte_free(bm);
> +               return -ENOMEM;
> +       }
> +
> +       bm->map = rte_bitmap_init(len, mem, alloc_size);
> +       if (!bm->map) {
> +               rte_free(mem);
> +               rte_free(bm);
> +               return -ENOMEM;
> +       }
> +
> +       bm->len = len;
> +
> +       *bitmap = bm;
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_free() - free a previously allocated bitmap data structure
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
> + */
> +static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
> +{
> +       if (!bitmap)
> +               return;
> +
> +       rte_free(bitmap->map);
> +       rte_free(bitmap);
> +}
> +
> +/**
> + * dlb2_bitmap_fill() - fill a bitmap with all 1s
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function sets all bitmap values to 1.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
> +{
> +       unsigned int i;
> +
> +       if (!bitmap || !bitmap->map)
> +               return -EINVAL;
> +
> +       /* TODO - optimize */
> +       for (i = 0; i != bitmap->len; i++)
> +               rte_bitmap_set(bitmap->map, i);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_fill() - fill a bitmap with all 0s
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function sets all bitmap values to 0.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
> +{
> +       if (!bitmap || !bitmap->map)
> +               return -EINVAL;
> +
> +       rte_bitmap_reset(bitmap->map);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_set() - set a bitmap entry
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: bit index.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
> + *         bitmap length.
> + */
> +static inline int dlb2_bitmap_set(struct dlb2_bitmap *bitmap,
> +                                 unsigned int bit)
> +{
> +       if (!bitmap || !bitmap->map)
> +               return -EINVAL;
> +
> +       if (bitmap->len <= bit)
> +               return -EINVAL;
> +
> +       rte_bitmap_set(bitmap->map, bit);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_set_range() - set a range of bitmap entries
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: starting bit index.
> + * @len: length of the range.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
> + *         length.
> + */
> +static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
> +                                       unsigned int bit,
> +                                       unsigned int len)
> +{
> +       unsigned int i;
> +
> +       if (!bitmap || !bitmap->map)
> +               return -EINVAL;
> +
> +       if (bitmap->len <= bit)
> +               return -EINVAL;
> +
> +       /* TODO - optimize */
> +       for (i = 0; i != len; i++)
> +               rte_bitmap_set(bitmap->map, bit + i);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_clear() - clear a bitmap entry
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: bit index.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or bit is larger than the
> + *         bitmap length.
> + */
> +static inline int dlb2_bitmap_clear(struct dlb2_bitmap *bitmap,
> +                                   unsigned int bit)
> +{
> +       if (!bitmap || !bitmap->map)
> +               return -EINVAL;
> +
> +       if (bitmap->len <= bit)
> +               return -EINVAL;
> +
> +       rte_bitmap_clear(bitmap->map, bit);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_clear_range() - clear a range of bitmap entries
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @bit: starting bit index.
> + * @len: length of the range.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
> + *         length.
> + */
> +static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
> +                                         unsigned int bit,
> +                                         unsigned int len)
> +{
> +       unsigned int i;
> +
> +       if (!bitmap || !bitmap->map)
> +               return -EINVAL;
> +
> +       if (bitmap->len <= bit)
> +               return -EINVAL;
> +
> +       /* TODO - optimize */
> +       for (i = 0; i != len; i++)
> +               rte_bitmap_clear(bitmap->map, bit + i);
> +
> +       return 0;
> +}
> +
> +/**
> + * dlb2_bitmap_find_set_bit_range() - find an range of set bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + * @len: length of the range.
> + *
> + * This function looks for a range of set bits of length @len.
> + *
> + * Return:
> + * Returns the base bit index upon success, < 0 otherwise.
> + *
> + * Errors:
> + * ENOENT - unable to find a length *len* range of set bits.
> + * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
> + */
> +static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
> +                                                unsigned int len)
> +{
> +       unsigned int i, j = 0;
> +
> +       if (!bitmap || !bitmap->map || len == 0)
> +               return -EINVAL;
> +
> +       if (bitmap->len < len)
> +               return -ENOENT;
> +
> +       /* TODO - optimize */
> +       for (i = 0; i != bitmap->len; i++) {
> +               if  (rte_bitmap_get(bitmap->map, i)) {
> +                       if (++j == len)
> +                               return i - j + 1;
> +               } else {
> +                       j = 0;
> +               }
> +       }
> +
> +       /* No set bit range of length len? */
> +       return -ENOENT;
> +}
> +
> +/**
> + * dlb2_bitmap_find_set_bit() - find an range of set bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function looks for a single set bit.
> + *
> + * Return:
> + * Returns the base bit index upon success, -1 if not found, <-1 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
> + */
> +static inline int dlb2_bitmap_find_set_bit(struct dlb2_bitmap *bitmap)
> +{
> +       unsigned int i;
> +
> +       if (!bitmap)
> +               return -EINVAL;
> +
> +       if (!bitmap->map)
> +               return -EINVAL;
> +
> +       /* TODO - optimize */
> +       for (i = 0; i != bitmap->len; i++) {
> +               if  (rte_bitmap_get(bitmap->map, i))
> +                       return i;
> +       }
> +
> +       return -ENOENT;
> +}
> +
> +/**
> + * dlb2_bitmap_count() - returns the number of set bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * This function looks for a single set bit.
> + *
> + * Return:
> + * Returns the number of set bits upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
> +{
> +       int weight = 0;
> +       unsigned int i;
> +
> +       if (!bitmap)
> +               return -EINVAL;
> +
> +       if (!bitmap->map)
> +               return -EINVAL;
> +
> +       /* TODO - optimize */
> +       for (i = 0; i != bitmap->len; i++) {
> +               if  (rte_bitmap_get(bitmap->map, i))
> +                       weight++;
> +       }
> +       return weight;
> +}
> +
> +/**
> + * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
> + *                                   bits
> + * @bitmap: pointer to dlb2_bitmap structure.
> + *
> + * Return:
> + * Returns the bitmap's longest contiguous range of of set bits upon success,
> + * <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
> +{
> +       int max_len = 0, len = 0;
> +       unsigned int i;
> +
> +       if (!bitmap)
> +               return -EINVAL;
> +
> +       if (!bitmap->map)
> +               return -EINVAL;
> +
> +       /* TODO - optimize */
> +       for (i = 0; i != bitmap->len; i++) {
> +               if  (rte_bitmap_get(bitmap->map, i)) {
> +                       len++;
> +               } else {
> +                       if (len > max_len)
> +                               max_len = len;
> +                       len = 0;
> +               }
> +       }
> +
> +       if (len > max_len)
> +               max_len = len;
> +
> +       return max_len;
> +}
> +
> +/**
> + * dlb2_bitmap_or() - store the logical 'or' of two bitmaps into a third
> + * @dest: pointer to dlb2_bitmap structure, which will contain the results of
> + *       the 'or' of src1 and src2.
> + * @src1: pointer to dlb2_bitmap structure, will be 'or'ed with src2.
> + * @src2: pointer to dlb2_bitmap structure, will be 'or'ed with src1.
> + *
> + * This function 'or's two bitmaps together and stores the result in a third
> + * bitmap. The source and destination bitmaps can be the same.
> + *
> + * Return:
> + * Returns the number of set bits upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - One of the bitmaps is NULL or is uninitialized.
> + */
> +static inline int dlb2_bitmap_or(struct dlb2_bitmap *dest,
> +                                struct dlb2_bitmap *src1,
> +                                struct dlb2_bitmap *src2)
> +{
> +       unsigned int i, min;
> +       int numset = 0;
> +
> +       if (!dest || !dest->map ||
> +           !src1 || !src1->map ||
> +           !src2 || !src2->map)
> +               return -EINVAL;
> +
> +       min = dest->len;
> +       min = (min > src1->len) ? src1->len : min;
> +       min = (min > src2->len) ? src2->len : min;
> +
> +       for (i = 0; i != min; i++) {
> +               if  (rte_bitmap_get(src1->map, i) ||
> +                    rte_bitmap_get(src2->map, i)) {
> +                       rte_bitmap_set(dest->map, i);
> +                       numset++;
> +               } else {
> +                       rte_bitmap_clear(dest->map, i);
> +               }
> +       }
> +
> +       return numset;
> +}
> +
> +#endif /*  __DLB2_OSDEP_BITMAP_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_list.h b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
> new file mode 100644
> index 0000000..5531739
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_osdep_list.h
> @@ -0,0 +1,131 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_OSDEP_LIST_H
> +#define __DLB2_OSDEP_LIST_H
> +
> +#include <rte_tailq.h>
> +
> +struct dlb2_list_entry {
> +       TAILQ_ENTRY(dlb2_list_entry) node;
> +};
> +
> +/* Dummy - just a struct definition */
> +TAILQ_HEAD(dlb2_list_head, dlb2_list_entry);
> +
> +/* =================
> + * TAILQ Supplements
> + * =================
> + */
> +
> +#ifndef TAILQ_FOREACH_ENTRY
> +#define TAILQ_FOREACH_ENTRY(ptr, head, name, iter)             \
> +       for ((iter) = TAILQ_FIRST(&head);                       \
> +           (iter)                                              \
> +               && (ptr = container_of(iter, typeof(*(ptr)), name)); \
> +           (iter) = TAILQ_NEXT((iter), node))
> +#endif
> +
> +#ifndef TAILQ_FOREACH_ENTRY_SAFE
> +#define TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, iter, tvar)  \
> +       for ((iter) = TAILQ_FIRST(&head);                       \
> +           (iter) &&                                           \
> +               (ptr = container_of(iter, typeof(*(ptr)), name)) &&\
> +               ((tvar) = TAILQ_NEXT((iter), node), 1); \
> +           (iter) = (tvar))
> +#endif
> +
> +/***********************/
> +/*** List operations ***/
> +/***********************/
> +
> +/**
> + * dlb2_list_init_head() - initialize the head of a list
> + * @head: list head
> + */
> +static inline void dlb2_list_init_head(struct dlb2_list_head *head)
> +{
> +       TAILQ_INIT(head);
> +}
> +
> +/**
> + * dlb2_list_add() - add an entry to a list
> + * @head: list head
> + * @entry: new list entry
> + */
> +static inline void
> +dlb2_list_add(struct dlb2_list_head *head, struct dlb2_list_entry *entry)
> +{
> +       TAILQ_INSERT_TAIL(head, entry, node);
> +}
> +
> +/**
> + * dlb2_list_del() - delete an entry from a list
> + * @entry: list entry
> + * @head: list head
> + */
> +static inline void dlb2_list_del(struct dlb2_list_head *head,
> +                                struct dlb2_list_entry *entry)
> +{
> +       TAILQ_REMOVE(head, entry, node);
> +}
> +
> +/**
> + * dlb2_list_empty() - check if a list is empty
> + * @head: list head
> + *
> + * Return:
> + * Returns 1 if empty, 0 if not.
> + */
> +static inline int dlb2_list_empty(struct dlb2_list_head *head)
> +{
> +       return TAILQ_EMPTY(head);
> +}
> +
> +/**
> + * dlb2_list_splice() - splice a list
> + * @src_head: list to be added
> + * @ head: where src_head will be inserted
> + */
> +static inline void dlb2_list_splice(struct dlb2_list_head *src_head,
> +                                   struct dlb2_list_head *head)
> +{
> +       TAILQ_CONCAT(head, src_head, node);
> +}
> +
> +/**
> + * DLB2_LIST_HEAD() - retrieve the head of the list
> + * @head: list head
> + * @type: type of the list variable
> + * @name: name of the list field within the containing struct
> + */
> +#define DLB2_LIST_HEAD(head, type, name)                       \
> +       (TAILQ_FIRST(&head) ?                                   \
> +               container_of(TAILQ_FIRST(&head), type, name) :  \
> +               NULL)
> +
> +/**
> + * DLB2_LIST_FOR_EACH() - iterate over a list
> + * @head: list head
> + * @ptr: pointer to struct containing a struct list
> + * @name: name of the list field within the containing struct
> + * @iter: iterator variable
> + */
> +#define DLB2_LIST_FOR_EACH(head, ptr, name, tmp_iter) \
> +       TAILQ_FOREACH_ENTRY(ptr, head, name, tmp_iter)
> +
> +/**
> + * DLB2_LIST_FOR_EACH_SAFE() - iterate over a list. This loop works even if
> + * an element is removed from the list while processing it.
> + * @ptr: pointer to struct containing a struct list
> + * @ptr_tmp: pointer to struct containing a struct list (temporary)
> + * @head: list head
> + * @name: name of the list field within the containing struct
> + * @iter: iterator variable
> + * @iter_tmp: iterator variable (temporary)
> + */
> +#define DLB2_LIST_FOR_EACH_SAFE(head, ptr, ptr_tmp, name, tmp_iter, saf_itr) \
> +       TAILQ_FOREACH_ENTRY_SAFE(ptr, head, name, tmp_iter, saf_itr)
> +
> +#endif /*  __DLB2_OSDEP_LIST_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_osdep_types.h b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
> new file mode 100644
> index 0000000..0a48f7e
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_osdep_types.h
> @@ -0,0 +1,31 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_OSDEP_TYPES_H
> +#define __DLB2_OSDEP_TYPES_H
> +
> +#include <linux/types.h>
> +
> +#include <inttypes.h>
> +#include <ctype.h>
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <string.h>
> +#include <unistd.h>
> +#include <errno.h>
> +
> +/* Types for user mode PF PMD */
> +typedef uint8_t         u8;
> +typedef int8_t          s8;
> +typedef uint16_t        u16;
> +typedef int16_t         s16;
> +typedef uint32_t        u32;
> +typedef int32_t         s32;
> +typedef uint64_t        u64;
> +
> +#define __iomem
> +
> +/* END types for user mode PF PMD */
> +
> +#endif /* __DLB2_OSDEP_TYPES_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_regs.h b/drivers/event/dlb2/pf/base/dlb2_regs.h
> new file mode 100644
> index 0000000..43ecad4
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_regs.h
> @@ -0,0 +1,2527 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_REGS_H
> +#define __DLB2_REGS_H
> +
> +#include "dlb2_osdep_types.h"
> +
> +#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
> +#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
> +       (0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
> +union dlb2_func_pf_vf2pf_mailbox {
> +       struct {
> +               u32 msg : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
> +       (0x1f00 + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
> +union dlb2_func_pf_vf2pf_mailbox_isr {
> +       struct {
> +               u32 vf0_isr : 1;
> +               u32 vf1_isr : 1;
> +               u32 vf2_isr : 1;
> +               u32 vf3_isr : 1;
> +               u32 vf4_isr : 1;
> +               u32 vf5_isr : 1;
> +               u32 vf6_isr : 1;
> +               u32 vf7_isr : 1;
> +               u32 vf8_isr : 1;
> +               u32 vf9_isr : 1;
> +               u32 vf10_isr : 1;
> +               u32 vf11_isr : 1;
> +               u32 vf12_isr : 1;
> +               u32 vf13_isr : 1;
> +               u32 vf14_isr : 1;
> +               u32 vf15_isr : 1;
> +               u32 rsvd0 : 16;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
> +       (0x1f04 + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
> +union dlb2_func_pf_vf2pf_flr_isr {
> +       struct {
> +               u32 vf0_isr : 1;
> +               u32 vf1_isr : 1;
> +               u32 vf2_isr : 1;
> +               u32 vf3_isr : 1;
> +               u32 vf4_isr : 1;
> +               u32 vf5_isr : 1;
> +               u32 vf6_isr : 1;
> +               u32 vf7_isr : 1;
> +               u32 vf8_isr : 1;
> +               u32 vf9_isr : 1;
> +               u32 vf10_isr : 1;
> +               u32 vf11_isr : 1;
> +               u32 vf12_isr : 1;
> +               u32 vf13_isr : 1;
> +               u32 vf14_isr : 1;
> +               u32 vf15_isr : 1;
> +               u32 rsvd0 : 16;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
> +       (0x1f10 + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
> +union dlb2_func_pf_vf2pf_isr_pend {
> +       struct {
> +               u32 isr_pend : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
> +#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
> +       (0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
> +union dlb2_func_pf_pf2vf_mailbox {
> +       struct {
> +               u32 msg : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
> +       (0x2f00 + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
> +union dlb2_func_pf_pf2vf_mailbox_isr {
> +       struct {
> +               u32 vf0_isr : 1;
> +               u32 vf1_isr : 1;
> +               u32 vf2_isr : 1;
> +               u32 vf3_isr : 1;
> +               u32 vf4_isr : 1;
> +               u32 vf5_isr : 1;
> +               u32 vf6_isr : 1;
> +               u32 vf7_isr : 1;
> +               u32 vf8_isr : 1;
> +               u32 vf9_isr : 1;
> +               u32 vf10_isr : 1;
> +               u32 vf11_isr : 1;
> +               u32 vf12_isr : 1;
> +               u32 vf13_isr : 1;
> +               u32 vf14_isr : 1;
> +               u32 vf15_isr : 1;
> +               u32 rsvd0 : 16;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
> +       (0x3000 + (vf_id) * 0x10000)
> +#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
> +union dlb2_func_pf_vf_reset_in_progress {
> +       struct {
> +               u32 vf0_reset_in_progress : 1;
> +               u32 vf1_reset_in_progress : 1;
> +               u32 vf2_reset_in_progress : 1;
> +               u32 vf3_reset_in_progress : 1;
> +               u32 vf4_reset_in_progress : 1;
> +               u32 vf5_reset_in_progress : 1;
> +               u32 vf6_reset_in_progress : 1;
> +               u32 vf7_reset_in_progress : 1;
> +               u32 vf8_reset_in_progress : 1;
> +               u32 vf9_reset_in_progress : 1;
> +               u32 vf10_reset_in_progress : 1;
> +               u32 vf11_reset_in_progress : 1;
> +               u32 vf12_reset_in_progress : 1;
> +               u32 vf13_reset_in_progress : 1;
> +               u32 vf14_reset_in_progress : 1;
> +               u32 vf15_reset_in_progress : 1;
> +               u32 rsvd0 : 16;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
> +       (0x100000c + (x) * 0x10)
> +#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
> +union dlb2_msix_mem_vector_ctrl {
> +       struct {
> +               u32 vec_mask : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
> +       (0x20 + (x) * 0x4)
> +#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
> +union dlb2_iosf_func_vf_bar_dsbl {
> +       struct {
> +               u32 func_vf_bar_dis : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_VAS 0x1000011c
> +#define DLB2_SYS_TOTAL_VAS_RST 0x20
> +union dlb2_sys_total_vas {
> +       struct {
> +               u32 total_vas : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
> +#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
> +union dlb2_sys_total_dir_ports {
> +       struct {
> +               u32 total_dir_ports : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
> +#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
> +union dlb2_sys_total_ldb_ports {
> +       struct {
> +               u32 total_ldb_ports : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
> +#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
> +union dlb2_sys_total_dir_qid {
> +       struct {
> +               u32 total_dir_qid : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
> +#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
> +union dlb2_sys_total_ldb_qid {
> +       struct {
> +               u32 total_ldb_qid : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
> +#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
> +union dlb2_sys_total_dir_crds {
> +       struct {
> +               u32 total_dir_credits : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
> +#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
> +union dlb2_sys_total_ldb_crds {
> +       struct {
> +               u32 total_ldb_credits : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
> +#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
> +union dlb2_sys_alarm_pf_synd2 {
> +       struct {
> +               u32 lock_id : 16;
> +               u32 meas : 1;
> +               u32 debug : 7;
> +               u32 cq_pop : 1;
> +               u32 qe_uhl : 1;
> +               u32 qe_orsp : 1;
> +               u32 qe_valid : 1;
> +               u32 cq_int_rearm : 1;
> +               u32 dsi_error : 1;
> +               u32 rsvd0 : 2;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
> +#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
> +union dlb2_sys_alarm_pf_synd1 {
> +       struct {
> +               u32 dsi : 16;
> +               u32 qid : 8;
> +               u32 qtype : 2;
> +               u32 qpri : 3;
> +               u32 msg_type : 3;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
> +#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
> +union dlb2_sys_alarm_pf_synd0 {
> +       struct {
> +               u32 syndrome : 8;
> +               u32 rtype : 2;
> +               u32 rsvd0 : 3;
> +               u32 is_ldb : 1;
> +               u32 cls : 2;
> +               u32 aid : 6;
> +               u32 unit : 4;
> +               u32 source : 4;
> +               u32 more : 1;
> +               u32 valid : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_LDB_VPP_V(x) \
> +       (0x10000f00 + (x) * 0x1000)
> +#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
> +union dlb2_sys_vf_ldb_vpp_v {
> +       struct {
> +               u32 vpp_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_LDB_VPP2PP(x) \
> +       (0x10000f04 + (x) * 0x1000)
> +#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
> +union dlb2_sys_vf_ldb_vpp2pp {
> +       struct {
> +               u32 pp : 6;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_DIR_VPP_V(x) \
> +       (0x10000f08 + (x) * 0x1000)
> +#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
> +union dlb2_sys_vf_dir_vpp_v {
> +       struct {
> +               u32 vpp_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_DIR_VPP2PP(x) \
> +       (0x10000f0c + (x) * 0x1000)
> +#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
> +union dlb2_sys_vf_dir_vpp2pp {
> +       struct {
> +               u32 pp : 6;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_LDB_VQID_V(x) \
> +       (0x10000f10 + (x) * 0x1000)
> +#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
> +union dlb2_sys_vf_ldb_vqid_v {
> +       struct {
> +               u32 vqid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_LDB_VQID2QID(x) \
> +       (0x10000f14 + (x) * 0x1000)
> +#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
> +union dlb2_sys_vf_ldb_vqid2qid {
> +       struct {
> +               u32 qid : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_QID2VQID(x) \
> +       (0x10000f18 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_QID2VQID_RST 0x0
> +union dlb2_sys_ldb_qid2vqid {
> +       struct {
> +               u32 vqid : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_DIR_VQID_V(x) \
> +       (0x10000f1c + (x) * 0x1000)
> +#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
> +union dlb2_sys_vf_dir_vqid_v {
> +       struct {
> +               u32 vqid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_VF_DIR_VQID2QID(x) \
> +       (0x10000f20 + (x) * 0x1000)
> +#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
> +union dlb2_sys_vf_dir_vqid2qid {
> +       struct {
> +               u32 qid : 6;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_VASQID_V(x) \
> +       (0x10000f24 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_VASQID_V_RST 0x0
> +union dlb2_sys_ldb_vasqid_v {
> +       struct {
> +               u32 vasqid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_VASQID_V(x) \
> +       (0x10000f28 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_VASQID_V_RST 0x0
> +union dlb2_sys_dir_vasqid_v {
> +       struct {
> +               u32 vasqid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_VF_SYND2(x) \
> +       (0x10000f48 + (x) * 0x1000)
> +#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
> +union dlb2_sys_alarm_vf_synd2 {
> +       struct {
> +               u32 lock_id : 16;
> +               u32 debug : 8;
> +               u32 cq_pop : 1;
> +               u32 qe_uhl : 1;
> +               u32 qe_orsp : 1;
> +               u32 qe_valid : 1;
> +               u32 isz : 1;
> +               u32 dsi_error : 1;
> +               u32 dlbrsvd : 2;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_VF_SYND1(x) \
> +       (0x10000f44 + (x) * 0x1000)
> +#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
> +union dlb2_sys_alarm_vf_synd1 {
> +       struct {
> +               u32 dsi : 16;
> +               u32 qid : 8;
> +               u32 qtype : 2;
> +               u32 qpri : 3;
> +               u32 msg_type : 3;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_VF_SYND0(x) \
> +       (0x10000f40 + (x) * 0x1000)
> +#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
> +union dlb2_sys_alarm_vf_synd0 {
> +       struct {
> +               u32 syndrome : 8;
> +               u32 rtype : 2;
> +               u32 vf_synd0_parity : 1;
> +               u32 vf_synd1_parity : 1;
> +               u32 vf_synd2_parity : 1;
> +               u32 is_ldb : 1;
> +               u32 cls : 2;
> +               u32 aid : 6;
> +               u32 unit : 4;
> +               u32 source : 4;
> +               u32 more : 1;
> +               u32 valid : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_QID_CFG_V(x) \
> +       (0x10000f58 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
> +union dlb2_sys_ldb_qid_cfg_v {
> +       struct {
> +               u32 sn_cfg_v : 1;
> +               u32 fid_cfg_v : 1;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_QID_ITS(x) \
> +       (0x10000f54 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_QID_ITS_RST 0x0
> +union dlb2_sys_ldb_qid_its {
> +       struct {
> +               u32 qid_its : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_QID_V(x) \
> +       (0x10000f50 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_QID_V_RST 0x0
> +union dlb2_sys_ldb_qid_v {
> +       struct {
> +               u32 qid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_QID_ITS(x) \
> +       (0x10000f64 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_QID_ITS_RST 0x0
> +union dlb2_sys_dir_qid_its {
> +       struct {
> +               u32 qid_its : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_QID_V(x) \
> +       (0x10000f60 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_QID_V_RST 0x0
> +union dlb2_sys_dir_qid_v {
> +       struct {
> +               u32 qid_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
> +       (0x10000fa8 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
> +union dlb2_sys_ldb_cq_ai_data {
> +       struct {
> +               u32 cq_ai_data : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
> +       (0x10000fa4 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
> +union dlb2_sys_ldb_cq_ai_addr {
> +       struct {
> +               u32 rsvd1 : 2;
> +               u32 cq_ai_addr : 18;
> +               u32 rsvd0 : 12;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_PASID(x) \
> +       (0x10000fa0 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
> +union dlb2_sys_ldb_cq_pasid {
> +       struct {
> +               u32 pasid : 20;
> +               u32 exe_req : 1;
> +               u32 priv_req : 1;
> +               u32 fmt2 : 1;
> +               u32 rsvd0 : 9;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_AT(x) \
> +       (0x10000f9c + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_AT_RST 0x0
> +union dlb2_sys_ldb_cq_at {
> +       struct {
> +               u32 cq_at : 2;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_ISR(x) \
> +       (0x10000f98 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
> +/* CQ Interrupt Modes */
> +#define DLB2_CQ_ISR_MODE_DIS  0
> +#define DLB2_CQ_ISR_MODE_MSI  1
> +#define DLB2_CQ_ISR_MODE_MSIX 2
> +#define DLB2_CQ_ISR_MODE_ADI  3
> +union dlb2_sys_ldb_cq_isr {
> +       struct {
> +               u32 vector : 6;
> +               u32 vf : 4;
> +               u32 en_code : 2;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
> +       (0x10000f94 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
> +union dlb2_sys_ldb_cq2vf_pf_ro {
> +       struct {
> +               u32 vf : 4;
> +               u32 is_pf : 1;
> +               u32 ro : 1;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_PP_V(x) \
> +       (0x10000f90 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_PP_V_RST 0x0
> +union dlb2_sys_ldb_pp_v {
> +       struct {
> +               u32 pp_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_PP2VDEV(x) \
> +       (0x10000f8c + (x) * 0x1000)
> +#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
> +union dlb2_sys_ldb_pp2vdev {
> +       struct {
> +               u32 vdev : 4;
> +               u32 rsvd0 : 28;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_PP2VAS(x) \
> +       (0x10000f88 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_PP2VAS_RST 0x0
> +union dlb2_sys_ldb_pp2vas {
> +       struct {
> +               u32 vas : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
> +       (0x10000f84 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
> +union dlb2_sys_ldb_cq_addr_u {
> +       struct {
> +               u32 addr_u : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
> +       (0x10000f80 + (x) * 0x1000)
> +#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
> +union dlb2_sys_ldb_cq_addr_l {
> +       struct {
> +               u32 rsvd0 : 6;
> +               u32 addr_l : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_FMT(x) \
> +       (0x10000fec + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
> +union dlb2_sys_dir_cq_fmt {
> +       struct {
> +               u32 keep_pf_ppid : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
> +       (0x10000fe8 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
> +union dlb2_sys_dir_cq_ai_data {
> +       struct {
> +               u32 cq_ai_data : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
> +       (0x10000fe4 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
> +union dlb2_sys_dir_cq_ai_addr {
> +       struct {
> +               u32 rsvd1 : 2;
> +               u32 cq_ai_addr : 18;
> +               u32 rsvd0 : 12;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_PASID(x) \
> +       (0x10000fe0 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
> +union dlb2_sys_dir_cq_pasid {
> +       struct {
> +               u32 pasid : 20;
> +               u32 exe_req : 1;
> +               u32 priv_req : 1;
> +               u32 fmt2 : 1;
> +               u32 rsvd0 : 9;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_AT(x) \
> +       (0x10000fdc + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_AT_RST 0x0
> +union dlb2_sys_dir_cq_at {
> +       struct {
> +               u32 cq_at : 2;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_ISR(x) \
> +       (0x10000fd8 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
> +union dlb2_sys_dir_cq_isr {
> +       struct {
> +               u32 vector : 6;
> +               u32 vf : 4;
> +               u32 en_code : 2;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
> +       (0x10000fd4 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
> +union dlb2_sys_dir_cq2vf_pf_ro {
> +       struct {
> +               u32 vf : 4;
> +               u32 is_pf : 1;
> +               u32 ro : 1;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_PP_V(x) \
> +       (0x10000fd0 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_PP_V_RST 0x0
> +union dlb2_sys_dir_pp_v {
> +       struct {
> +               u32 pp_v : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_PP2VDEV(x) \
> +       (0x10000fcc + (x) * 0x1000)
> +#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
> +union dlb2_sys_dir_pp2vdev {
> +       struct {
> +               u32 vdev : 4;
> +               u32 rsvd0 : 28;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_PP2VAS(x) \
> +       (0x10000fc8 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_PP2VAS_RST 0x0
> +union dlb2_sys_dir_pp2vas {
> +       struct {
> +               u32 vas : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
> +       (0x10000fc4 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
> +union dlb2_sys_dir_cq_addr_u {
> +       struct {
> +               u32 addr_u : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
> +       (0x10000fc0 + (x) * 0x1000)
> +#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
> +union dlb2_sys_dir_cq_addr_l {
> +       struct {
> +               u32 rsvd0 : 6;
> +               u32 addr_l : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
> +#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
> +union dlb2_sys_ingress_alarm_enbl {
> +       struct {
> +               u32 illegal_hcw : 1;
> +               u32 illegal_pp : 1;
> +               u32 illegal_pasid : 1;
> +               u32 illegal_qid : 1;
> +               u32 disabled_qid : 1;
> +               u32 illegal_ldb_qid_cfg : 1;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_MSIX_ACK 0x10000400
> +#define DLB2_SYS_MSIX_ACK_RST 0x0
> +union dlb2_sys_msix_ack {
> +       struct {
> +               u32 msix_0_ack : 1;
> +               u32 msix_1_ack : 1;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
> +#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
> +union dlb2_sys_msix_passthru {
> +       struct {
> +               u32 msix_0_passthru : 1;
> +               u32 msix_1_passthru : 1;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_MSIX_MODE 0x10000408
> +#define DLB2_SYS_MSIX_MODE_RST 0x0
> +/* MSI-X Modes */
> +#define DLB2_MSIX_MODE_PACKED     0
> +#define DLB2_MSIX_MODE_COMPRESSED 1
> +union dlb2_sys_msix_mode {
> +       struct {
> +               u32 mode : 1;
> +               u32 poll_mode : 1;
> +               u32 poll_mask : 1;
> +               u32 poll_lock : 1;
> +               u32 rsvd0 : 28;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
> +#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
> +union dlb2_sys_dir_cq_31_0_occ_int_sts {
> +       struct {
> +               u32 cq_0_occ_int : 1;
> +               u32 cq_1_occ_int : 1;
> +               u32 cq_2_occ_int : 1;
> +               u32 cq_3_occ_int : 1;
> +               u32 cq_4_occ_int : 1;
> +               u32 cq_5_occ_int : 1;
> +               u32 cq_6_occ_int : 1;
> +               u32 cq_7_occ_int : 1;
> +               u32 cq_8_occ_int : 1;
> +               u32 cq_9_occ_int : 1;
> +               u32 cq_10_occ_int : 1;
> +               u32 cq_11_occ_int : 1;
> +               u32 cq_12_occ_int : 1;
> +               u32 cq_13_occ_int : 1;
> +               u32 cq_14_occ_int : 1;
> +               u32 cq_15_occ_int : 1;
> +               u32 cq_16_occ_int : 1;
> +               u32 cq_17_occ_int : 1;
> +               u32 cq_18_occ_int : 1;
> +               u32 cq_19_occ_int : 1;
> +               u32 cq_20_occ_int : 1;
> +               u32 cq_21_occ_int : 1;
> +               u32 cq_22_occ_int : 1;
> +               u32 cq_23_occ_int : 1;
> +               u32 cq_24_occ_int : 1;
> +               u32 cq_25_occ_int : 1;
> +               u32 cq_26_occ_int : 1;
> +               u32 cq_27_occ_int : 1;
> +               u32 cq_28_occ_int : 1;
> +               u32 cq_29_occ_int : 1;
> +               u32 cq_30_occ_int : 1;
> +               u32 cq_31_occ_int : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
> +#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
> +union dlb2_sys_dir_cq_63_32_occ_int_sts {
> +       struct {
> +               u32 cq_32_occ_int : 1;
> +               u32 cq_33_occ_int : 1;
> +               u32 cq_34_occ_int : 1;
> +               u32 cq_35_occ_int : 1;
> +               u32 cq_36_occ_int : 1;
> +               u32 cq_37_occ_int : 1;
> +               u32 cq_38_occ_int : 1;
> +               u32 cq_39_occ_int : 1;
> +               u32 cq_40_occ_int : 1;
> +               u32 cq_41_occ_int : 1;
> +               u32 cq_42_occ_int : 1;
> +               u32 cq_43_occ_int : 1;
> +               u32 cq_44_occ_int : 1;
> +               u32 cq_45_occ_int : 1;
> +               u32 cq_46_occ_int : 1;
> +               u32 cq_47_occ_int : 1;
> +               u32 cq_48_occ_int : 1;
> +               u32 cq_49_occ_int : 1;
> +               u32 cq_50_occ_int : 1;
> +               u32 cq_51_occ_int : 1;
> +               u32 cq_52_occ_int : 1;
> +               u32 cq_53_occ_int : 1;
> +               u32 cq_54_occ_int : 1;
> +               u32 cq_55_occ_int : 1;
> +               u32 cq_56_occ_int : 1;
> +               u32 cq_57_occ_int : 1;
> +               u32 cq_58_occ_int : 1;
> +               u32 cq_59_occ_int : 1;
> +               u32 cq_60_occ_int : 1;
> +               u32 cq_61_occ_int : 1;
> +               u32 cq_62_occ_int : 1;
> +               u32 cq_63_occ_int : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
> +#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
> +union dlb2_sys_ldb_cq_31_0_occ_int_sts {
> +       struct {
> +               u32 cq_0_occ_int : 1;
> +               u32 cq_1_occ_int : 1;
> +               u32 cq_2_occ_int : 1;
> +               u32 cq_3_occ_int : 1;
> +               u32 cq_4_occ_int : 1;
> +               u32 cq_5_occ_int : 1;
> +               u32 cq_6_occ_int : 1;
> +               u32 cq_7_occ_int : 1;
> +               u32 cq_8_occ_int : 1;
> +               u32 cq_9_occ_int : 1;
> +               u32 cq_10_occ_int : 1;
> +               u32 cq_11_occ_int : 1;
> +               u32 cq_12_occ_int : 1;
> +               u32 cq_13_occ_int : 1;
> +               u32 cq_14_occ_int : 1;
> +               u32 cq_15_occ_int : 1;
> +               u32 cq_16_occ_int : 1;
> +               u32 cq_17_occ_int : 1;
> +               u32 cq_18_occ_int : 1;
> +               u32 cq_19_occ_int : 1;
> +               u32 cq_20_occ_int : 1;
> +               u32 cq_21_occ_int : 1;
> +               u32 cq_22_occ_int : 1;
> +               u32 cq_23_occ_int : 1;
> +               u32 cq_24_occ_int : 1;
> +               u32 cq_25_occ_int : 1;
> +               u32 cq_26_occ_int : 1;
> +               u32 cq_27_occ_int : 1;
> +               u32 cq_28_occ_int : 1;
> +               u32 cq_29_occ_int : 1;
> +               u32 cq_30_occ_int : 1;
> +               u32 cq_31_occ_int : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
> +#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
> +union dlb2_sys_ldb_cq_63_32_occ_int_sts {
> +       struct {
> +               u32 cq_32_occ_int : 1;
> +               u32 cq_33_occ_int : 1;
> +               u32 cq_34_occ_int : 1;
> +               u32 cq_35_occ_int : 1;
> +               u32 cq_36_occ_int : 1;
> +               u32 cq_37_occ_int : 1;
> +               u32 cq_38_occ_int : 1;
> +               u32 cq_39_occ_int : 1;
> +               u32 cq_40_occ_int : 1;
> +               u32 cq_41_occ_int : 1;
> +               u32 cq_42_occ_int : 1;
> +               u32 cq_43_occ_int : 1;
> +               u32 cq_44_occ_int : 1;
> +               u32 cq_45_occ_int : 1;
> +               u32 cq_46_occ_int : 1;
> +               u32 cq_47_occ_int : 1;
> +               u32 cq_48_occ_int : 1;
> +               u32 cq_49_occ_int : 1;
> +               u32 cq_50_occ_int : 1;
> +               u32 cq_51_occ_int : 1;
> +               u32 cq_52_occ_int : 1;
> +               u32 cq_53_occ_int : 1;
> +               u32 cq_54_occ_int : 1;
> +               u32 cq_55_occ_int : 1;
> +               u32 cq_56_occ_int : 1;
> +               u32 cq_57_occ_int : 1;
> +               u32 cq_58_occ_int : 1;
> +               u32 cq_59_occ_int : 1;
> +               u32 cq_60_occ_int : 1;
> +               u32 cq_61_occ_int : 1;
> +               u32 cq_62_occ_int : 1;
> +               u32 cq_63_occ_int : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
> +#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
> +union dlb2_sys_dir_cq_opt_clr {
> +       struct {
> +               u32 cq : 6;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
> +#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
> +union dlb2_sys_alarm_hw_synd {
> +       struct {
> +               u32 syndrome : 8;
> +               u32 rtype : 2;
> +               u32 alarm : 1;
> +               u32 cwd : 1;
> +               u32 vf_pf_mb : 1;
> +               u32 rsvd0 : 1;
> +               u32 cls : 2;
> +               u32 aid : 6;
> +               u32 unit : 4;
> +               u32 source : 4;
> +               u32 more : 1;
> +               u32 valid : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
> +       (0x20000000 + (x) * 0x1000)
> +#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
> +union dlb2_aqed_pipe_qid_fid_lim {
> +       struct {
> +               u32 qid_fid_limit : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
> +       (0x20080000 + (x) * 0x1000)
> +#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
> +union dlb2_aqed_pipe_qid_hid_width {
> +       struct {
> +               u32 compress_code : 3;
> +               u32 rsvd0 : 29;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
> +#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
> +union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_ATM_QID2CQIDIX_00(x) \
> +       (0x30080000 + (x) * 0x1000)
> +#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
> +#define DLB2_ATM_QID2CQIDIX(x, y) \
> +       (DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
> +#define DLB2_ATM_QID2CQIDIX_NUM 16
> +union dlb2_atm_qid2cqidix_00 {
> +       struct {
> +               u32 cq_p0 : 8;
> +               u32 cq_p1 : 8;
> +               u32 cq_p2 : 8;
> +               u32 cq_p3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
> +#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
> +union dlb2_atm_cfg_arb_weights_rdy_bin {
> +       struct {
> +               u32 bin0 : 8;
> +               u32 bin1 : 8;
> +               u32 bin2 : 8;
> +               u32 bin3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
> +#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
> +union dlb2_atm_cfg_arb_weights_sched_bin {
> +       struct {
> +               u32 bin0 : 8;
> +               u32 bin1 : 8;
> +               u32 bin2 : 8;
> +               u32 bin3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
> +       (0x40000000 + (x) * 0x1000)
> +#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
> +union dlb2_chp_cfg_dir_vas_crd {
> +       struct {
> +               u32 count : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
> +       (0x40080000 + (x) * 0x1000)
> +#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
> +union dlb2_chp_cfg_ldb_vas_crd {
> +       struct {
> +               u32 count : 15;
> +               u32 rsvd0 : 17;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_ORD_QID_SN(x) \
> +       (0x40100000 + (x) * 0x1000)
> +#define DLB2_CHP_ORD_QID_SN_RST 0x0
> +union dlb2_chp_ord_qid_sn {
> +       struct {
> +               u32 sn : 10;
> +               u32 rsvd0 : 22;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_ORD_QID_SN_MAP(x) \
> +       (0x40180000 + (x) * 0x1000)
> +#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
> +union dlb2_chp_ord_qid_sn_map {
> +       struct {
> +               u32 mode : 3;
> +               u32 slot : 4;
> +               u32 rsvz0 : 1;
> +               u32 grp : 1;
> +               u32 rsvz1 : 1;
> +               u32 rsvd0 : 22;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_SN_CHK_ENBL(x) \
> +       (0x40200000 + (x) * 0x1000)
> +#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
> +union dlb2_chp_sn_chk_enbl {
> +       struct {
> +               u32 en : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_DEPTH(x) \
> +       (0x40280000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
> +union dlb2_chp_dir_cq_depth {
> +       struct {
> +               u32 depth : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
> +       (0x40300000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
> +union dlb2_chp_dir_cq_int_depth_thrsh {
> +       struct {
> +               u32 depth_threshold : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
> +       (0x40380000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
> +union dlb2_chp_dir_cq_int_enb {
> +       struct {
> +               u32 en_tim : 1;
> +               u32 en_depth : 1;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
> +       (0x40480000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
> +union dlb2_chp_dir_cq_tmr_thrsh {
> +       struct {
> +               u32 thrsh_0 : 1;
> +               u32 thrsh_13_1 : 13;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
> +       (0x40500000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
> +union dlb2_chp_dir_cq_tkn_depth_sel {
> +       struct {
> +               u32 token_depth_select : 4;
> +               u32 rsvd0 : 28;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
> +       (0x40580000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
> +union dlb2_chp_dir_cq_wd_enb {
> +       struct {
> +               u32 wd_enable : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_WPTR(x) \
> +       (0x40600000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
> +union dlb2_chp_dir_cq_wptr {
> +       struct {
> +               u32 write_pointer : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ2VAS(x) \
> +       (0x40680000 + (x) * 0x1000)
> +#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
> +union dlb2_chp_dir_cq2vas {
> +       struct {
> +               u32 cq2vas : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_HIST_LIST_BASE(x) \
> +       (0x40700000 + (x) * 0x1000)
> +#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
> +union dlb2_chp_hist_list_base {
> +       struct {
> +               u32 base : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_HIST_LIST_LIM(x) \
> +       (0x40780000 + (x) * 0x1000)
> +#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
> +union dlb2_chp_hist_list_lim {
> +       struct {
> +               u32 limit : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
> +       (0x40800000 + (x) * 0x1000)
> +#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
> +union dlb2_chp_hist_list_pop_ptr {
> +       struct {
> +               u32 pop_ptr : 13;
> +               u32 generation : 1;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
> +       (0x40880000 + (x) * 0x1000)
> +#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
> +union dlb2_chp_hist_list_push_ptr {
> +       struct {
> +               u32 push_ptr : 13;
> +               u32 generation : 1;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_DEPTH(x) \
> +       (0x40900000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
> +union dlb2_chp_ldb_cq_depth {
> +       struct {
> +               u32 depth : 11;
> +               u32 rsvd0 : 21;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
> +       (0x40980000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
> +union dlb2_chp_ldb_cq_int_depth_thrsh {
> +       struct {
> +               u32 depth_threshold : 11;
> +               u32 rsvd0 : 21;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
> +       (0x40a00000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
> +union dlb2_chp_ldb_cq_int_enb {
> +       struct {
> +               u32 en_tim : 1;
> +               u32 en_depth : 1;
> +               u32 rsvd0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
> +       (0x40b00000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
> +union dlb2_chp_ldb_cq_tmr_thrsh {
> +       struct {
> +               u32 thrsh_0 : 1;
> +               u32 thrsh_13_1 : 13;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
> +       (0x40b80000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
> +union dlb2_chp_ldb_cq_tkn_depth_sel {
> +       struct {
> +               u32 token_depth_select : 4;
> +               u32 rsvd0 : 28;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
> +       (0x40c00000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
> +union dlb2_chp_ldb_cq_wd_enb {
> +       struct {
> +               u32 wd_enable : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_WPTR(x) \
> +       (0x40c80000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
> +union dlb2_chp_ldb_cq_wptr {
> +       struct {
> +               u32 write_pointer : 11;
> +               u32 rsvd0 : 21;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ2VAS(x) \
> +       (0x40d00000 + (x) * 0x1000)
> +#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
> +union dlb2_chp_ldb_cq2vas {
> +       struct {
> +               u32 cq2vas : 5;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
> +#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
> +union dlb2_chp_cfg_chp_csr_ctrl {
> +       struct {
> +               u32 int_cor_alarm_dis : 1;
> +               u32 int_cor_synd_dis : 1;
> +               u32 int_uncr_alarm_dis : 1;
> +               u32 int_unc_synd_dis : 1;
> +               u32 int_inf0_alarm_dis : 1;
> +               u32 int_inf0_synd_dis : 1;
> +               u32 int_inf1_alarm_dis : 1;
> +               u32 int_inf1_synd_dis : 1;
> +               u32 int_inf2_alarm_dis : 1;
> +               u32 int_inf2_synd_dis : 1;
> +               u32 int_inf3_alarm_dis : 1;
> +               u32 int_inf3_synd_dis : 1;
> +               u32 int_inf4_alarm_dis : 1;
> +               u32 int_inf4_synd_dis : 1;
> +               u32 int_inf5_alarm_dis : 1;
> +               u32 int_inf5_synd_dis : 1;
> +               u32 dlb_cor_alarm_enable : 1;
> +               u32 cfg_64bytes_qe_ldb_cq_mode : 1;
> +               u32 cfg_64bytes_qe_dir_cq_mode : 1;
> +               u32 pad_write_ldb : 1;
> +               u32 pad_write_dir : 1;
> +               u32 pad_first_write_ldb : 1;
> +               u32 pad_first_write_dir : 1;
> +               u32 rsvz0 : 9;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
> +#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
> +union dlb2_chp_dir_cq_intr_armed0 {
> +       struct {
> +               u32 armed : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
> +#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
> +union dlb2_chp_dir_cq_intr_armed1 {
> +       struct {
> +               u32 armed : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
> +#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
> +union dlb2_chp_cfg_dir_cq_timer_ctl {
> +       struct {
> +               u32 sample_interval : 8;
> +               u32 enb : 1;
> +               u32 rsvz0 : 23;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
> +#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
> +union dlb2_chp_cfg_dir_wdto_0 {
> +       struct {
> +               u32 wdto : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
> +#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
> +union dlb2_chp_cfg_dir_wdto_1 {
> +       struct {
> +               u32 wdto : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
> +#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
> +union dlb2_chp_cfg_dir_wd_disable0 {
> +       struct {
> +               u32 wd_disable : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
> +#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
> +union dlb2_chp_cfg_dir_wd_disable1 {
> +       struct {
> +               u32 wd_disable : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
> +#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
> +union dlb2_chp_cfg_dir_wd_enb_interval {
> +       struct {
> +               u32 sample_interval : 28;
> +               u32 enb : 1;
> +               u32 rsvz0 : 3;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
> +#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
> +union dlb2_chp_cfg_dir_wd_threshold {
> +       struct {
> +               u32 wd_threshold : 8;
> +               u32 rsvz0 : 24;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
> +#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
> +union dlb2_chp_ldb_cq_intr_armed0 {
> +       struct {
> +               u32 armed : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
> +#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
> +union dlb2_chp_ldb_cq_intr_armed1 {
> +       struct {
> +               u32 armed : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
> +#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
> +union dlb2_chp_cfg_ldb_cq_timer_ctl {
> +       struct {
> +               u32 sample_interval : 8;
> +               u32 enb : 1;
> +               u32 rsvz0 : 23;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
> +#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
> +union dlb2_chp_cfg_ldb_wdto_0 {
> +       struct {
> +               u32 wdto : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
> +#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
> +union dlb2_chp_cfg_ldb_wdto_1 {
> +       struct {
> +               u32 wdto : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
> +#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
> +union dlb2_chp_cfg_ldb_wd_disable0 {
> +       struct {
> +               u32 wd_disable : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
> +#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
> +union dlb2_chp_cfg_ldb_wd_disable1 {
> +       struct {
> +               u32 wd_disable : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
> +#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
> +union dlb2_chp_cfg_ldb_wd_enb_interval {
> +       struct {
> +               u32 sample_interval : 28;
> +               u32 enb : 1;
> +               u32 rsvz0 : 3;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
> +#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
> +union dlb2_chp_cfg_ldb_wd_threshold {
> +       struct {
> +               u32 wd_threshold : 8;
> +               u32 rsvz0 : 24;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
> +#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
> +union dlb2_chp_ctrl_diag_02 {
> +       struct {
> +               u32 egress_credit_status_empty : 1;
> +               u32 egress_credit_status_afull : 1;
> +               u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
> +               u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
> +               u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
> +               u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
> +               u32 chp_lsp_tok_pipe_credit_status_empty : 1;
> +               u32 chp_lsp_tok_pipe_credit_status_afull : 1;
> +               u32 chp_rop_pipe_credit_status_empty : 1;
> +               u32 chp_rop_pipe_credit_status_afull : 1;
> +               u32 qed_to_cq_pipe_credit_status_empty : 1;
> +               u32 qed_to_cq_pipe_credit_status_afull : 1;
> +               u32 egress_lsp_token_credit_status_empty : 1;
> +               u32 egress_lsp_token_credit_status_afull : 1;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
> +union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
> +union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
> +union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
> +#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
> +union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_DP_DIR_CSR_CTRL 0x54000010
> +#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
> +union dlb2_dp_dir_csr_ctrl {
> +       struct {
> +               u32 int_cor_alarm_dis : 1;
> +               u32 int_cor_synd_dis : 1;
> +               u32 int_uncr_alarm_dis : 1;
> +               u32 int_unc_synd_dis : 1;
> +               u32 int_inf0_alarm_dis : 1;
> +               u32 int_inf0_synd_dis : 1;
> +               u32 int_inf1_alarm_dis : 1;
> +               u32 int_inf1_synd_dis : 1;
> +               u32 int_inf2_alarm_dis : 1;
> +               u32 int_inf2_synd_dis : 1;
> +               u32 int_inf3_alarm_dis : 1;
> +               u32 int_inf3_synd_dis : 1;
> +               u32 int_inf4_alarm_dis : 1;
> +               u32 int_inf4_synd_dis : 1;
> +               u32 int_inf5_alarm_dis : 1;
> +               u32 int_inf5_synd_dis : 1;
> +               u32 rsvz0 : 16;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
> +       struct {
> +               u32 pri0 : 8;
> +               u32 pri1 : 8;
> +               u32 pri2 : 8;
> +               u32 pri3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
> +#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
> +union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
> +       (0x96000000 + (x) * 0x4)
> +#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
> +union dlb2_ro_pipe_grp_0_slt_shft {
> +       struct {
> +               u32 change : 10;
> +               u32 rsvd0 : 22;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
> +       (0x96010000 + (x) * 0x4)
> +#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
> +union dlb2_ro_pipe_grp_1_slt_shft {
> +       struct {
> +               u32 change : 10;
> +               u32 rsvd0 : 22;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
> +#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
> +union dlb2_ro_pipe_grp_sn_mode {
> +       struct {
> +               u32 sn_mode_0 : 3;
> +               u32 rszv0 : 5;
> +               u32 sn_mode_1 : 3;
> +               u32 rszv1 : 21;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
> +#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
> +union dlb2_ro_pipe_cfg_ctrl_general_0 {
> +       struct {
> +               u32 unit_single_step_mode : 1;
> +               u32 rr_en : 1;
> +               u32 rszv0 : 30;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ2PRIOV(x) \
> +       (0xa0000000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ2PRIOV_RST 0x0
> +union dlb2_lsp_cq2priov {
> +       struct {
> +               u32 prio : 24;
> +               u32 v : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ2QID0(x) \
> +       (0xa0080000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ2QID0_RST 0x0
> +union dlb2_lsp_cq2qid0 {
> +       struct {
> +               u32 qid_p0 : 7;
> +               u32 rsvd3 : 1;
> +               u32 qid_p1 : 7;
> +               u32 rsvd2 : 1;
> +               u32 qid_p2 : 7;
> +               u32 rsvd1 : 1;
> +               u32 qid_p3 : 7;
> +               u32 rsvd0 : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ2QID1(x) \
> +       (0xa0100000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ2QID1_RST 0x0
> +union dlb2_lsp_cq2qid1 {
> +       struct {
> +               u32 qid_p4 : 7;
> +               u32 rsvd3 : 1;
> +               u32 qid_p5 : 7;
> +               u32 rsvd2 : 1;
> +               u32 qid_p6 : 7;
> +               u32 rsvd1 : 1;
> +               u32 qid_p7 : 7;
> +               u32 rsvd0 : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_DIR_DSBL(x) \
> +       (0xa0180000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
> +union dlb2_lsp_cq_dir_dsbl {
> +       struct {
> +               u32 disabled : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
> +       (0xa0200000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
> +union dlb2_lsp_cq_dir_tkn_cnt {
> +       struct {
> +               u32 count : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
> +       (0xa0280000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
> +union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
> +       struct {
> +               u32 token_depth_select : 4;
> +               u32 disable_wb_opt : 1;
> +               u32 ignore_depth : 1;
> +               u32 rsvd0 : 26;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
> +       (0xa0300000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
> +union dlb2_lsp_cq_dir_tot_sch_cntl {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
> +       (0xa0380000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
> +union dlb2_lsp_cq_dir_tot_sch_cnth {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_DSBL(x) \
> +       (0xa0400000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
> +union dlb2_lsp_cq_ldb_dsbl {
> +       struct {
> +               u32 disabled : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
> +       (0xa0480000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
> +union dlb2_lsp_cq_ldb_infl_cnt {
> +       struct {
> +               u32 count : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
> +       (0xa0500000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
> +union dlb2_lsp_cq_ldb_infl_lim {
> +       struct {
> +               u32 limit : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
> +       (0xa0580000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
> +union dlb2_lsp_cq_ldb_tkn_cnt {
> +       struct {
> +               u32 token_count : 11;
> +               u32 rsvd0 : 21;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
> +       (0xa0600000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
> +union dlb2_lsp_cq_ldb_tkn_depth_sel {
> +       struct {
> +               u32 token_depth_select : 4;
> +               u32 ignore_depth : 1;
> +               u32 rsvd0 : 27;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
> +       (0xa0680000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
> +union dlb2_lsp_cq_ldb_tot_sch_cntl {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
> +       (0xa0700000 + (x) * 0x1000)
> +#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
> +union dlb2_lsp_cq_ldb_tot_sch_cnth {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
> +       (0xa0780000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
> +union dlb2_lsp_qid_dir_max_depth {
> +       struct {
> +               u32 depth : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
> +       (0xa0800000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
> +union dlb2_lsp_qid_dir_tot_enq_cntl {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
> +       (0xa0880000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
> +union dlb2_lsp_qid_dir_tot_enq_cnth {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
> +       (0xa0900000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
> +union dlb2_lsp_qid_dir_enqueue_cnt {
> +       struct {
> +               u32 count : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
> +       (0xa0980000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
> +union dlb2_lsp_qid_dir_depth_thrsh {
> +       struct {
> +               u32 thresh : 13;
> +               u32 rsvd0 : 19;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
> +       (0xa0a00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
> +union dlb2_lsp_qid_aqed_active_cnt {
> +       struct {
> +               u32 count : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
> +       (0xa0a80000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
> +union dlb2_lsp_qid_aqed_active_lim {
> +       struct {
> +               u32 limit : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
> +       (0xa0b00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
> +union dlb2_lsp_qid_atm_tot_enq_cntl {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
> +       (0xa0b80000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
> +union dlb2_lsp_qid_atm_tot_enq_cnth {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
> +       (0xa0c00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
> +union dlb2_lsp_qid_atq_enqueue_cnt {
> +       struct {
> +               u32 count : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
> +       (0xa0c80000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
> +union dlb2_lsp_qid_ldb_enqueue_cnt {
> +       struct {
> +               u32 count : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
> +       (0xa0d00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
> +union dlb2_lsp_qid_ldb_infl_cnt {
> +       struct {
> +               u32 count : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
> +       (0xa0d80000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
> +union dlb2_lsp_qid_ldb_infl_lim {
> +       struct {
> +               u32 limit : 12;
> +               u32 rsvd0 : 20;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID2CQIDIX_00(x) \
> +       (0xa0e00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
> +#define DLB2_LSP_QID2CQIDIX(x, y) \
> +       (DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
> +#define DLB2_LSP_QID2CQIDIX_NUM 16
> +union dlb2_lsp_qid2cqidix_00 {
> +       struct {
> +               u32 cq_p0 : 8;
> +               u32 cq_p1 : 8;
> +               u32 cq_p2 : 8;
> +               u32 cq_p3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID2CQIDIX2_00(x) \
> +       (0xa1600000 + (x) * 0x1000)
> +#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
> +#define DLB2_LSP_QID2CQIDIX2(x, y) \
> +       (DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
> +#define DLB2_LSP_QID2CQIDIX2_NUM 16
> +union dlb2_lsp_qid2cqidix2_00 {
> +       struct {
> +               u32 cq_p0 : 8;
> +               u32 cq_p1 : 8;
> +               u32 cq_p2 : 8;
> +               u32 cq_p3 : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
> +       (0xa1e00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
> +union dlb2_lsp_qid_ldb_replay_cnt {
> +       struct {
> +               u32 count : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
> +       (0xa1f00000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
> +union dlb2_lsp_qid_naldb_max_depth {
> +       struct {
> +               u32 depth : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
> +       (0xa1f80000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
> +union dlb2_lsp_qid_naldb_tot_enq_cntl {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
> +       (0xa2000000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
> +union dlb2_lsp_qid_naldb_tot_enq_cnth {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
> +       (0xa2080000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
> +union dlb2_lsp_qid_atm_depth_thrsh {
> +       struct {
> +               u32 thresh : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
> +       (0xa2100000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
> +union dlb2_lsp_qid_naldb_depth_thrsh {
> +       struct {
> +               u32 thresh : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_QID_ATM_ACTIVE(x) \
> +       (0xa2180000 + (x) * 0x1000)
> +#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
> +union dlb2_lsp_qid_atm_active {
> +       struct {
> +               u32 count : 14;
> +               u32 rsvd0 : 18;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
> +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
> +union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
> +       struct {
> +               u32 pri0_weight : 8;
> +               u32 pri1_weight : 8;
> +               u32 pri2_weight : 8;
> +               u32 pri3_weight : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
> +#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
> +union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
> +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
> +union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
> +       struct {
> +               u32 pri0_weight : 8;
> +               u32 pri1_weight : 8;
> +               u32 pri2_weight : 8;
> +               u32 pri3_weight : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
> +#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
> +union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
> +       struct {
> +               u32 rsvz0 : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
> +#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
> +union dlb2_lsp_ldb_sched_ctrl {
> +       struct {
> +               u32 cq : 8;
> +               u32 qidix : 3;
> +               u32 value : 1;
> +               u32 nalb_haswork_v : 1;
> +               u32 rlist_haswork_v : 1;
> +               u32 slist_haswork_v : 1;
> +               u32 inflight_ok_v : 1;
> +               u32 aqed_nfull_v : 1;
> +               u32 rsvz0 : 15;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
> +#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
> +union dlb2_lsp_dir_sch_cnt_l {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
> +#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
> +union dlb2_lsp_dir_sch_cnt_h {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
> +#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
> +union dlb2_lsp_ldb_sch_cnt_l {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
> +#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
> +union dlb2_lsp_ldb_sch_cnt_h {
> +       struct {
> +               u32 count : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
> +#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
> +union dlb2_lsp_cfg_shdw_ctrl {
> +       struct {
> +               u32 transfer : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
> +       (0xa4000074 + (x) * 4)
> +#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
> +union dlb2_lsp_cfg_shdw_range_cos {
> +       struct {
> +               u32 bw_range : 9;
> +               u32 rsvz0 : 22;
> +               u32 no_extra_credit : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
> +#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
> +union dlb2_lsp_cfg_ctrl_general_0 {
> +       struct {
> +               u32 disab_atq_empty_arb : 1;
> +               u32 inc_tok_unit_idle : 1;
> +               u32 disab_rlist_pri : 1;
> +               u32 inc_cmp_unit_idle : 1;
> +               u32 rsvz0 : 2;
> +               u32 dir_single_op : 1;
> +               u32 dir_half_bw : 1;
> +               u32 dir_single_out : 1;
> +               u32 dir_disab_multi : 1;
> +               u32 atq_single_op : 1;
> +               u32 atq_half_bw : 1;
> +               u32 atq_single_out : 1;
> +               u32 atq_disab_multi : 1;
> +               u32 dirrpl_single_op : 1;
> +               u32 dirrpl_half_bw : 1;
> +               u32 dirrpl_single_out : 1;
> +               u32 lbrpl_single_op : 1;
> +               u32 lbrpl_half_bw : 1;
> +               u32 lbrpl_single_out : 1;
> +               u32 ldb_single_op : 1;
> +               u32 ldb_half_bw : 1;
> +               u32 ldb_disab_multi : 1;
> +               u32 atm_single_sch : 1;
> +               u32 atm_single_cmp : 1;
> +               u32 ldb_ce_tog_arb : 1;
> +               u32 rsvz1 : 1;
> +               u32 smon0_valid_sel : 2;
> +               u32 smon0_value_sel : 1;
> +               u32 smon0_compare_sel : 2;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
> +#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
> +union dlb2_cfg_mstr_diag_reset_sts {
> +       struct {
> +               u32 chp_pf_reset_done : 1;
> +               u32 rop_pf_reset_done : 1;
> +               u32 lsp_pf_reset_done : 1;
> +               u32 nalb_pf_reset_done : 1;
> +               u32 ap_pf_reset_done : 1;
> +               u32 dp_pf_reset_done : 1;
> +               u32 qed_pf_reset_done : 1;
> +               u32 dqed_pf_reset_done : 1;
> +               u32 aqed_pf_reset_done : 1;
> +               u32 sys_pf_reset_done : 1;
> +               u32 pf_reset_active : 1;
> +               u32 flrsm_state : 7;
> +               u32 rsvd0 : 13;
> +               u32 dlb_proc_reset_done : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
> +#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
> +union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
> +       struct {
> +               u32 chp_pipeidle : 1;
> +               u32 rop_pipeidle : 1;
> +               u32 lsp_pipeidle : 1;
> +               u32 nalb_pipeidle : 1;
> +               u32 ap_pipeidle : 1;
> +               u32 dp_pipeidle : 1;
> +               u32 qed_pipeidle : 1;
> +               u32 dqed_pipeidle : 1;
> +               u32 aqed_pipeidle : 1;
> +               u32 sys_pipeidle : 1;
> +               u32 chp_unit_idle : 1;
> +               u32 rop_unit_idle : 1;
> +               u32 lsp_unit_idle : 1;
> +               u32 nalb_unit_idle : 1;
> +               u32 ap_unit_idle : 1;
> +               u32 dp_unit_idle : 1;
> +               u32 qed_unit_idle : 1;
> +               u32 dqed_unit_idle : 1;
> +               u32 aqed_unit_idle : 1;
> +               u32 sys_unit_idle : 1;
> +               u32 rsvd1 : 4;
> +               u32 mstr_cfg_ring_idle : 1;
> +               u32 mstr_cfg_mstr_idle : 1;
> +               u32 mstr_flr_clkreq_b : 1;
> +               u32 mstr_proc_idle : 1;
> +               u32 mstr_proc_idle_masked : 1;
> +               u32 rsvd0 : 2;
> +               u32 dlb_func_idle : 1;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
> +#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
> +union dlb2_cfg_mstr_cfg_pm_status {
> +       struct {
> +               u32 prochot : 1;
> +               u32 pgcb_dlb_idle : 1;
> +               u32 pgcb_dlb_pg_rdy_ack_b : 1;
> +               u32 pmsm_pgcb_req_b : 1;
> +               u32 pgbc_pmc_pg_req_b : 1;
> +               u32 pmc_pgcb_pg_ack_b : 1;
> +               u32 pmc_pgcb_fet_en_b : 1;
> +               u32 pgcb_fet_en_b : 1;
> +               u32 rsvz0 : 1;
> +               u32 rsvz1 : 1;
> +               u32 fuse_force_on : 1;
> +               u32 fuse_proc_disable : 1;
> +               u32 rsvz2 : 1;
> +               u32 rsvz3 : 1;
> +               u32 pm_fsm_d0tod3_ok : 1;
> +               u32 pm_fsm_d3tod0_ok : 1;
> +               u32 dlb_in_d3 : 1;
> +               u32 rsvz4 : 7;
> +               u32 pmsm : 8;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
> +#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
> +union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
> +       struct {
> +               u32 disable : 1;
> +               u32 rsvz0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
> +#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
> +       (0x1000 + (x) * 0x4)
> +#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
> +union dlb2_func_vf_vf2pf_mailbox {
> +       struct {
> +               u32 msg : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
> +#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
> +#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
> +union dlb2_func_vf_vf2pf_mailbox_isr {
> +       struct {
> +               u32 isr : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
> +#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
> +       (0x2000 + (x) * 0x4)
> +#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
> +union dlb2_func_vf_pf2vf_mailbox {
> +       struct {
> +               u32 msg : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
> +#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
> +union dlb2_func_vf_pf2vf_mailbox_isr {
> +       struct {
> +               u32 pf_isr : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
> +#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
> +union dlb2_func_vf_vf_msi_isr_pend {
> +       struct {
> +               u32 isr_pend : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
> +#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
> +union dlb2_func_vf_vf_reset_in_progress {
> +       struct {
> +               u32 reset_in_progress : 1;
> +               u32 rsvd0 : 31;
> +       } field;
> +       u32 val;
> +};
> +
> +#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
> +#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
> +union dlb2_func_vf_vf_msi_isr {
> +       struct {
> +               u32 vf_msi_isr : 32;
> +       } field;
> +       u32 val;
> +};
> +
> +#endif /* __DLB2_REGS_H */
> diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
> new file mode 100644
> index 0000000..6de8b95
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
> @@ -0,0 +1,274 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include "dlb2_user.h"
> +
> +#include "dlb2_hw_types.h"
> +#include "dlb2_mbox.h"
> +#include "dlb2_osdep.h"
> +#include "dlb2_osdep_bitmap.h"
> +#include "dlb2_osdep_types.h"
> +#include "dlb2_regs.h"
> +#include "dlb2_resource.h"
> +
> +static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
> +{
> +       int i;
> +
> +       dlb2_list_init_head(&domain->used_ldb_queues);
> +       dlb2_list_init_head(&domain->used_dir_pq_pairs);
> +       dlb2_list_init_head(&domain->avail_ldb_queues);
> +       dlb2_list_init_head(&domain->avail_dir_pq_pairs);
> +
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               dlb2_list_init_head(&domain->used_ldb_ports[i]);
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               dlb2_list_init_head(&domain->avail_ldb_ports[i]);
> +}
> +
> +static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
> +{
> +       int i;
> +
> +       dlb2_list_init_head(&rsrc->avail_domains);
> +       dlb2_list_init_head(&rsrc->used_domains);
> +       dlb2_list_init_head(&rsrc->avail_ldb_queues);
> +       dlb2_list_init_head(&rsrc->avail_dir_pq_pairs);
> +
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               dlb2_list_init_head(&rsrc->avail_ldb_ports[i]);
> +}
> +
> +void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
> +{
> +       union dlb2_chp_cfg_chp_csr_ctrl r0;
> +
> +       r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
> +
> +       r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
> +
> +       DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
> +}
> +
> +int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
> +                             struct dlb2_get_num_resources_args *arg,
> +                             bool vdev_req,
> +                             unsigned int vdev_id)
> +{
> +       struct dlb2_function_resources *rsrcs;
> +       struct dlb2_bitmap *map;
> +       int i;
> +
> +       if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
> +               return -EINVAL;
> +
> +       if (vdev_req)
> +               rsrcs = &hw->vdev[vdev_id];
> +       else
> +               rsrcs = &hw->pf;
> +
> +       arg->num_sched_domains = rsrcs->num_avail_domains;
> +
> +       arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
> +
> +       arg->num_ldb_ports = 0;
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
> +
> +       arg->num_cos_ldb_ports[0] = rsrcs->num_avail_ldb_ports[0];
> +       arg->num_cos_ldb_ports[1] = rsrcs->num_avail_ldb_ports[1];
> +       arg->num_cos_ldb_ports[2] = rsrcs->num_avail_ldb_ports[2];
> +       arg->num_cos_ldb_ports[3] = rsrcs->num_avail_ldb_ports[3];
> +
> +       arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
> +
> +       arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
> +
> +       map = rsrcs->avail_hist_list_entries;
> +
> +       arg->num_hist_list_entries = dlb2_bitmap_count(map);
> +
> +       arg->max_contiguous_hist_list_entries =
> +               dlb2_bitmap_longest_set_range(map);
> +
> +       arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
> +
> +       arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
> +
> +       return 0;
> +}
> +
> +void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
> +{
> +       union dlb2_chp_cfg_chp_csr_ctrl r0;
> +
> +       r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
> +
> +       r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
> +
> +       DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
> +}
> +
> +void dlb2_resource_free(struct dlb2_hw *hw)
> +{
> +       int i;
> +
> +       if (hw->pf.avail_hist_list_entries)
> +               dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
> +
> +       for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
> +               if (hw->vdev[i].avail_hist_list_entries)
> +                       dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
> +       }
> +}
> +
> +int dlb2_resource_init(struct dlb2_hw *hw)
> +{
> +       struct dlb2_list_entry *list;
> +       unsigned int i;
> +       int ret;
> +
> +       /*
> +        * For optimal load-balancing, ports that map to one or more QIDs in
> +        * common should not be in numerical sequence. This is application
> +        * dependent, but the driver interleaves port IDs as much as possible
> +        * to reduce the likelihood of this. This initial allocation maximizes
> +        * the average distance between an ID and its immediate neighbors (i.e.
> +        * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
> +        * 3, etc.).
> +        */
> +       u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
> +               0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
> +               16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
> +               32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
> +               48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
> +       };
> +
> +       /* Zero-out resource tracking data structures */
> +       memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
> +       memset(&hw->pf, 0, sizeof(hw->pf));
> +
> +       dlb2_init_fn_rsrc_lists(&hw->pf);
> +
> +       for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
> +               memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
> +               dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
> +       }
> +
> +       for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
> +               memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
> +               dlb2_init_domain_rsrc_lists(&hw->domains[i]);
> +               hw->domains[i].parent_func = &hw->pf;
> +       }
> +
> +       /* Give all resources to the PF driver */
> +       hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
> +       for (i = 0; i < hw->pf.num_avail_domains; i++) {
> +               list = &hw->domains[i].func_list;
> +
> +               dlb2_list_add(&hw->pf.avail_domains, list);
> +       }
> +
> +       hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
> +       for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
> +               list = &hw->rsrcs.ldb_queues[i].func_list;
> +
> +               dlb2_list_add(&hw->pf.avail_ldb_queues, list);
> +       }
> +
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               hw->pf.num_avail_ldb_ports[i] =
> +                       DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
> +
> +       for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
> +               int cos_id = i >> DLB2_NUM_COS_DOMAINS;
> +               struct dlb2_ldb_port *port;
> +
> +               port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
> +
> +               dlb2_list_add(&hw->pf.avail_ldb_ports[cos_id],
> +                             &port->func_list);
> +       }
> +
> +       hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
> +       for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
> +               list = &hw->rsrcs.dir_pq_pairs[i].func_list;
> +
> +               dlb2_list_add(&hw->pf.avail_dir_pq_pairs, list);
> +       }
> +
> +       hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
> +       hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
> +       hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
> +
> +       ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
> +                               DLB2_MAX_NUM_HIST_LIST_ENTRIES);
> +       if (ret)
> +               goto unwind;
> +
> +       ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
> +       if (ret)
> +               goto unwind;
> +
> +       for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
> +               ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
> +                                       DLB2_MAX_NUM_HIST_LIST_ENTRIES);
> +               if (ret)
> +                       goto unwind;
> +
> +               ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
> +               if (ret)
> +                       goto unwind;
> +       }
> +
> +       /* Initialize the hardware resource IDs */
> +       for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
> +               hw->domains[i].id.phys_id = i;
> +               hw->domains[i].id.vdev_owned = false;
> +       }
> +
> +       for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
> +               hw->rsrcs.ldb_queues[i].id.phys_id = i;
> +               hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
> +       }
> +
> +       for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
> +               hw->rsrcs.ldb_ports[i].id.phys_id = i;
> +               hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
> +       }
> +
> +       for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
> +               hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
> +               hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
> +       }
> +
> +       for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
> +               hw->rsrcs.sn_groups[i].id = i;
> +               /* Default mode (0) is 64 sequence numbers per queue */
> +               hw->rsrcs.sn_groups[i].mode = 0;
> +               hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
> +               hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
> +       }
> +
> +       for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
> +               hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
> +
> +       return 0;
> +
> +unwind:
> +       dlb2_resource_free(hw);
> +
> +       return ret;
> +}
> +
> +void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
> +{
> +       union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
> +
> +       r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
> +
> +       r0.field.disable = 0;
> +
> +       DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
> +}
> diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.h b/drivers/event/dlb2/pf/base/dlb2_resource.h
> new file mode 100644
> index 0000000..503fdf3
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/base/dlb2_resource.h
> @@ -0,0 +1,1913 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_RESOURCE_H
> +#define __DLB2_RESOURCE_H
> +
> +#include "dlb2_user.h"
> +
> +#include "dlb2_hw_types.h"
> +#include "dlb2_osdep_types.h"
> +
> +/**
> + * dlb2_resource_init() - initialize the device
> + * @hw: pointer to struct dlb2_hw.
> + *
> + * This function initializes the device's software state (pointed to by the hw
> + * argument) and programs global scheduling QoS registers. This function should
> + * be called during driver initialization.
> + *
> + * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
> + * device is reset.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + */
> +int dlb2_resource_init(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_resource_free() - free device state memory
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function frees software state pointed to by dlb2_hw. This function
> + * should be called when resetting the device or unloading the driver.
> + */
> +void dlb2_resource_free(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_resource_reset() - reset in-use resources to their initial state
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function resets in-use resources, and makes them available for use.
> + * All resources go back to their owning function, whether a PF or a VF.
> + */
> +void dlb2_resource_reset(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_create_sched_domain() - create a scheduling domain
> + * @hw: dlb2_hw handle for a particular device.
> + * @args: scheduling domain creation arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function creates a scheduling domain containing the resources specified
> + * in args. The individual resources (queues, ports, credits) can be configured
> + * after creating a scheduling domain.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the domain ID.
> + *
> + * resp->id contains a virtual ID if vdev_request is true.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, or the requested domain name
> + *         is already in use.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
> +                               struct dlb2_create_sched_domain_args *args,
> +                               struct dlb2_cmd_response *resp,
> +                               bool vdev_request,
> +                               unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_create_ldb_queue() - create a load-balanced queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: queue creation arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function creates a load-balanced queue.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the queue ID.
> + *
> + * resp->id contains a virtual ID if vdev_request is true.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, the domain is not configured,
> + *         the domain has already been started, or the requested queue name is
> + *         already in use.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_create_ldb_queue_args *args,
> +                            struct dlb2_cmd_response *resp,
> +                            bool vdev_request,
> +                            unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_create_dir_queue() - create a directed queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: queue creation arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function creates a directed queue.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the queue ID.
> + *
> + * resp->id contains a virtual ID if vdev_request is true.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, the domain is not configured,
> + *         or the domain has already been started.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_create_dir_queue_args *args,
> +                            struct dlb2_cmd_response *resp,
> +                            bool vdev_request,
> +                            unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_create_dir_port() - create a directed port
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port creation arguments.
> + * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function creates a directed port.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the port ID.
> + *
> + * resp->id contains a virtual ID if vdev_request is true.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
> + *         pointer address is not properly aligned, the domain is not
> + *         configured, or the domain has already been started.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_create_dir_port_args *args,
> +                           uintptr_t cq_dma_base,
> +                           struct dlb2_cmd_response *resp,
> +                           bool vdev_request,
> +                           unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_create_ldb_port() - create a load-balanced port
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port creation arguments.
> + * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function creates a load-balanced port.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the port ID.
> + *
> + * resp->id contains a virtual ID if vdev_request is true.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
> + *         pointer address is not properly aligned, the domain is not
> + *         configured, or the domain has already been started.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_create_ldb_port_args *args,
> +                           uintptr_t cq_dma_base,
> +                           struct dlb2_cmd_response *resp,
> +                           bool vdev_request,
> +                           unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_start_domain() - start a scheduling domain
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: start domain arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function starts a scheduling domain, which allows applications to send
> + * traffic through it. Once a domain is started, its resources can no longer be
> + * configured (besides QID remapping and port enable/disable).
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - the domain is not configured, or the domain is already started.
> + */
> +int dlb2_hw_start_domain(struct dlb2_hw *hw,
> +                        u32 domain_id,
> +                        struct dlb2_start_domain_args *args,
> +                        struct dlb2_cmd_response *resp,
> +                        bool vdev_request,
> +                        unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: map QID arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to schedule QEs from the specified queue
> + * to the specified port. Each load-balanced port can be mapped to up to 8
> + * queues; each load-balanced queue can potentially map to all the
> + * load-balanced ports.
> + *
> + * A successful return does not necessarily mean the mapping was configured. If
> + * this function is unable to immediately map the queue to the port, it will
> + * add the requested operation to a per-port list of pending map/unmap
> + * operations, and (if it's not already running) launch a kernel thread that
> + * periodically attempts to process all pending operations. In a sense, this is
> + * an asynchronous function.
> + *
> + * This asynchronicity creates two views of the state of hardware: the actual
> + * hardware state and the requested state (as if every request completed
> + * immediately). If there are any pending map/unmap operations, the requested
> + * state will differ from the actual state. All validation is performed with
> + * respect to the pending state; for instance, if there are 8 pending map
> + * operations for port X, a request for a 9th will fail because a load-balanced
> + * port can only map up to 8 queues.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
> + *         the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_map_qid(struct dlb2_hw *hw,
> +                   u32 domain_id,
> +                   struct dlb2_map_qid_args *args,
> +                   struct dlb2_cmd_response *resp,
> +                   bool vdev_request,
> +                   unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: unmap QID arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to stop scheduling QEs from the specified
> + * queue to the specified port.
> + *
> + * A successful return does not necessarily mean the mapping was removed. If
> + * this function is unable to immediately unmap the queue from the port, it
> + * will add the requested operation to a per-port list of pending map/unmap
> + * operations, and (if it's not already running) launch a kernel thread that
> + * periodically attempts to process all pending operations. See
> + * dlb2_hw_map_qid() for more details.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
> + *         the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
> +                     u32 domain_id,
> +                     struct dlb2_unmap_qid_args *args,
> +                     struct dlb2_cmd_response *resp,
> +                     bool vdev_request,
> +                     unsigned int vdev_id);
> +
> +/**
> + * dlb2_finish_unmap_qid_procedures() - finish any pending unmap procedures
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function attempts to finish any outstanding unmap procedures.
> + * This function should be called by the kernel thread responsible for
> + * finishing map/unmap procedures.
> + *
> + * Return:
> + * Returns the number of procedures that weren't completed.
> + */
> +unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_finish_map_qid_procedures() - finish any pending map procedures
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function attempts to finish any outstanding map procedures.
> + * This function should be called by the kernel thread responsible for
> + * finishing map/unmap procedures.
> + *
> + * Return:
> + * Returns the number of procedures that weren't completed.
> + */
> +unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port enable arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to schedule QEs to a load-balanced port.
> + * Ports are enabled by default.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid or the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_enable_ldb_port_args *args,
> +                           struct dlb2_cmd_response *resp,
> +                           bool vdev_request,
> +                           unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port disable arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to stop scheduling QEs to a load-balanced
> + * port. Ports are enabled by default.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid or the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_disable_ldb_port_args *args,
> +                            struct dlb2_cmd_response *resp,
> +                            bool vdev_request,
> +                            unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port enable arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to schedule QEs to a directed port.
> + * Ports are enabled by default.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid or the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_enable_dir_port_args *args,
> +                           struct dlb2_cmd_response *resp,
> +                           bool vdev_request,
> +                           unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: port disable arguments.
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function configures the DLB to stop scheduling QEs to a directed port.
> + * Ports are enabled by default.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid or the domain is not configured.
> + * EFAULT - Internal error (resp->status not set).
> + */
> +int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_disable_dir_port_args *args,
> +                            struct dlb2_cmd_response *resp,
> +                            bool vdev_request,
> +                            unsigned int vdev_id);
> +
> +/**
> + * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
> + *                                     interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: load-balanced port ID.
> + * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
> + *         else a value up to 64.
> + * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
> + * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
> + *     virtual port ID to a physical port ID. Ignored if mode is not MSI.
> + * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
> + * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
> + *     greater than 0.
> + *
> + * This function configures the DLB registers for load-balanced CQ's
> + * interrupts. This doesn't enable the CQ's interrupt; that can be done with
> + * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid.
> + */
> +int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
> +                                   int port_id,
> +                                   int vector,
> +                                   int mode,
> +                                   unsigned int vf,
> +                                   unsigned int owner_vf,
> +                                   u16 threshold);
> +
> +/**
> + * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: load-balanced port ID.
> + * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
> + *         else a value up to 64.
> + * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
> + * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
> + *     virtual port ID to a physical port ID. Ignored if mode is not MSI.
> + * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
> + * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
> + *     greater than 0.
> + *
> + * This function configures the DLB registers for directed CQ's interrupts.
> + * This doesn't enable the CQ's interrupt; that can be done with
> + * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - The port ID is invalid.
> + */
> +int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
> +                                   int port_id,
> +                                   int vector,
> +                                   int mode,
> +                                   unsigned int vf,
> +                                   unsigned int owner_vf,
> +                                   u16 threshold);
> +
> +/**
> + * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + */
> +void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + */
> +void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
> + *
> + * This function configures the hardware to use either packed or compressed
> + * mode. This function should not be called if using MSI interrupts.
> + */
> +void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
> +
> +/**
> + * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + * @vector: interrupt vector.
> + *
> + * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
> + * acked in dlb2_ack_compressed_cq_intr().
> + */
> +void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
> +
> +/**
> + * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: port ID
> + * @is_ldb: true for load-balanced port, false for a directed port
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function arms the CQ's interrupt. The CQ must be configured prior to
> + * calling this function.
> + *
> + * The function does no parameter validation; that is the caller's
> + * responsibility.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return: returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - Invalid port ID.
> + */
> +int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
> +                         int port_id,
> +                         bool is_ldb,
> +                         bool vdev_request,
> +                         unsigned int vdev_id);
> +
> +/**
> + * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
> + * @hw: dlb2_hw handle for a particular device.
> + * @ldb_interrupts: 2-entry array of u32 bitmaps
> + * @dir_interrupts: 4-entry array of u32 bitmaps
> + *
> + * This function can be called from a compressed CQ interrupt handler to
> + * determine which CQ interrupts have fired. The caller should take appropriate
> + * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
> + * with dlb2_ack_compressed_cq_intr().
> + */
> +void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
> +                                        u32 *ldb_interrupts,
> +                                        u32 *dir_interrupts);
> +
> +/**
> + * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + * @ldb_interrupts: 2-entry array of u32 bitmaps
> + * @dir_interrupts: 4-entry array of u32 bitmaps
> + *
> + * This function ACKs compressed CQ interrupts. Its arguments should be the
> + * same ones passed to dlb2_read_compressed_cq_intr_status().
> + */
> +void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
> +                                u32 *ldb_interrupts,
> +                                u32 *dir_interrupts);
> +
> +/**
> + * dlb2_read_vf_intr_status() - read the VF interrupt status register
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function can be called from a VF's interrupt handler to determine
> + * which interrupts have fired. The first 31 bits correspond to CQ interrupt
> + * vectors, and the final bit is for the PF->VF mailbox interrupt vector.
> + *
> + * Return:
> + * Returns a bit vector indicating which interrupt vectors are active.
> + */
> +u32 dlb2_read_vf_intr_status(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_ack_vf_intr_status() - ack VF interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + * @interrupts: 32-bit bitmap
> + *
> + * This function ACKs a VF's interrupts. Its interrupts argument should be the
> + * value returned by dlb2_read_vf_intr_status().
> + */
> +void dlb2_ack_vf_intr_status(struct dlb2_hw *hw, u32 interrupts);
> +
> +/**
> + * dlb2_ack_vf_msi_intr() - ack VF MSI interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + * @interrupts: 32-bit bitmap
> + *
> + * This function clears the VF's MSI interrupt pending register. Its interrupts
> + * argument should be contain the MSI vectors to ACK. For example, if MSI MME
> + * is in mode 0, then one bit 0 should ever be set.
> + */
> +void dlb2_ack_vf_msi_intr(struct dlb2_hw *hw, u32 interrupts);
> +
> +/**
> + * dlb2_ack_pf_mbox_int() - ack PF->VF mailbox interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * When done processing the PF mailbox request, this function unsets
> + * the PF's mailbox ISR register.
> + */
> +void dlb2_ack_pf_mbox_int(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_read_vdev_to_pf_int_bitvec() - return a bit vector of all requesting
> + *                                     vdevs
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * When the vdev->PF ISR fires, this function can be called to determine which
> + * vdev(s) are requesting service. This bitvector must be passed to
> + * dlb2_ack_vdev_to_pf_int() when processing is complete for all requesting
> + * vdevs.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns a bit vector indicating which VFs (0-15) have requested service.
> + */
> +u32 dlb2_read_vdev_to_pf_int_bitvec(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_ack_vdev_mbox_int() - ack processed vdev->PF mailbox interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + * @bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
> + *
> + * When done processing all VF mailbox requests, this function unsets the VF's
> + * mailbox ISR register.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_ack_vdev_mbox_int(struct dlb2_hw *hw, u32 bitvec);
> +
> +/**
> + * dlb2_read_vf_flr_int_bitvec() - return a bit vector of all VFs requesting
> + *                                 FLR
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * When the VF FLR ISR fires, this function can be called to determine which
> + * VF(s) are requesting FLRs. This bitvector must passed to
> + * dlb2_ack_vf_flr_int() when processing is complete for all requesting VFs.
> + *
> + * Return:
> + * Returns a bit vector indicating which VFs (0-15) have requested FLRs.
> + */
> +u32 dlb2_read_vf_flr_int_bitvec(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_ack_vf_flr_int() - ack processed VF<->PF interrupt(s)
> + * @hw: dlb2_hw handle for a particular device.
> + * @bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
> + *
> + * When done processing all VF FLR requests, this function unsets the VF's FLR
> + * ISR register.
> + */
> +void dlb2_ack_vf_flr_int(struct dlb2_hw *hw, u32 bitvec);
> +
> +/**
> + * dlb2_ack_vdev_to_pf_int() - ack processed VF mbox and FLR interrupt(s)
> + * @hw: dlb2_hw handle for a particular device.
> + * @mbox_bitvec: bit vector returned by dlb2_read_vdev_to_pf_int_bitvec()
> + * @flr_bitvec: bit vector returned by dlb2_read_vf_flr_int_bitvec()
> + *
> + * When done processing all VF requests, this function communicates to the
> + * hardware that processing is complete.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_ack_vdev_to_pf_int(struct dlb2_hw *hw,
> +                            u32 mbox_bitvec,
> +                            u32 flr_bitvec);
> +
> +/**
> + * dlb2_process_wdt_interrupt() - process watchdog timer interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function reads the watchdog timer interrupt cause registers to
> + * determine which port(s) had a watchdog timeout, and notifies the
> + * application(s) that own the port(s).
> + */
> +void dlb2_process_wdt_interrupt(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_process_alarm_interrupt() - process an alarm interrupt
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function reads and logs the alarm syndrome, then acks the interrupt.
> + * This function should be called from the alarm interrupt handler when
> + * interrupt vector DLB2_INT_ALARM fires.
> + */
> +void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function reads the alarm syndrome, logs it, notifies user-space, and
> + * acks the interrupt. This function should be called from the alarm interrupt
> + * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
> + *
> + * Return:
> + * Returns true if an ingress error interrupt occurred, false otherwise
> + */
> +bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @group_id: sequence number group ID.
> + *
> + * This function returns the configured number of sequence numbers per queue
> + * for the specified group.
> + *
> + * Return:
> + * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
> + */
> +int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
> +                                   unsigned int group_id);
> +
> +/**
> + * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
> + * @hw: dlb2_hw handle for a particular device.
> + * @group_id: sequence number group ID.
> + *
> + * This function returns the group's number of in-use slots (i.e. load-balanced
> + * queues using the specified group).
> + *
> + * Return:
> + * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
> + */
> +int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
> +                                            unsigned int group_id);
> +
> +/**
> + * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @group_id: sequence number group ID.
> + * @val: requested amount of sequence numbers per queue.
> + *
> + * This function configures the group's number of sequence numbers per queue.
> + * val can be a power-of-two between 32 and 1024, inclusive. This setting can
> + * be configured until the first ordered load-balanced queue is configured, at
> + * which point the configuration is locked.
> + *
> + * Return:
> + * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
> + * ordered queue is configured.
> + */
> +int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
> +                                   unsigned int group_id,
> +                                   unsigned long val);
> +
> +/**
> + * dlb2_reset_domain() - reset a scheduling domain
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function resets and frees a DLB 2.0 scheduling domain and its associated
> + * resources.
> + *
> + * Pre-condition: the driver must ensure software has stopped sending QEs
> + * through this domain's producer ports before invoking this function, or
> + * undefined behavior will result.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, -1 otherwise.
> + *
> + * EINVAL - Invalid domain ID, or the domain is not configured.
> + * EFAULT - Internal error. (Possibly caused if software is the pre-condition
> + *         is not met.)
> + * ETIMEDOUT - Hardware component didn't reset in the expected time.
> + */
> +int dlb2_reset_domain(struct dlb2_hw *hw,
> +                     u32 domain_id,
> +                     bool vdev_request,
> +                     unsigned int vdev_id);
> +
> +/**
> + * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @port_id: indicates whether this request came from a VF.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns whether a load-balanced port is owned by a specified
> + * domain.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 if false, 1 if true, <0 otherwise.
> + *
> + * EINVAL - Invalid domain or port ID, or the domain is not configured.
> + */
> +int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
> +                                 u32 domain_id,
> +                                 u32 port_id,
> +                                 bool vdev_request,
> +                                 unsigned int vdev_id);
> +
> +/**
> + * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @port_id: indicates whether this request came from a VF.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns whether a directed port is owned by a specified
> + * domain.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 if false, 1 if true, <0 otherwise.
> + *
> + * EINVAL - Invalid domain or port ID, or the domain is not configured.
> + */
> +int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
> +                                 u32 domain_id,
> +                                 u32 port_id,
> +                                 bool vdev_request,
> +                                 unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_get_num_resources() - query the PCI function's available resources
> + * @hw: dlb2_hw handle for a particular device.
> + * @arg: pointer to resource counts.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns the number of available resources for the PF or for a
> + * VF.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
> + * invalid.
> + */
> +int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
> +                             struct dlb2_get_num_resources_args *arg,
> +                             bool vdev_request,
> +                             unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_get_num_used_resources() - query the PCI function's used resources
> + * @hw: dlb2_hw handle for a particular device.
> + * @arg: pointer to resource counts.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns the number of resources in use by the PF or a VF. It
> + * fills in the fields that args points to, except the following:
> + * - max_contiguous_atomic_inflights
> + * - max_contiguous_hist_list_entries
> + * - max_contiguous_ldb_credits
> + * - max_contiguous_dir_credits
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
> + * invalid.
> + */
> +int dlb2_hw_get_num_used_resources(struct dlb2_hw *hw,
> +                                  struct dlb2_get_num_resources_args *arg,
> +                                  bool vdev_request,
> +                                  unsigned int vdev_id);
> +
> +/**
> + * dlb2_send_async_vdev_to_pf_msg() - (vdev only) send a mailbox message to
> + *                                    the PF
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function sends a VF->PF mailbox message. It is asynchronous, so it
> + * returns once the message is sent but potentially before the PF has processed
> + * the message. The caller must call dlb2_vdev_to_pf_complete() to determine
> + * when the PF has finished processing the request.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_send_async_vdev_to_pf_msg(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_vdev_to_pf_complete() - check the status of an asynchronous mailbox
> + *                              request
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function returns a boolean indicating whether the PF has finished
> + * processing a VF->PF mailbox request. It should only be called after sending
> + * an asynchronous request with dlb2_send_async_vdev_to_pf_msg().
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +bool dlb2_vdev_to_pf_complete(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_vf_flr_complete() - check the status of a VF FLR
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function returns a boolean indicating whether the PF has finished
> + * executing the VF FLR. It should only be called after setting the VF's FLR
> + * bit.
> + */
> +bool dlb2_vf_flr_complete(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_send_async_pf_to_vdev_msg() - (PF only) send a mailbox message to a
> + *                                     vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + *
> + * This function sends a PF->vdev mailbox message. It is asynchronous, so it
> + * returns once the message is sent but potentially before the vdev has
> + * processed the message. The caller must call dlb2_pf_to_vdev_complete() to
> + * determine when the vdev has finished processing the request.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_send_async_pf_to_vdev_msg(struct dlb2_hw *hw, unsigned int vdev_id);
> +
> +/**
> + * dlb2_pf_to_vdev_complete() - check the status of an asynchronous mailbox
> + *                            request
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + *
> + * This function returns a boolean indicating whether the vdev has finished
> + * processing a PF->vdev mailbox request. It should only be called after
> + * sending an asynchronous request with dlb2_send_async_pf_to_vdev_msg().
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +bool dlb2_pf_to_vdev_complete(struct dlb2_hw *hw, unsigned int vdev_id);
> +
> +/**
> + * dlb2_pf_read_vf_mbox_req() - (PF only) read a VF->PF mailbox request
> + * @hw: dlb2_hw handle for a particular device.
> + * @vf_id: VF ID.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies one of the PF's VF->PF mailboxes into the array pointed
> + * to by data.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
> + */
> +int dlb2_pf_read_vf_mbox_req(struct dlb2_hw *hw,
> +                            unsigned int vf_id,
> +                            void *data,
> +                            int len);
> +
> +/**
> + * dlb2_pf_read_vf_mbox_resp() - (PF only) read a VF->PF mailbox response
> + * @hw: dlb2_hw handle for a particular device.
> + * @vf_id: VF ID.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies one of the PF's VF->PF mailboxes into the array pointed
> + * to by data.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
> + */
> +int dlb2_pf_read_vf_mbox_resp(struct dlb2_hw *hw,
> +                             unsigned int vf_id,
> +                             void *data,
> +                             int len);
> +
> +/**
> + * dlb2_pf_write_vf_mbox_resp() - (PF only) write a PF->VF mailbox response
> + * @hw: dlb2_hw handle for a particular device.
> + * @vf_id: VF ID.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the user-provided message data into of the PF's VF->PF
> + * mailboxes.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
> + */
> +int dlb2_pf_write_vf_mbox_resp(struct dlb2_hw *hw,
> +                              unsigned int vf_id,
> +                              void *data,
> +                              int len);
> +
> +/**
> + * dlb2_pf_write_vf_mbox_req() - (PF only) write a PF->VF mailbox request
> + * @hw: dlb2_hw handle for a particular device.
> + * @vf_id: VF ID.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the user-provided message data into of the PF's VF->PF
> + * mailboxes.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
> + */
> +int dlb2_pf_write_vf_mbox_req(struct dlb2_hw *hw,
> +                             unsigned int vf_id,
> +                             void *data,
> +                             int len);
> +
> +/**
> + * dlb2_vf_read_pf_mbox_resp() - (VF only) read a PF->VF mailbox response
> + * @hw: dlb2_hw handle for a particular device.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the VF's PF->VF mailbox into the array pointed to by
> + * data.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_PF2VF_RESP_BYTES.
> + */
> +int dlb2_vf_read_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
> +
> +/**
> + * dlb2_vf_read_pf_mbox_req() - (VF only) read a PF->VF mailbox request
> + * @hw: dlb2_hw handle for a particular device.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the VF's PF->VF mailbox into the array pointed to by
> + * data.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_PF2VF_REQ_BYTES.
> + */
> +int dlb2_vf_read_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
> +
> +/**
> + * dlb2_vf_write_pf_mbox_req() - (VF only) write a VF->PF mailbox request
> + * @hw: dlb2_hw handle for a particular device.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the user-provided message data into of the VF's PF->VF
> + * mailboxes.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_VF2PF_REQ_BYTES.
> + */
> +int dlb2_vf_write_pf_mbox_req(struct dlb2_hw *hw, void *data, int len);
> +
> +/**
> + * dlb2_vf_write_pf_mbox_resp() - (VF only) write a VF->PF mailbox response
> + * @hw: dlb2_hw handle for a particular device.
> + * @data: pointer to message data.
> + * @len: size, in bytes, of the data array.
> + *
> + * This function copies the user-provided message data into of the VF's PF->VF
> + * mailboxes.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * EINVAL - len >= DLB2_VF2PF_RESP_BYTES.
> + */
> +int dlb2_vf_write_pf_mbox_resp(struct dlb2_hw *hw, void *data, int len);
> +
> +/**
> + * dlb2_reset_vdev() - reset the hardware owned by a virtual device
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function resets the hardware owned by a vdev, by resetting the vdev's
> + * domains one by one.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +int dlb2_reset_vdev(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_vdev_is_locked() - check whether the vdev's resources are locked
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function returns whether or not the vdev's resource assignments are
> + * locked. If locked, no resources can be added to or subtracted from the
> + * group.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +bool dlb2_vdev_is_locked(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_lock_vdev() - lock the vdev's resources
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function sets a flag indicating that the vdev is using its resources.
> + * When the vdev is locked, its resource assignment cannot be changed.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_lock_vdev(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_unlock_vdev() - unlock the vdev's resources
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function unlocks the vdev's resource assignment, allowing it to be
> + * modified.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + */
> +void dlb2_unlock_vdev(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_update_vdev_sched_domains() - update the domains assigned to a vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of scheduling domains to assign to this vdev
> + *
> + * This function assigns num scheduling domains to the specified vdev. If the
> + * vdev already has domains assigned, this existing assignment is adjusted
> + * accordingly.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_sched_domains(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_ldb_queues() - update the LDB queues assigned to a vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of LDB queues to assign to this vdev
> + *
> + * This function assigns num LDB queues to the specified vdev. If the vdev
> + * already has LDB queues assigned, this existing assignment is adjusted
> + * accordingly.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_ldb_queues(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_ldb_ports() - update the LDB ports assigned to a vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of LDB ports to assign to this vdev
> + *
> + * This function assigns num LDB ports to the specified vdev. If the vdev
> + * already has LDB ports assigned, this existing assignment is adjusted
> + * accordingly.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_ldb_ports(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_ldb_cos_ports() - update the LDB ports assigned to a vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @cos: class-of-service ID
> + * @num: number of LDB ports to assign to this vdev
> + *
> + * This function assigns num LDB ports from class-of-service cos to the
> + * specified vdev. If the vdev already has LDB ports from this class-of-service
> + * assigned, this existing assignment is adjusted accordingly.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_ldb_cos_ports(struct dlb2_hw *hw,
> +                                  u32 id,
> +                                  u32 cos,
> +                                  u32 num);
> +
> +/**
> + * dlb2_update_vdev_dir_ports() - update the DIR ports assigned to a vdev
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of DIR ports to assign to this vdev
> + *
> + * This function assigns num DIR ports to the specified vdev. If the vdev
> + * already has DIR ports assigned, this existing assignment is adjusted
> + * accordingly.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_dir_ports(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_ldb_credits() - update the vdev's assigned LDB credits
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of LDB credit credits to assign to this vdev
> + *
> + * This function assigns num LDB credit to the specified vdev. If the vdev
> + * already has LDB credits assigned, this existing assignment is adjusted
> + * accordingly. vdevs are assigned a contiguous chunk of credits, so this
> + * function may fail if a sufficiently large contiguous chunk is not available.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_ldb_credits(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_dir_credits() - update the vdev's assigned DIR credits
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of DIR credits to assign to this vdev
> + *
> + * This function assigns num DIR credit to the specified vdev. If the vdev
> + * already has DIR credits assigned, this existing assignment is adjusted
> + * accordingly. vdevs are assigned a contiguous chunk of credits, so this
> + * function may fail if a sufficiently large contiguous chunk is not available.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_dir_credits(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_hist_list_entries() - update the vdev's assigned HL entries
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of history list entries to assign to this vdev
> + *
> + * This function assigns num history list entries to the specified vdev. If the
> + * vdev already has history list entries assigned, this existing assignment is
> + * adjusted accordingly. vdevs are assigned a contiguous chunk of entries, so
> + * this function may fail if a sufficiently large contiguous chunk is not
> + * available.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_hist_list_entries(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_update_vdev_atomic_inflights() - update the vdev's atomic inflights
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + * @num: number of atomic inflights to assign to this vdev
> + *
> + * This function assigns num atomic inflights to the specified vdev. If the vdev
> + * already has atomic inflights assigned, this existing assignment is adjusted
> + * accordingly. vdevs are assigned a contiguous chunk of entries, so this
> + * function may fail if a sufficiently large contiguous chunk is not available.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid, or the requested number of resources are
> + *         unavailable.
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_update_vdev_atomic_inflights(struct dlb2_hw *hw, u32 id, u32 num);
> +
> +/**
> + * dlb2_reset_vdev_resources() - reassign the vdev's resources to the PF
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function takes any resources currently assigned to the vdev and
> + * reassigns them to the PF.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, <0 otherwise.
> + *
> + * Errors:
> + * EINVAL - id is invalid
> + * EPERM  - The vdev's resource assignment is locked and cannot be changed.
> + */
> +int dlb2_reset_vdev_resources(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_notify_vf() - send an alarm to a VF
> + * @hw: dlb2_hw handle for a particular device.
> + * @vf_id: VF ID
> + * @notification: notification
> + *
> + * This function sends a notification (as defined in dlb2_mbox.h) to a VF.
> + *
> + * Return:
> + * Returns 0 upon success, <0 if the VF doesn't ACK the PF->VF interrupt.
> + */
> +int dlb2_notify_vf(struct dlb2_hw *hw,
> +                  unsigned int vf_id,
> +                  u32 notification);
> +
> +/**
> + * dlb2_vdev_in_use() - query whether a virtual device is in use
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual device ID
> + *
> + * This function sends a mailbox request to the vdev to query whether the vdev
> + * is in use.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 for false, 1 for true, and <0 if the mailbox request times out or
> + * an internal error occurs.
> + */
> +int dlb2_vdev_in_use(struct dlb2_hw *hw, unsigned int id);
> +
> +/**
> + * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * Clearing the PMCSR must be done at initialization to make the device fully
> + * operational.
> + */
> +void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: queue depth args
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns the depth of a load-balanced queue.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the depth.
> + *
> + * Errors:
> + * EINVAL - Invalid domain ID or queue ID.
> + */
> +int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
> +                               u32 domain_id,
> +                               struct dlb2_get_ldb_queue_depth_args *args,
> +                               struct dlb2_cmd_response *resp,
> +                               bool vdev_request,
> +                               unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: queue depth args
> + * @resp: response structure.
> + * @vdev_request: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * This function returns the depth of a directed queue.
> + *
> + * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
> + * device.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the depth.
> + *
> + * Errors:
> + * EINVAL - Invalid domain ID or queue ID.
> + */
> +int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
> +                               u32 domain_id,
> +                               struct dlb2_get_dir_queue_depth_args *args,
> +                               struct dlb2_cmd_response *resp,
> +                               bool vdev_request,
> +                               unsigned int vdev_id);
> +
> +enum dlb2_virt_mode {
> +       DLB2_VIRT_NONE,
> +       DLB2_VIRT_SRIOV,
> +       DLB2_VIRT_SIOV,
> +
> +       /* NUM_DLB2_VIRT_MODES must be last */
> +       NUM_DLB2_VIRT_MODES,
> +};
> +
> +/**
> + * dlb2_hw_set_virt_mode() - set the device's virtualization mode
> + * @hw: dlb2_hw handle for a particular device.
> + * @mode: either none, SR-IOV, or Scalable IOV.
> + *
> + * This function sets the virtualization mode of the device. This controls
> + * whether the device uses a software or hardware mailbox.
> + *
> + * This should be called by the PF driver when either SR-IOV or Scalable IOV is
> + * selected as the virtualization mechanism, and by the VF/VDEV driver during
> + * initialization after recognizing itself as an SR-IOV or Scalable IOV device.
> + *
> + * Errors:
> + * EINVAL - Invalid mode.
> + */
> +int dlb2_hw_set_virt_mode(struct dlb2_hw *hw, enum dlb2_virt_mode mode);
> +
> +/**
> + * dlb2_hw_get_virt_mode() - get the device's virtualization mode
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function gets the virtualization mode of the device.
> + */
> +enum dlb2_virt_mode dlb2_hw_get_virt_mode(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_get_ldb_port_phys_id() - get a physical port ID from its virt ID
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual port ID.
> + * @vdev_id: vdev ID.
> + *
> + * Return:
> + * Returns >= 0 upon success, -1 otherwise.
> + */
> +s32 dlb2_hw_get_ldb_port_phys_id(struct dlb2_hw *hw,
> +                                u32 id,
> +                                unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_get_dir_port_phys_id() - get a physical port ID from its virt ID
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: virtual port ID.
> + * @vdev_id: vdev ID.
> + *
> + * Return:
> + * Returns >= 0 upon success, -1 otherwise.
> + */
> +s32 dlb2_hw_get_dir_port_phys_id(struct dlb2_hw *hw,
> +                                u32 id,
> +                                unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_register_sw_mbox() - register a software mailbox
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + * @vdev2pf_mbox: pointer to a 4KB memory page used for vdev->PF communication.
> + * @pf2vdev_mbox: pointer to a 4KB memory page used for PF->vdev communication.
> + * @pf2vdev_inject: callback function for injecting a PF->vdev interrupt.
> + * @inject_arg: user argument for pf2vdev_inject callback.
> + *
> + * When Scalable IOV is enabled, the VDCM must register a software mailbox for
> + * every virtual device during vdev creation.
> + *
> + * This function notifies the driver to use a software mailbox using the
> + * provided pointers, instead of the device's hardware mailbox. When the driver
> + * calls mailbox functions like dlb2_pf_write_vf_mbox_req(), the request will
> + * go to the software mailbox instead of the hardware one. This is used in
> + * Scalable IOV virtualization.
> + */
> +void dlb2_hw_register_sw_mbox(struct dlb2_hw *hw,
> +                             unsigned int vdev_id,
> +                             u32 *vdev2pf_mbox,
> +                             u32 *pf2vdev_mbox,
> +                             void (*pf2vdev_inject)(void *),
> +                             void *inject_arg);
> +
> +/**
> + * dlb2_hw_unregister_sw_mbox() - unregister a software mailbox
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + *
> + * This function notifies the driver to stop using a previously registered
> + * software mailbox.
> + */
> +void dlb2_hw_unregister_sw_mbox(struct dlb2_hw *hw, unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_setup_cq_ims_entry() - setup a CQ's IMS entry
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + * @virt_cq_id: virtual CQ ID.
> + * @is_ldb: CQ is load-balanced.
> + * @addr_lo: least-significant 32 bits of address.
> + * @data: 32 data bits.
> + *
> + * This sets up the CQ's IMS entry with the provided address and data values.
> + * This function should only be called if the device is configured for Scalable
> + * IOV virtualization. The upper 32 address bits are fixed in hardware and thus
> + * not needed.
> + */
> +void dlb2_hw_setup_cq_ims_entry(struct dlb2_hw *hw,
> +                               unsigned int vdev_id,
> +                               u32 virt_cq_id,
> +                               bool is_ldb,
> +                               u32 addr_lo,
> +                               u32 data);
> +
> +/**
> + * dlb2_hw_clear_cq_ims_entry() - clear a CQ's IMS entry
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + * @virt_cq_id: virtual CQ ID.
> + * @is_ldb: CQ is load-balanced.
> + *
> + * This clears the CQ's IMS entry, reverting it to its reset state.
> + */
> +void dlb2_hw_clear_cq_ims_entry(struct dlb2_hw *hw,
> +                               unsigned int vdev_id,
> +                               u32 virt_cq_id,
> +                               bool is_ldb);
> +
> +/**
> + * dlb2_hw_register_pasid() - register a vdev's PASID
> + * @hw: dlb2_hw handle for a particular device.
> + * @vdev_id: vdev ID.
> + * @pasid: the vdev's PASID.
> + *
> + * This function stores the user-supplied PASID, and uses it when configuring
> + * the vdev's CQs.
> + *
> + * Return:
> + * Returns >= 0 upon success, -1 otherwise.
> + */
> +int dlb2_hw_register_pasid(struct dlb2_hw *hw,
> +                          unsigned int vdev_id,
> +                          unsigned int pasid);
> +
> +/**
> + * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
> + *     progress.
> + * @hw: dlb2_hw handle for a particular device.
> + * @domain_id: domain ID.
> + * @args: number of unmaps in progress args
> + * @resp: response structure.
> + * @vf_request: indicates whether this request came from a VF.
> + * @vf_id: If vf_request is true, this contains the VF's ID.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
> + * assigned a detailed error code from enum dlb2_error. If successful, resp->id
> + * contains the number of unmaps in progress.
> + *
> + * Errors:
> + * EINVAL - Invalid port ID.
> + */
> +int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
> +                               u32 domain_id,
> +                               struct dlb2_pending_port_unmaps_args *args,
> +                               struct dlb2_cmd_response *resp,
> +                               bool vf_request,
> +                               unsigned int vf_id);
> +
> +/**
> + * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
> + *     to a port class-of-service.
> + * @hw: dlb2_hw handle for a particular device.
> + * @cos_id: class-of-service ID.
> + *
> + * Return:
> + * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
> + */
> +int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
> +
> +/**
> + * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
> + *     port class-of-service.
> + * @hw: dlb2_hw handle for a particular device.
> + * @cos_id: class-of-service ID.
> + * @bandwidth: class-of-service bandwidth.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
> + *         cause the total bandwidth across all classes of service to exceed
> + *         100%.
> + */
> +int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
> +
> +enum dlb2_wd_tmo {
> +       /* 40s watchdog timeout */
> +       DLB2_WD_TMO_40S,
> +       /* 10s watchdog timeout */
> +       DLB2_WD_TMO_10S,
> +       /* 1s watchdog timeout */
> +       DLB2_WD_TMO_1S,
> +
> +       /* Must be last */
> +       NUM_DLB2_WD_TMOS,
> +};
> +
> +/**
> + * dlb2_hw_enable_wd_timer() - enable the CQ watchdog timers with a
> + *     caller-specified timeout.
> + * @hw: dlb2_hw handle for a particular device.
> + * @tmo: watchdog timeout.
> + *
> + * This function should be called during device initialization and after reset.
> + * The watchdog timer interrupt must also be enabled per-CQ, using either
> + * dlb2_hw_enable_dir_cq_wd_int() or dlb2_hw_enable_ldb_cq_wd_int().
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - Invalid timeout.
> + */
> +int dlb2_hw_enable_wd_timer(struct dlb2_hw *hw, enum dlb2_wd_tmo tmo);
> +
> +/**
> + * dlb2_hw_enable_dir_cq_wd_int() - enable the CQ watchdog interrupt on an
> + *     individual CQ.
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: port ID.
> + * @vdev_req: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - Invalid directed port ID.
> + */
> +int dlb2_hw_enable_dir_cq_wd_int(struct dlb2_hw *hw,
> +                                u32 id,
> +                                bool vdev_req,
> +                                unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_enable_ldb_cq_wd_int() - enable the CQ watchdog interrupt on an
> + *     individual CQ.
> + * @hw: dlb2_hw handle for a particular device.
> + * @id: port ID.
> + * @vdev_req: indicates whether this request came from a vdev.
> + * @vdev_id: If vdev_request is true, this contains the vdev's ID.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - Invalid load-balanced port ID.
> + */
> +int dlb2_hw_enable_ldb_cq_wd_int(struct dlb2_hw *hw,
> +                                u32 id,
> +                                bool vdev_req,
> +                                unsigned int vdev_id);
> +
> +/**
> + * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
> + *     ports.
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function must be called prior to configuring scheduling domains.
> + */
> +void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
> + * @hw: dlb2_hw handle for a particular device.
> + *
> + * This function must be called prior to configuring scheduling domains.
> + */
> +void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
> +
> +/**
> + * dlb2_hw_set_qe_arbiter_weights() - program QE arbiter weights
> + * @hw: dlb2_hw handle for a particular device.
> + * @weight: 8-entry array of arbiter weights.
> + *
> + * weight[N] programs priority N's weight. In cases where the 8 priorities are
> + * reduced to 4 bins, the mapping is:
> + * - weight[1] programs bin 0
> + * - weight[3] programs bin 1
> + * - weight[5] programs bin 2
> + * - weight[7] programs bin 3
> + */
> +void dlb2_hw_set_qe_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
> +
> +/**
> + * dlb2_hw_set_qid_arbiter_weights() - program QID arbiter weights
> + * @hw: dlb2_hw handle for a particular device.
> + * @weight: 8-entry array of arbiter weights.
> + *
> + * weight[N] programs priority N's weight. In cases where the 8 priorities are
> + * reduced to 4 bins, the mapping is:
> + * - weight[1] programs bin 0
> + * - weight[3] programs bin 1
> + * - weight[5] programs bin 2
> + * - weight[7] programs bin 3
> + */
> +void dlb2_hw_set_qid_arbiter_weights(struct dlb2_hw *hw, u8 weight[8]);
> +
> +/**
> + * dlb2_hw_ldb_cq_interrupt_enabled() - Check if the interrupt is enabled
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: physical load-balanced port ID.
> + *
> + * This function returns whether the load-balanced CQ interrupt is enabled.
> + */
> +int dlb2_hw_ldb_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
> +
> +/**
> + * dlb2_hw_ldb_cq_interrupt_set_mode() - Program the CQ interrupt mode
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: physical load-balanced port ID.
> + * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
> + *
> + * This function can be used to disable (MODE_DIS) and re-enable the
> + * load-balanced CQ's interrupt. It should only be called after the interrupt
> + * has been configured with dlb2_configure_ldb_cq_interrupt().
> + */
> +void dlb2_hw_ldb_cq_interrupt_set_mode(struct dlb2_hw *hw,
> +                                      int port_id,
> +                                      int mode);
> +
> +/**
> + * dlb2_hw_dir_cq_interrupt_enabled() - Check if the interrupt is enabled
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: physical load-balanced port ID.
> + *
> + * This function returns whether the load-balanced CQ interrupt is enabled.
> + */
> +int dlb2_hw_dir_cq_interrupt_enabled(struct dlb2_hw *hw, int port_id);
> +
> +/**
> + * dlb2_hw_dir_cq_interrupt_set_mode() - Program the CQ interrupt mode
> + * @hw: dlb2_hw handle for a particular device.
> + * @port_id: physical directed port ID.
> + * @mode: interrupt type (DLB2_CQ_ISR_MODE_{DIS, MSI, MSIX, ADI})
> + *
> + * This function can be used to disable (MODE_DIS) and re-enable the
> + * directed CQ's interrupt. It should only be called after the interrupt
> + * has been configured with dlb2_configure_dir_cq_interrupt().
> + */
> +void dlb2_hw_dir_cq_interrupt_set_mode(struct dlb2_hw *hw,
> +                                      int port_id,
> +                                      int mode);
> +
> +#endif /* __DLB2_RESOURCE_H */
> diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
> new file mode 100644
> index 0000000..1c275ff
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/dlb2_main.c
> @@ -0,0 +1,620 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <errno.h>
> +#include <assert.h>
> +#include <unistd.h>
> +#include <string.h>
> +
> +#include <rte_malloc.h>
> +#include <rte_errno.h>
> +
> +#include "base/dlb2_resource.h"
> +#include "base/dlb2_osdep.h"
> +#include "base/dlb2_regs.h"
> +#include "dlb2_main.h"
> +#include "../dlb2_user.h"
> +#include "../dlb2_priv.h"
> +#include "../dlb2_iface.h"
> +#include "../dlb2_inline_fns.h"
> +
> +#define PF_ID_ZERO 0   /* PF ONLY! */
> +#define NO_OWNER_VF 0  /* PF ONLY! */
> +#define NOT_VF_REQ false /* PF ONLY! */
> +
> +#define DLB2_PCI_CFG_SPACE_SIZE 256
> +#define DLB2_PCI_CAP_POINTER 0x34
> +#define DLB2_PCI_CAP_NEXT(hdr) (((hdr) >> 8) & 0xFC)
> +#define DLB2_PCI_CAP_ID(hdr) ((hdr) & 0xFF)
> +#define DLB2_PCI_EXT_CAP_NEXT(hdr) (((hdr) >> 20) & 0xFFC)
> +#define DLB2_PCI_EXT_CAP_ID(hdr) ((hdr) & 0xFFFF)
> +#define DLB2_PCI_EXT_CAP_ID_ERR 1
> +#define DLB2_PCI_ERR_UNCOR_MASK 8
> +#define DLB2_PCI_ERR_UNC_UNSUP  0x00100000
> +
> +#define DLB2_PCI_EXP_DEVCTL 8
> +#define DLB2_PCI_LNKCTL 16
> +#define DLB2_PCI_SLTCTL 24
> +#define DLB2_PCI_RTCTL 28
> +#define DLB2_PCI_EXP_DEVCTL2 40
> +#define DLB2_PCI_LNKCTL2 48
> +#define DLB2_PCI_SLTCTL2 56
> +#define DLB2_PCI_CMD 4
> +#define DLB2_PCI_X_CMD 2
> +#define DLB2_PCI_EXP_DEVSTA 10
> +#define DLB2_PCI_EXP_DEVSTA_TRPND 0x20
> +#define DLB2_PCI_EXP_DEVCTL_BCR_FLR 0x8000
> +
> +#define DLB2_PCI_CAP_ID_EXP       0x10
> +#define DLB2_PCI_CAP_ID_MSIX      0x11
> +#define DLB2_PCI_EXT_CAP_ID_PAS   0x1B
> +#define DLB2_PCI_EXT_CAP_ID_PRI   0x13
> +#define DLB2_PCI_EXT_CAP_ID_ACS   0xD
> +
> +#define DLB2_PCI_PRI_CTRL_ENABLE         0x1
> +#define DLB2_PCI_PRI_ALLOC_REQ           0xC
> +#define DLB2_PCI_PRI_CTRL                0x4
> +#define DLB2_PCI_MSIX_FLAGS              0x2
> +#define DLB2_PCI_MSIX_FLAGS_ENABLE       0x8000
> +#define DLB2_PCI_MSIX_FLAGS_MASKALL      0x4000
> +#define DLB2_PCI_ERR_ROOT_STATUS         0x30
> +#define DLB2_PCI_ERR_COR_STATUS          0x10
> +#define DLB2_PCI_ERR_UNCOR_STATUS        0x4
> +#define DLB2_PCI_COMMAND_INTX_DISABLE    0x400
> +#define DLB2_PCI_ACS_CAP                 0x4
> +#define DLB2_PCI_ACS_CTRL                0x6
> +#define DLB2_PCI_ACS_SV                  0x1
> +#define DLB2_PCI_ACS_RR                  0x4
> +#define DLB2_PCI_ACS_CR                  0x8
> +#define DLB2_PCI_ACS_UF                  0x10
> +#define DLB2_PCI_ACS_EC                  0x20
> +
> +static int
> +dlb2_pci_find_ext_capability(struct rte_pci_device *pdev, uint32_t id)
> +{
> +       uint32_t hdr;
> +       size_t sz;
> +       int pos;
> +
> +       pos = DLB2_PCI_CFG_SPACE_SIZE;
> +       sz = sizeof(hdr);
> +
> +       while (pos > 0xFF) {
> +               if (rte_pci_read_config(pdev, &hdr, sz, pos) != (int)sz)
> +                       return -1;
> +
> +               if (DLB2_PCI_EXT_CAP_ID(hdr) == id)
> +                       return pos;
> +
> +               pos = DLB2_PCI_EXT_CAP_NEXT(hdr);
> +       }
> +
> +       return -1;
> +}
> +
> +static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
> +{
> +       uint8_t pos;
> +       int ret;
> +       uint16_t hdr;
> +
> +       ret = rte_pci_read_config(pdev, &pos, 1, DLB2_PCI_CAP_POINTER);
> +       pos &= 0xFC;
> +
> +       if (ret != 1)
> +               return -1;
> +
> +       while (pos > 0x3F) {
> +               ret = rte_pci_read_config(pdev, &hdr, 2, pos);
> +               if (ret != 2)
> +                       return -1;
> +
> +               if (DLB2_PCI_CAP_ID(hdr) == id)
> +                       return pos;
> +
> +               if (DLB2_PCI_CAP_ID(hdr) == 0xFF)
> +                       return -1;
> +
> +               pos = DLB2_PCI_CAP_NEXT(hdr);
> +       }
> +
> +       return -1;
> +}
> +
> +static int
> +dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
> +{
> +       int i;
> +
> +       if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIR64B))
> +               dlb2_dev->enqueue_four = dlb2_movdir64b;
> +       else
> +               dlb2_dev->enqueue_four = dlb2_movntdq;
> +
> +       /* Initialize software state */
> +       for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++)
> +               dlb2_list_init_head(&dlb2_dev->ldb_port_pages[i].list);
> +
> +       for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++)
> +               dlb2_list_init_head(&dlb2_dev->dir_port_pages[i].list);
> +
> +       rte_spinlock_init(&dlb2_dev->resource_mutex);
> +       rte_spinlock_init(&dlb2_dev->measurement_lock);
> +
> +       return 0;
> +}
> +
> +static void dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
> +{
> +       dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
> +}
> +
> +#define DLB2_READY_RETRY_LIMIT 1000
> +static int dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev)
> +{
> +       u32 retries = 0;
> +
> +       /* Allow at least 1s for the device to become active after power-on */
> +       for (retries = 0; retries < DLB2_READY_RETRY_LIMIT; retries++) {
> +               union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
> +               union dlb2_cfg_mstr_cfg_pm_status pm_st;
> +               u32 addr;
> +
> +               addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
> +               pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
> +               addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
> +               idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
> +               if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
> +                       break;
> +
> +               rte_delay_ms(1);
> +       };
> +
> +       if (retries == DLB2_READY_RETRY_LIMIT) {
> +               printf("[%s()] wait for device ready timed out\n",
> +                      __func__);
> +               return -1;
> +       }
> +
> +       return 0;
> +}
> +
> +struct dlb2_dev *
> +dlb2_probe(struct rte_pci_device *pdev)
> +{
> +       struct dlb2_dev *dlb2_dev;
> +       int ret = 0;
> +
> +       DLB2_INFO(dlb2_dev, "probe\n");
> +
> +       dlb2_dev = rte_malloc("DLB2_PF", sizeof(struct dlb2_dev),
> +                             RTE_CACHE_LINE_SIZE);
> +
> +       if (!dlb2_dev) {
> +               ret = -ENOMEM;
> +               goto dlb2_dev_malloc_fail;
> +       }
> +
> +       /* PCI Bus driver has already mapped bar space into process.
> +        * Save off our IO register and FUNC addresses.
> +        */
> +
> +       /* BAR 0 */
> +       if (pdev->mem_resource[0].addr == NULL) {
> +               DLB2_ERR(dlb2_dev, "probe: BAR 0 addr (csr_kva) is NULL\n");
> +               ret = -EINVAL;
> +               goto pci_mmap_bad_addr;
> +       }
> +       dlb2_dev->hw.func_kva = (void *)(uintptr_t)pdev->mem_resource[0].addr;
> +       dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0].phys_addr;
> +
> +       DLB2_INFO(dlb2_dev, "DLB2 FUNC VA=%p, PA=%p, len=%p\n",
> +                 (void *)dlb2_dev->hw.func_kva,
> +                 (void *)dlb2_dev->hw.func_phys_addr,
> +                 (void *)(pdev->mem_resource[0].len));
> +
> +       /* BAR 2 */
> +       if (pdev->mem_resource[2].addr == NULL) {
> +               DLB2_ERR(dlb2_dev, "probe: BAR 2 addr (func_kva) is NULL\n");
> +               ret = -EINVAL;
> +               goto pci_mmap_bad_addr;
> +       }
> +       dlb2_dev->hw.csr_kva = (void *)(uintptr_t)pdev->mem_resource[2].addr;
> +       dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2].phys_addr;
> +
> +       DLB2_INFO(dlb2_dev, "DLB2 CSR VA=%p, PA=%p, len=%p\n",
> +                 (void *)dlb2_dev->hw.csr_kva,
> +                 (void *)dlb2_dev->hw.csr_phys_addr,
> +                 (void *)(pdev->mem_resource[2].len));
> +
> +       dlb2_dev->pdev = pdev;
> +
> +       /* PM enable must be done before any other MMIO accesses, and this
> +        * setting is persistent across device reset.
> +        */
> +       dlb2_pf_enable_pm(dlb2_dev);
> +
> +       ret = dlb2_pf_wait_for_device_ready(dlb2_dev);
> +       if (ret)
> +               goto wait_for_device_ready_fail;
> +
> +       ret = dlb2_pf_reset(dlb2_dev);
> +       if (ret)
> +               goto dlb2_reset_fail;
> +
> +       ret = dlb2_pf_init_driver_state(dlb2_dev);
> +       if (ret)
> +               goto init_driver_state_fail;
> +
> +       ret = dlb2_resource_init(&dlb2_dev->hw);
> +       if (ret)
> +               goto resource_init_fail;
> +
> +       return dlb2_dev;
> +
> +resource_init_fail:
> +       dlb2_resource_free(&dlb2_dev->hw);
> +init_driver_state_fail:
> +dlb2_reset_fail:
> +pci_mmap_bad_addr:
> +wait_for_device_ready_fail:
> +       rte_free(dlb2_dev);
> +dlb2_dev_malloc_fail:
> +       rte_errno = ret;
> +       return NULL;
> +}
> +
> +int
> +dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
> +{
> +       int ret = 0;
> +       int i = 0;
> +       uint32_t dword[16];
> +       uint16_t cmd;
> +       off_t off;
> +
> +       uint16_t dev_ctl_word;
> +       uint16_t dev_ctl2_word;
> +       uint16_t lnk_word;
> +       uint16_t lnk_word2;
> +       uint16_t slt_word;
> +       uint16_t slt_word2;
> +       uint16_t rt_ctl_word;
> +       uint32_t pri_reqs_dword;
> +       uint16_t pri_ctrl_word;
> +
> +       int pcie_cap_offset;
> +       int pri_cap_offset;
> +       int msix_cap_offset;
> +       int err_cap_offset;
> +       int acs_cap_offset;
> +       int wait_count;
> +
> +       uint16_t devsta_busy_word;
> +       uint16_t devctl_word;
> +
> +       struct rte_pci_device *pdev = dlb2_dev->pdev;
> +
> +       /* Save PCI config state */
> +
> +       for (i = 0; i < 16; i++) {
> +               if (rte_pci_read_config(pdev, &dword[i], 4, i * 4) != 4)
> +                       return ret;
> +       }
> +
> +       pcie_cap_offset = dlb2_pci_find_capability(pdev, DLB2_PCI_CAP_ID_EXP);
> +
> +       if (pcie_cap_offset < 0) {
> +               printf("[%s()] failed to find the pcie capability\n",
> +                      __func__);
> +               return pcie_cap_offset;
> +       }
> +
> +       off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> +       if (rte_pci_read_config(pdev, &dev_ctl_word, 2, off) != 2)
> +               dev_ctl_word = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_LNKCTL;
> +       if (rte_pci_read_config(pdev, &lnk_word, 2, off) != 2)
> +               lnk_word = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_SLTCTL;
> +       if (rte_pci_read_config(pdev, &slt_word, 2, off) != 2)
> +               slt_word = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_RTCTL;
> +       if (rte_pci_read_config(pdev, &rt_ctl_word, 2, off) != 2)
> +               rt_ctl_word = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
> +       if (rte_pci_read_config(pdev, &dev_ctl2_word, 2, off) != 2)
> +               dev_ctl2_word = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
> +       if (rte_pci_read_config(pdev, &lnk_word2, 2, off) != 2)
> +               lnk_word2 = 0;
> +
> +       off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
> +       if (rte_pci_read_config(pdev, &slt_word2, 2, off) != 2)
> +               slt_word2 = 0;
> +
> +       off = DLB2_PCI_EXT_CAP_ID_PRI;
> +       pri_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> +
> +       if (pri_cap_offset >= 0) {
> +               off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
> +               if (rte_pci_read_config(pdev, &pri_reqs_dword, 4, off) != 4)
> +                       pri_reqs_dword = 0;
> +       }
> +
> +       /* clear the PCI command register before issuing the FLR */
> +
> +       off = DLB2_PCI_CMD;
> +       cmd = 0;
> +       if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +               printf("[%s()] failed to write the pci command\n",
> +                      __func__);
> +               return ret;
> +       }
> +
> +       /* issue the FLR */
> +       for (wait_count = 0; wait_count < 4; wait_count++) {
> +               int sleep_time;
> +
> +               off = pcie_cap_offset + DLB2_PCI_EXP_DEVSTA;
> +               ret = rte_pci_read_config(pdev, &devsta_busy_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to read the pci device status\n",
> +                              __func__);
> +                       return ret;
> +               }
> +
> +               if (!(devsta_busy_word & DLB2_PCI_EXP_DEVSTA_TRPND))
> +                       break;
> +
> +               sleep_time = (1 << (wait_count)) * 100;
> +               rte_delay_ms(sleep_time);
> +       }
> +
> +       if (wait_count == 4) {
> +               printf("[%s()] wait for pci pending transactions timed out\n",
> +                      __func__);
> +               return -1;
> +       }
> +
> +       off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> +       ret = rte_pci_read_config(pdev, &devctl_word, 2, off);
> +       if (ret != 2) {
> +               printf("[%s()] failed to read the pcie device control\n",
> +                      __func__);
> +               return ret;
> +       }
> +
> +       devctl_word |= DLB2_PCI_EXP_DEVCTL_BCR_FLR;
> +
> +       ret = rte_pci_write_config(pdev, &devctl_word, 2, off);
> +       if (ret != 2) {
> +               printf("[%s()] failed to write the pcie device control\n",
> +                      __func__);
> +               return ret;
> +       }
> +
> +       rte_delay_ms(100);
> +
> +       /* Restore PCI config state */
> +
> +       if (pcie_cap_offset >= 0) {
> +               off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL;
> +               ret = rte_pci_write_config(pdev, &dev_ctl_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie device control at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_LNKCTL;
> +               ret = rte_pci_write_config(pdev, &lnk_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_SLTCTL;
> +               ret = rte_pci_write_config(pdev, &slt_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_RTCTL;
> +               ret = rte_pci_write_config(pdev, &rt_ctl_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_EXP_DEVCTL2;
> +               ret = rte_pci_write_config(pdev, &dev_ctl2_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_LNKCTL2;
> +               ret = rte_pci_write_config(pdev, &lnk_word2, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pcie_cap_offset + DLB2_PCI_SLTCTL2;
> +               ret = rte_pci_write_config(pdev, &slt_word2, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +       }
> +
> +       if (pri_cap_offset >= 0) {
> +               pri_ctrl_word = DLB2_PCI_PRI_CTRL_ENABLE;
> +
> +               off = pri_cap_offset + DLB2_PCI_PRI_ALLOC_REQ;
> +               ret = rte_pci_write_config(pdev, &pri_reqs_dword, 4, off);
> +               if (ret != 4) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = pri_cap_offset + DLB2_PCI_PRI_CTRL;
> +               ret = rte_pci_write_config(pdev, &pri_ctrl_word, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +       }
> +
> +       off = DLB2_PCI_EXT_CAP_ID_ERR;
> +       err_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> +
> +       if (err_cap_offset >= 0) {
> +               uint32_t tmp;
> +
> +               off = err_cap_offset + DLB2_PCI_ERR_ROOT_STATUS;
> +               if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> +                       tmp = 0;
> +
> +               ret = rte_pci_write_config(pdev, &tmp, 4, off);
> +               if (ret != 4) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = err_cap_offset + DLB2_PCI_ERR_COR_STATUS;
> +               if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> +                       tmp = 0;
> +
> +               ret = rte_pci_write_config(pdev, &tmp, 4, off);
> +               if (ret != 4) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = err_cap_offset + DLB2_PCI_ERR_UNCOR_STATUS;
> +               if (rte_pci_read_config(pdev, &tmp, 4, off) != 4)
> +                       tmp = 0;
> +
> +               ret = rte_pci_write_config(pdev, &tmp, 4, off);
> +               if (ret != 4) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +       }
> +
> +       for (i = 16; i > 0; i--) {
> +               off = (i - 1) * 4;
> +               ret = rte_pci_write_config(pdev, &dword[i - 1], 4, off);
> +               if (ret != 4) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +       }
> +
> +       off = DLB2_PCI_CMD;
> +       if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> +               cmd &= ~DLB2_PCI_COMMAND_INTX_DISABLE;
> +               if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +                       printf("[%s()] failed to write the pci command\n",
> +                              __func__);
> +                       return ret;
> +               }
> +       }
> +
> +       msix_cap_offset = dlb2_pci_find_capability(pdev,
> +                                                  DLB2_PCI_CAP_ID_MSIX);
> +       if (msix_cap_offset >= 0) {
> +               off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
> +               if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> +                       cmd |= DLB2_PCI_MSIX_FLAGS_ENABLE;
> +                       cmd |= DLB2_PCI_MSIX_FLAGS_MASKALL;
> +                       if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +                               printf("[%s()] failed to write msix flags\n",
> +                                      __func__);
> +                               return ret;
> +                       }
> +               }
> +
> +               off = msix_cap_offset + DLB2_PCI_MSIX_FLAGS;
> +               if (rte_pci_read_config(pdev, &cmd, 2, off) == 2) {
> +                       cmd &= ~DLB2_PCI_MSIX_FLAGS_MASKALL;
> +                       if (rte_pci_write_config(pdev, &cmd, 2, off) != 2) {
> +                               printf("[%s()] failed to write msix flags\n",
> +                                      __func__);
> +                               return ret;
> +                       }
> +               }
> +       }
> +
> +       off = DLB2_PCI_EXT_CAP_ID_ACS;
> +       acs_cap_offset = dlb2_pci_find_ext_capability(pdev, off);
> +
> +       if (acs_cap_offset >= 0) {
> +               uint16_t acs_cap, acs_ctrl, acs_mask;
> +               off = acs_cap_offset + DLB2_PCI_ACS_CAP;
> +               if (rte_pci_read_config(pdev, &acs_cap, 2, off) != 2)
> +                       acs_cap = 0;
> +
> +               off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> +               if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
> +                       acs_ctrl = 0;
> +
> +               acs_mask = DLB2_PCI_ACS_SV | DLB2_PCI_ACS_RR;
> +               acs_mask |= (DLB2_PCI_ACS_CR | DLB2_PCI_ACS_UF);
> +               acs_ctrl |= (acs_cap & acs_mask);
> +
> +               ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +
> +               off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> +               if (rte_pci_read_config(pdev, &acs_ctrl, 2, off) != 2)
> +                       acs_ctrl = 0;
> +
> +               acs_mask = DLB2_PCI_ACS_RR | DLB2_PCI_ACS_CR;
> +               acs_mask |= DLB2_PCI_ACS_EC;
> +               acs_ctrl &= ~acs_mask;
> +
> +               off = acs_cap_offset + DLB2_PCI_ACS_CTRL;
> +               ret = rte_pci_write_config(pdev, &acs_ctrl, 2, off);
> +               if (ret != 2) {
> +                       printf("[%s()] failed to write the pcie config space at offset %d\n",
> +                               __func__, (int)off);
> +                       return ret;
> +               }
> +       }
> +
> +       return 0;
> +}
> +
> +/**********************************/
> +/****** Device configuration ******/
> +/**********************************/
> +
> diff --git a/drivers/event/dlb2/pf/dlb2_main.h b/drivers/event/dlb2/pf/dlb2_main.h
> new file mode 100644
> index 0000000..a914077
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/dlb2_main.h
> @@ -0,0 +1,107 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_MAIN_H
> +#define __DLB2_MAIN_H
> +
> +#include <rte_debug.h>
> +#include <rte_log.h>
> +#include <rte_spinlock.h>
> +#include <rte_pci.h>
> +#include <rte_bus_pci.h>
> +
> +#ifndef PAGE_SIZE
> +#define PAGE_SIZE (sysconf(_SC_PAGESIZE))
> +#endif
> +
> +#include "base/dlb2_hw_types.h"
> +#include "../dlb2_user.h"
> +
> +#define DLB2_DEFAULT_UNREGISTER_TIMEOUT_S 5
> +
> +struct dlb2_dev;
> +
> +struct dlb2_port_memory {
> +       struct dlb2_list_head list;
> +       void *cq_base;
> +       bool valid;
> +};
> +
> +struct dlb2_dev {
> +       struct rte_pci_device *pdev;
> +       struct dlb2_hw hw;
> +       /* struct list_head list; */
> +       struct device *dlb2_device;
> +       struct dlb2_port_memory ldb_port_pages[DLB2_MAX_NUM_LDB_PORTS];
> +       struct dlb2_port_memory dir_port_pages[DLB2_MAX_NUM_DIR_PORTS];
> +       /* The enqueue_four function enqueues four HCWs (one cache-line worth)
> +        * to the DLB2, using whichever mechanism is supported by the platform
> +        * on which this driver is running.
> +        */
> +       void (*enqueue_four)(void *qe4, void *pp_addr);
> +
> +       bool domain_reset_failed;
> +       /* The resource mutex serializes access to driver data structures and
> +        * hardware registers.
> +        */
> +       rte_spinlock_t resource_mutex;
> +       rte_spinlock_t measurement_lock;
> +       bool worker_launched;
> +       u8 revision;
> +};
> +
> +struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
> +
> +/* The following functions were pf_ops in kernel driver implementation */
> +int dlb2_pf_reset(struct dlb2_dev *dlb2_dev);
> +int dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
> +                               struct dlb2_create_sched_domain_args *args,
> +                               struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_create_ldb_queue_args *args,
> +                            struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_create_dir_queue_args *args,
> +                            struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_create_ldb_port_args *args,
> +                           uintptr_t cq_dma_base,
> +                           struct dlb2_cmd_response *resp);
> +int dlb2_pf_create_dir_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_create_dir_port_args *args,
> +                           uintptr_t cq_dma_base,
> +                           struct dlb2_cmd_response *resp);
> +int dlb2_pf_start_domain(struct dlb2_hw *hw,
> +                        u32 domain_id,
> +                        struct dlb2_start_domain_args *args,
> +                        struct dlb2_cmd_response *resp);
> +int dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_enable_ldb_port_args *args,
> +                           struct dlb2_cmd_response *resp);
> +int dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_disable_ldb_port_args *args,
> +                            struct dlb2_cmd_response *resp);
> +int dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
> +                           u32 domain_id,
> +                           struct dlb2_enable_dir_port_args *args,
> +                           struct dlb2_cmd_response *resp);
> +int dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
> +                            u32 domain_id,
> +                            struct dlb2_disable_dir_port_args *args,
> +                            struct dlb2_cmd_response *resp);
> +int dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 domain_id);
> +int dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
> +                                    u32 domain_id,
> +                                    u32 port_id);
> +int dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
> +                                    u32 domain_id,
> +                                    u32 port_id);
> +
> +#endif /* __DLB2_MAIN_H */
> diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
> new file mode 100644
> index 0000000..8c5ec20
> --- /dev/null
> +++ b/drivers/event/dlb2/pf/dlb2_pf.c
> @@ -0,0 +1,251 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <sys/mman.h>
> +#include <sys/fcntl.h>
> +#include <sys/time.h>
> +#include <errno.h>
> +#include <assert.h>
> +#include <unistd.h>
> +#include <string.h>
> +#include <rte_debug.h>
> +#include <rte_log.h>
> +#include <rte_dev.h>
> +#include <rte_devargs.h>
> +#include <rte_mbuf.h>
> +#include <rte_ring.h>
> +#include <rte_errno.h>
> +#include <rte_kvargs.h>
> +#include <rte_malloc.h>
> +#include <rte_cycles.h>
> +#include <rte_io.h>
> +#include <rte_pci.h>
> +#include <rte_bus_pci.h>
> +#include <rte_eventdev.h>
> +#include <rte_eventdev_pmd.h>
> +#include <rte_eventdev_pmd_pci.h>
> +#include <rte_memory.h>
> +#include <rte_string_fns.h>
> +
> +#include "../dlb2_priv.h"
> +#include "../dlb2_iface.h"
> +#include "../dlb2_inline_fns.h"
> +#include "dlb2_main.h"
> +#include "base/dlb2_hw_types.h"
> +#include "base/dlb2_osdep.h"
> +#include "base/dlb2_resource.h"
> +
> +extern struct dlb2_dev *dlb2_probe(struct rte_pci_device *pdev);
> +
> +#if !defined RTE_ARCH_X86_64
> +#error "This implementation only supports RTE_ARCH_X86_64 architecture."
> +#endif
> +
> +static const char *event_dlb2_pf_name = RTE_STR(EVDEV_DLB2_NAME_PMD);
> +
> +static void
> +dlb2_pf_low_level_io_init(void)
> +{
> +       int i;
> +       /* Addresses will be initialized at port create */
> +       for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
> +               /* First directed ports */
> +               dlb2_port[i][DLB2_DIR_PORT].pp_addr = NULL;
> +               dlb2_port[i][DLB2_DIR_PORT].cq_base = NULL;
> +               dlb2_port[i][DLB2_DIR_PORT].mmaped = true;
> +
> +               /* Now load balanced ports */
> +               dlb2_port[i][DLB2_LDB_PORT].pp_addr = NULL;
> +               dlb2_port[i][DLB2_LDB_PORT].cq_base = NULL;
> +               dlb2_port[i][DLB2_LDB_PORT].mmaped = true;
> +       }
> +}
> +
> +static int
> +dlb2_pf_open(struct dlb2_hw_dev *handle, const char *name)
> +{
> +       RTE_SET_USED(handle);
> +       RTE_SET_USED(name);
> +
> +       return 0;
> +}
> +
> +static int
> +dlb2_pf_get_device_version(struct dlb2_hw_dev *handle,
> +                          uint8_t *revision)
> +{
> +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> +
> +       *revision = dlb2_dev->revision;
> +
> +       return 0;
> +}
> +
> +static void
> +dlb2_pf_hardware_init(struct dlb2_hw_dev *handle)
> +{
> +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> +
> +       dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
> +       dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
> +}
> +
> +static int
> +dlb2_pf_get_num_resources(struct dlb2_hw_dev *handle,
> +                         struct dlb2_get_num_resources_args *rsrcs)
> +{
> +       struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> +
> +       return dlb2_hw_get_num_resources(&dlb2_dev->hw, rsrcs, false, 0);
> +}
> +
> +static int
> +dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
> +                        enum dlb2_cq_poll_modes *mode)
> +{
> +       RTE_SET_USED(handle);
> +
> +       *mode = DLB2_CQ_POLL_MODE_SPARSE;
> +
> +       return 0;
> +}
> +
> +static void
> +dlb2_pf_iface_fn_ptrs_init(void)
> +{
> +
> +       dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
> +       dlb2_iface_open = dlb2_pf_open;
> +       dlb2_iface_get_device_version = dlb2_pf_get_device_version;
> +       dlb2_iface_hardware_init = dlb2_pf_hardware_init;
> +       dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
> +       dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
> +}
> +
> +/* PCI DEV HOOKS */
> +static int
> +dlb2_eventdev_pci_init(struct rte_eventdev *eventdev)
> +{
> +       int ret = 0;
> +       struct rte_pci_device *pci_dev;
> +       struct dlb2_devargs dlb2_args = {
> +               .socket_id = rte_socket_id(),
> +               .max_num_events = DLB2_MAX_NUM_LDB_CREDITS,
> +               .num_dir_credits_override = -1,
> +               .qid_depth_thresholds = { {0} },
> +               .cos_id = DLB2_COS_DEFAULT
> +       };
> +       struct dlb2_eventdev *dlb2;
> +
> +       DLB2_LOG_DBG("Enter with dev_id=%d socket_id=%d",
> +                    eventdev->data->dev_id, eventdev->data->socket_id);
> +
> +       dlb2_pf_iface_fn_ptrs_init();
> +
> +       pci_dev = RTE_DEV_TO_PCI(eventdev->dev);
> +
> +       if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> +               dlb2 = dlb2_pmd_priv(eventdev); /* rte_zmalloc_socket mem */
> +
> +               /* Probe the DLB2 PF layer */
> +               dlb2->qm_instance.pf_dev = dlb2_probe(pci_dev);
> +
> +               if (dlb2->qm_instance.pf_dev == NULL) {
> +                       DLB2_LOG_ERR("DLB2 PF Probe failed with error %d\n",
> +                                    rte_errno);
> +                       ret = -rte_errno;
> +                       goto dlb2_probe_failed;
> +               }
> +
> +               /* Were we invoked with runtime parameters? */
> +               if (pci_dev->device.devargs) {
> +                       ret = dlb2_parse_params(pci_dev->device.devargs->args,
> +                                               pci_dev->device.devargs->name,
> +                                               &dlb2_args);
> +                       if (ret) {
> +                               DLB2_LOG_ERR("PFPMD failed to parse args ret=%d, errno=%d\n",
> +                                            ret, rte_errno);
> +                               goto dlb2_probe_failed;
> +                       }
> +               }
> +
> +               ret = dlb2_primary_eventdev_probe(eventdev,
> +                                                 event_dlb2_pf_name,
> +                                                 &dlb2_args);
> +       } else {
> +               ret = dlb2_secondary_eventdev_probe(eventdev,
> +                                                   event_dlb2_pf_name);
> +       }
> +       if (ret)
> +               goto dlb2_probe_failed;
> +
> +       DLB2_LOG_INFO("DLB2 PF Probe success\n");
> +
> +       return 0;
> +
> +dlb2_probe_failed:
> +
> +       DLB2_LOG_INFO("DLB2 PF Probe failed, ret=%d\n", ret);
> +
> +       return ret;
> +}
> +
> +#define EVENTDEV_INTEL_VENDOR_ID 0x8086
> +
> +static const struct rte_pci_id pci_id_dlb2_map[] = {
> +       {
> +               RTE_PCI_DEVICE(EVENTDEV_INTEL_VENDOR_ID,
> +                              PCI_DEVICE_ID_INTEL_DLB2_PF)
> +       },
> +       {
> +               .vendor_id = 0,
> +       },
> +};
> +
> +static int
> +event_dlb2_pci_probe(struct rte_pci_driver *pci_drv,
> +                    struct rte_pci_device *pci_dev)
> +{
> +       int ret;
> +
> +       ret = rte_event_pmd_pci_probe_named(pci_drv, pci_dev,
> +                                            sizeof(struct dlb2_eventdev),
> +                                            dlb2_eventdev_pci_init,
> +                                            event_dlb2_pf_name);
> +       if (ret) {
> +               DLB2_LOG_INFO("rte_event_pmd_pci_probe_named() failed, "
> +                               "ret=%d\n", ret);
> +       }
> +
> +       return ret;
> +}
> +
> +static int
> +event_dlb2_pci_remove(struct rte_pci_device *pci_dev)
> +{
> +       int ret;
> +
> +       ret = rte_event_pmd_pci_remove(pci_dev, NULL);
> +
> +       if (ret) {
> +               DLB2_LOG_INFO("rte_event_pmd_pci_remove() failed, "
> +                               "ret=%d\n", ret);
> +       }
> +
> +       return ret;
> +
> +}
> +
> +static struct rte_pci_driver pci_eventdev_dlb2_pmd = {
> +       .id_table = pci_id_dlb2_map,
> +       .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
> +       .probe = event_dlb2_pci_probe,
> +       .remove = event_dlb2_pci_remove,
> +};
> +
> +RTE_PMD_REGISTER_PCI(event_dlb2_pf, pci_eventdev_dlb2_pmd);
> +RTE_PMD_REGISTER_PCI_TABLE(event_dlb2_pf, pci_id_dlb2_map);
> --
> 2.6.4
>

Re: [dpdk-dev] [PATCH 06/22] event/dlb2: add probe

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Sunday, October 18, 2020 4:06 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; dpdk-dev <dev@dpdk.org>;
> Carrillo, Erik G <Erik.G.Carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> Jerin Jacob <jerinj@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH 06/22] event/dlb2: add probe
> 
> On Sat, Sep 12, 2020 at 2:01 AM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > The DLB2 hardware is a PCI device. This commit adds
> > support for probe and other initialization. The
> > dlb2_iface.[ch] files implement a flexible interface
> > that supports both the PF PMD and the bifurcated PMD.
> > The bifurcated PMD will be released in a future
> > patch set. Note that the flexible interface is only
> > used for configuration, and is not used in the data
> > path. The shared code is added in pf/base.
> >
> > Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> 
> > +static void
> > +dlb2_qm_mmio_fn_init(void)
> > +{
> > +       /* Process-local function pointers for performing low level port i/o */
> > +
> > +       if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIR64B))
> > +               qm_mmio_fns.pp_enqueue_four = dlb2_movdir64b;
> > +       else
> > +               qm_mmio_fns.pp_enqueue_four = dlb2_movntdq;
> > +}
> > +
> > +#define RTE_BASE_10 10
> 
> DONT create RTE_ symbols in drivers as it will create conflict if
> someone adds such symbols in the future in EAL.
> Either make it an internal symbol or push an EAL patch.
> 
> Also please split this patch as at minimum as two as it is a huge patch.
> 
> 

Working on this now.

[dpdk-dev] [PATCH 07/22] event/dlb2: add xstats

[Timothy McDaniel]

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c        |   35 +-
 drivers/event/dlb2/dlb2_xstats.c | 1269 ++++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build   |    1 +
 3 files changed, 1302 insertions(+), 3 deletions(-)
 create mode 100644 drivers/event/dlb2/dlb2_xstats.c

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7ff7dac..0d6fea4 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -77,6 +77,21 @@ static struct dlb2_port_low_level_io_functions qm_mmio_fns;
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
+/*
+ * DUMMY - added so that xstats path will compile/link.
+ * Will be replaced by real version in a subsequent
+ * patch.
+ */
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	RTE_SET_USED(dlb2);
+	RTE_SET_USED(queue);
+
+	return 0;
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -353,9 +368,16 @@ set_qid_depth_thresh(const char *key __rte_unused,
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
-	RTE_SET_USED(dev);
-
-	/* Eventdev PMD entry points */
+	/* Expose PMD's eventdev interface */
+	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dump             = dlb2_eventdev_dump,
+		.xstats_get       = dlb2_eventdev_xstats_get,
+		.xstats_get_names = dlb2_eventdev_xstats_get_names,
+		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
+		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+	};
+
+	dev->dev_ops = &dlb2_eventdev_entry_ops;
 }
 
 int
@@ -411,6 +433,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev *dev,
 		return err;
 	}
 
+	/* Complete xtstats runtime initialization */
+	err = dlb2_xstats_init(dlb2);
+	if (err) {
+		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
+		return err;
+	}
+
 	/* Initialize each port's token pop mode */
 	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
 		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP;
diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
new file mode 100644
index 0000000..9a69d78
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_xstats.c
@@ -0,0 +1,1269 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/mman.h>
+#include <sys/fcntl.h>
+#include <errno.h>
+#include <assert.h>
+#include <unistd.h>
+#include <string.h>
+
+#include <rte_debug.h>
+#include <rte_log.h>
+#include <rte_dev.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+#include <rte_malloc.h>
+#include <rte_cycles.h>
+#include <rte_io.h>
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+#include <rte_eventdev_pmd_vdev.h>
+
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+enum dlb2_xstats_type {
+	/* common to device and port */
+	rx_ok,				/**< Receive an event */
+	rx_drop,                        /**< Error bit set in received QE */
+	rx_interrupt_wait,		/**< Wait on an interrupt */
+	rx_umonitor_umwait,		/**< Block using umwait */
+	tx_ok,				/**< Transmit an event */
+	total_polls,			/**< Call dequeue_burst */
+	zero_polls,			/**< Call dequeue burst and return 0 */
+	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
+	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
+	tx_nospc_inflight_max,		/**< Reach the new_event_threshold */
+	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
+	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
+	/* device specific */
+	nb_events_limit,
+	inflight_events,
+	ldb_pool_size,
+	dir_pool_size,
+	/* port specific */
+	tx_new,				/**< Send an OP_NEW event */
+	tx_fwd,				/**< Send an OP_FORWARD event */
+	tx_rel,				/**< Send an OP_RELEASE event */
+	tx_implicit_rel,		/**< Issue an implicit event release */
+	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED event */
+	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL event */
+	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC event */
+	tx_sched_directed,		/**< Send a directed event */
+	tx_invalid,                     /**< Send an event with an invalid op */
+	outstanding_releases,		/**< # of releases a port owes */
+	max_outstanding_releases,	/**< max # of releases a port can owe */
+	rx_sched_ordered,		/**< Dequeue an ordered event */
+	rx_sched_unordered,		/**< Dequeue an unordered event */
+	rx_sched_atomic,		/**< Dequeue an atomic event */
+	rx_sched_directed,		/**< Dequeue an directed event */
+	rx_sched_invalid,               /**< Dequeue event sched type invalid */
+	/* common to port and queue */
+	is_configured,			/**< Port is configured */
+	is_load_balanced,		/**< Port is LDB */
+	hw_id,				/**< Hardware ID */
+	/* queue specific */
+	num_links,			/**< Number of ports linked */
+	sched_type,			/**< Queue sched type */
+	enq_ok,				/**< # events enqueued to the queue */
+	current_depth,			/**< Current queue depth */
+	depth_threshold,		/**< Programmed depth threshold */
+	depth_le50_threshold,
+	/**< Depth LE to 50% of the configured hardware threshold */
+	depth_gt50_le75_threshold,
+	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold */
+	depth_gt75_le100_threshold,
+	/**< Depth GT 75%. but LE to the configured hardware threshold */
+	depth_gt100_threshold
+	/**< Depth GT 100% of the configured hw threshold */
+};
+
+typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
+		uint16_t obj_idx, /* port or queue id */
+		enum dlb2_xstats_type stat, int extra_arg);
+
+enum dlb2_xstats_fn_type {
+	DLB2_XSTATS_FN_DEV,
+	DLB2_XSTATS_FN_PORT,
+	DLB2_XSTATS_FN_QUEUE
+};
+
+struct dlb2_xstats_entry {
+	struct rte_event_dev_xstats_name name;
+	uint64_t reset_value; /* an offset to be taken away to emulate resets */
+	enum dlb2_xstats_fn_type fn_id;
+	enum dlb2_xstats_type stat;
+	enum rte_event_dev_xstats_mode mode;
+	int extra_arg;
+	uint16_t obj_idx;
+	uint8_t reset_allowed; /* when set, this value can be reset */
+};
+
+/* Some device stats are simply a summation of the corresponding port values */
+static uint64_t
+dlb2_device_traffic_stat_get(struct dlb2_eventdev *dlb2,
+			     int which_stat)
+{
+	int i;
+	uint64_t val = 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
+
+		if (!port->setup_done)
+			continue;
+
+		switch (which_stat) {
+		case rx_ok:
+			val += port->stats.traffic.rx_ok;
+			break;
+		case rx_drop:
+			val += port->stats.traffic.rx_drop;
+			break;
+		case rx_interrupt_wait:
+			val += port->stats.traffic.rx_interrupt_wait;
+			break;
+		case rx_umonitor_umwait:
+			val += port->stats.traffic.rx_umonitor_umwait;
+			break;
+		case tx_ok:
+			val += port->stats.traffic.tx_ok;
+			break;
+		case total_polls:
+			val += port->stats.traffic.total_polls;
+			break;
+		case zero_polls:
+			val += port->stats.traffic.zero_polls;
+			break;
+		case tx_nospc_ldb_hw_credits:
+			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
+			break;
+		case tx_nospc_dir_hw_credits:
+			val += port->stats.traffic.tx_nospc_dir_hw_credits;
+			break;
+		case tx_nospc_inflight_max:
+			val += port->stats.traffic.tx_nospc_inflight_max;
+			break;
+		case tx_nospc_new_event_limit:
+			val += port->stats.traffic.tx_nospc_new_event_limit;
+			break;
+		case tx_nospc_inflight_credits:
+			val += port->stats.traffic.tx_nospc_inflight_credits;
+			break;
+		default:
+			return -1;
+		}
+	}
+	return val;
+}
+
+static uint64_t
+get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
+	     enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	switch (type) {
+	case rx_ok:
+	case rx_drop:
+	case rx_interrupt_wait:
+	case rx_umonitor_umwait:
+	case tx_ok:
+	case total_polls:
+	case zero_polls:
+	case tx_nospc_ldb_hw_credits:
+	case tx_nospc_dir_hw_credits:
+	case tx_nospc_inflight_max:
+	case tx_nospc_new_event_limit:
+	case tx_nospc_inflight_credits:
+		return dlb2_device_traffic_stat_get(dlb2, type);
+	case nb_events_limit:
+		return dlb2->new_event_limit;
+	case inflight_events:
+		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
+	case ldb_pool_size:
+		return dlb2->num_ldb_credits;
+	case dir_pool_size:
+		return dlb2->num_dir_credits;
+	default: return -1;
+	}
+}
+
+static uint64_t
+get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	      enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
+
+	switch (type) {
+	case rx_ok: return ev_port->stats.traffic.rx_ok;
+
+	case rx_drop: return ev_port->stats.traffic.rx_drop;
+
+	case rx_interrupt_wait: return ev_port->stats.traffic.rx_interrupt_wait;
+
+	case rx_umonitor_umwait:
+		return ev_port->stats.traffic.rx_umonitor_umwait;
+
+	case tx_ok: return ev_port->stats.traffic.tx_ok;
+
+	case total_polls: return ev_port->stats.traffic.total_polls;
+
+	case zero_polls: return ev_port->stats.traffic.zero_polls;
+
+	case tx_nospc_ldb_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
+
+	case tx_nospc_dir_hw_credits:
+		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
+
+	case tx_nospc_inflight_max:
+		return ev_port->stats.traffic.tx_nospc_inflight_max;
+
+	case tx_nospc_new_event_limit:
+		return ev_port->stats.traffic.tx_nospc_new_event_limit;
+
+	case tx_nospc_inflight_credits:
+		return ev_port->stats.traffic.tx_nospc_inflight_credits;
+
+	case is_configured: return ev_port->setup_done;
+
+	case is_load_balanced: return !ev_port->qm_port.is_directed;
+
+	case hw_id: return ev_port->qm_port.id;
+
+	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
+
+	case tx_fwd: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
+
+	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
+
+	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
+
+	case tx_sched_ordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case tx_sched_unordered:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case tx_sched_atomic:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case tx_sched_directed:
+		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case tx_invalid: return ev_port->stats.tx_invalid;
+
+	case outstanding_releases: return ev_port->outstanding_releases;
+
+	case max_outstanding_releases:
+		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	case rx_sched_ordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
+
+	case rx_sched_unordered:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
+
+	case rx_sched_atomic:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
+
+	case rx_sched_directed:
+		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
+
+	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
+
+	default: return -1;
+	}
+}
+
+static uint64_t
+dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
+{
+	int port = 0;
+	uint64_t tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
+
+	return tally;
+}
+
+static uint64_t
+dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid)
+{
+	int port = 0;
+	uint64_t enq_ok_tally = 0;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
+		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
+
+	return enq_ok_tally;
+}
+
+static uint64_t
+get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
+	       enum dlb2_xstats_type type, int extra_arg __rte_unused)
+{
+	struct dlb2_eventdev_queue *ev_queue =
+		&dlb2->ev_queues[obj_idx];
+
+	switch (type) {
+	case is_configured: return ev_queue->setup_done;
+
+	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
+
+	case hw_id: return ev_queue->qm_queue.id;
+
+	case num_links: return ev_queue->num_links;
+
+	case sched_type: return ev_queue->qm_queue.sched_type;
+
+	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
+
+	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
+
+	case depth_threshold: return ev_queue->depth_threshold;
+
+	case depth_le50_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_LE50);
+
+	case depth_gt50_le75_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT50_LE75);
+
+	case depth_gt75_le100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT75_LE100);
+
+	case depth_gt100_threshold:
+		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
+					       DLB2_QID_DEPTH_GT100);
+
+	default: return -1;
+	}
+}
+
+int
+dlb2_xstats_init(struct dlb2_eventdev *dlb2)
+{
+	/*
+	 * define the stats names and types. Used to build up the device
+	 * xstats array
+	 * There are multiple set of stats:
+	 *   - device-level,
+	 *   - per-port,
+	 *   - per-qid,
+	 *
+	 * For each of these sets, we have three parallel arrays, one for the
+	 * names, the other for the stat type parameter to be passed in the fn
+	 * call to get that stat. The third array allows resetting or not.
+	 * All these arrays must be kept in sync
+	 */
+	static const char * const dev_stats[] = {
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"nb_events_limit",
+		"inflight_events",
+		"ldb_pool_size",
+		"dir_pool_size",
+	};
+	static const enum dlb2_xstats_type dev_types[] = {
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		nb_events_limit,
+		inflight_events,
+		ldb_pool_size,
+		dir_pool_size,
+	};
+	/* Note: generated device stats are not allowed to be reset. */
+	static const uint8_t dev_reset_allowed[] = {
+		0, /* rx_ok */
+		0, /* rx_drop */
+		0, /* rx_interrupt_wait */
+		0, /* rx_umonitor_umwait */
+		0, /* tx_ok */
+		0, /* total_polls */
+		0, /* zero_polls */
+		0, /* tx_nospc_ldb_hw_credits */
+		0, /* tx_nospc_dir_hw_credits */
+		0, /* tx_nospc_inflight_max */
+		0, /* tx_nospc_new_event_limit */
+		0, /* tx_nospc_inflight_credits */
+		0, /* nb_events_limit */
+		0, /* inflight_events */
+		0, /* ldb_pool_size */
+		0, /* dir_pool_size */
+	};
+	static const char * const port_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"rx_ok",
+		"rx_drop",
+		"rx_interrupt_wait",
+		"rx_umonitor_umwait",
+		"tx_ok",
+		"total_polls",
+		"zero_polls",
+		"tx_nospc_ldb_hw_credits",
+		"tx_nospc_dir_hw_credits",
+		"tx_nospc_inflight_max",
+		"tx_nospc_new_event_limit",
+		"tx_nospc_inflight_credits",
+		"tx_new",
+		"tx_fwd",
+		"tx_rel",
+		"tx_implicit_rel",
+		"tx_sched_ordered",
+		"tx_sched_unordered",
+		"tx_sched_atomic",
+		"tx_sched_directed",
+		"tx_invalid",
+		"outstanding_releases",
+		"max_outstanding_releases",
+		"rx_sched_ordered",
+		"rx_sched_unordered",
+		"rx_sched_atomic",
+		"rx_sched_directed",
+		"rx_sched_invalid"
+	};
+	static const enum dlb2_xstats_type port_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		rx_ok,
+		rx_drop,
+		rx_interrupt_wait,
+		rx_umonitor_umwait,
+		tx_ok,
+		total_polls,
+		zero_polls,
+		tx_nospc_ldb_hw_credits,
+		tx_nospc_dir_hw_credits,
+		tx_nospc_inflight_max,
+		tx_nospc_new_event_limit,
+		tx_nospc_inflight_credits,
+		tx_new,
+		tx_fwd,
+		tx_rel,
+		tx_implicit_rel,
+		tx_sched_ordered,
+		tx_sched_unordered,
+		tx_sched_atomic,
+		tx_sched_directed,
+		tx_invalid,
+		outstanding_releases,
+		max_outstanding_releases,
+		rx_sched_ordered,
+		rx_sched_unordered,
+		rx_sched_atomic,
+		rx_sched_directed,
+		rx_sched_invalid
+	};
+	static const uint8_t port_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		1, /* rx_ok */
+		1, /* rx_drop */
+		1, /* rx_interrupt_wait */
+		1, /* rx_umonitor_umwait */
+		1, /* tx_ok */
+		1, /* total_polls */
+		1, /* zero_polls */
+		1, /* tx_nospc_ldb_hw_credits */
+		1, /* tx_nospc_dir_hw_credits */
+		1, /* tx_nospc_inflight_max */
+		1, /* tx_nospc_new_event_limit */
+		1, /* tx_nospc_inflight_credits */
+		1, /* tx_new */
+		1, /* tx_fwd */
+		1, /* tx_rel */
+		1, /* tx_implicit_rel */
+		1, /* tx_sched_ordered */
+		1, /* tx_sched_unordered */
+		1, /* tx_sched_atomic */
+		1, /* tx_sched_directed */
+		1, /* tx_invalid */
+		0, /* outstanding_releases */
+		0, /* max_outstanding_releases */
+		1, /* rx_sched_ordered */
+		1, /* rx_sched_unordered */
+		1, /* rx_sched_atomic */
+		1, /* rx_sched_directed */
+		1  /* rx_sched_invalid */
+	};
+
+	/* QID specific stats */
+	static const char * const qid_stats[] = {
+		"is_configured",
+		"is_load_balanced",
+		"hw_id",
+		"num_links",
+		"sched_type",
+		"enq_ok",
+		"current_depth",
+		"depth_threshold",
+		"depth_le50_threshold",
+		"depth_gt50_le75_threshold",
+		"depth_gt75_le100_threshold",
+		"depth_gt100_threshold",
+	};
+	static const enum dlb2_xstats_type qid_types[] = {
+		is_configured,
+		is_load_balanced,
+		hw_id,
+		num_links,
+		sched_type,
+		enq_ok,
+		current_depth,
+		depth_threshold,
+		depth_le50_threshold,
+		depth_gt50_le75_threshold,
+		depth_gt75_le100_threshold,
+		depth_gt100_threshold,
+	};
+	static const uint8_t qid_reset_allowed[] = {
+		0, /* is_configured */
+		0, /* is_load_balanced */
+		0, /* hw_id */
+		0, /* num_links */
+		0, /* sched_type */
+		1, /* enq_ok */
+		0, /* current_depth */
+		0, /* depth_threshold */
+		1, /* depth_le50_threshold */
+		1, /* depth_gt50_le75_threshold */
+		1, /* depth_gt75_le100_threshold */
+		1, /* depth_gt100_threshold */
+	};
+
+	/* ---- end of stat definitions ---- */
+
+	/* check sizes, since a missed comma can lead to strings being
+	 * joined by the compiler.
+	 */
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
+
+	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_reset_allowed));
+	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_reset_allowed));
+
+	/* other vars */
+	const unsigned int count = RTE_DIM(dev_stats) +
+			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
+			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
+	unsigned int i, port, qid, stat_id = 0;
+
+	dlb2->xstats = rte_zmalloc_socket(NULL,
+			sizeof(dlb2->xstats[0]) * count, 0,
+			dlb2->qm_instance.info.socket_id);
+	if (dlb2->xstats == NULL)
+		return -ENOMEM;
+
+#define sname dlb2->xstats[stat_id].name.name
+	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
+		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
+			.fn_id = DLB2_XSTATS_FN_DEV,
+			.stat = dev_types[i],
+			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
+			.reset_allowed = dev_reset_allowed[i],
+		};
+		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
+	}
+	dlb2->xstats_count_mode_dev = stat_id;
+
+	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
+		dlb2->xstats_offset_for_port[port] = stat_id;
+
+		uint32_t count_offset = stat_id;
+
+		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_PORT,
+				.obj_idx = port,
+				.stat = port_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_PORT,
+				.reset_allowed = port_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "port_%u_%s",
+				 port, port_stats[i]);
+		}
+
+		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_port = stat_id - dlb2->xstats_count_mode_dev;
+
+	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
+		uint32_t count_offset = stat_id;
+
+		dlb2->xstats_offset_for_qid[qid] = stat_id;
+
+		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
+			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
+				.fn_id = DLB2_XSTATS_FN_QUEUE,
+				.obj_idx = qid,
+				.stat = qid_types[i],
+				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
+				.reset_allowed = qid_reset_allowed[i],
+			};
+			snprintf(sname, sizeof(sname), "qid_%u_%s",
+				 qid, qid_stats[i]);
+		}
+
+		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
+	}
+
+	dlb2->xstats_count_mode_queue = stat_id -
+		(dlb2->xstats_count_mode_dev + dlb2->xstats_count_mode_port);
+#undef sname
+
+	dlb2->xstats_count = stat_id;
+
+	return 0;
+}
+
+void
+dlb2_xstats_uninit(struct dlb2_eventdev *dlb2)
+{
+	rte_free(dlb2->xstats);
+	dlb2->xstats_count = 0;
+}
+
+int
+dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		struct rte_event_dev_xstats_name *xstats_names,
+		unsigned int *ids, unsigned int size)
+{
+	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	unsigned int xidx = 0;
+
+	RTE_SET_USED(mode);
+	RTE_SET_USED(queue_port_id);
+
+	uint32_t xstats_mode_count = 0;
+	uint32_t start_offset = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			break;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			break;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
+		break;
+	default:
+		return -EINVAL;
+	};
+
+	if (xstats_mode_count > size || !ids || !xstats_names)
+		return xstats_mode_count;
+
+	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
+		if (dlb2->xstats[i].mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != dlb2->xstats[i].obj_idx)
+			continue;
+
+		xstats_names[xidx] = dlb2->xstats[i].name;
+		if (ids)
+			ids[xidx] = start_offset + xidx;
+		xidx++;
+	}
+	return xidx;
+}
+
+static int
+dlb2_xstats_update(struct dlb2_eventdev *dlb2,
+		enum rte_event_dev_xstats_mode mode,
+		uint8_t queue_port_id, const unsigned int ids[],
+		uint64_t values[], unsigned int n, const uint32_t reset)
+{
+	unsigned int i;
+	unsigned int xidx = 0;
+
+	RTE_SET_USED(mode);
+	RTE_SET_USED(queue_port_id);
+
+	uint32_t xstats_mode_count = 0;
+
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		xstats_mode_count = dlb2->xstats_count_mode_dev;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
+			goto invalid_value;
+		xstats_mode_count = dlb2->xstats_count_per_port[queue_port_id];
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
+		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
+			goto invalid_value;
+#endif
+		xstats_mode_count = dlb2->xstats_count_per_qid[queue_port_id];
+		break;
+	default:
+		goto invalid_value;
+	};
+
+	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
+		dlb2_xstats_fn fn;
+
+		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
+			continue;
+
+		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
+		    queue_port_id != xs->obj_idx)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			goto invalid_value;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+
+		if (values)
+			values[xidx] = val;
+
+		if (xs->reset_allowed && reset)
+			xs->reset_value += val;
+
+		xidx++;
+	}
+
+	return xidx;
+
+invalid_value:
+	return -EINVAL;
+}
+
+int
+dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
+		enum rte_event_dev_xstats_mode mode, uint8_t queue_port_id,
+		const unsigned int ids[], uint64_t values[], unsigned int n)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	const uint32_t reset = 0;
+
+	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
+				  reset);
+}
+
+uint64_t
+dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
+				 const char *name, unsigned int *id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	unsigned int i;
+	dlb2_xstats_fn fn;
+
+	for (i = 0; i < dlb2->xstats_count; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (strncmp(xs->name.name, name,
+			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
+			if (id != NULL)
+				*id = i;
+
+			switch (xs->fn_id) {
+			case DLB2_XSTATS_FN_DEV:
+				fn = get_dev_stat;
+				break;
+			case DLB2_XSTATS_FN_PORT:
+				fn = get_port_stat;
+				break;
+			case DLB2_XSTATS_FN_QUEUE:
+				fn = get_queue_stat;
+				break;
+			default:
+				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
+					  xs->fn_id);
+				return (uint64_t)-1;
+			}
+
+			return fn(dlb2, xs->obj_idx, xs->stat,
+				  xs->extra_arg) - xs->reset_value;
+		}
+	}
+	if (id != NULL)
+		*id = (uint32_t)-1;
+	return (uint64_t)-1;
+}
+
+static void
+dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
+			uint32_t num)
+{
+	uint32_t i;
+	dlb2_xstats_fn fn;
+
+	for (i = start; i < start + num; i++) {
+		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
+
+		if (!xs->reset_allowed)
+			continue;
+
+		switch (xs->fn_id) {
+		case DLB2_XSTATS_FN_DEV:
+			fn = get_dev_stat;
+			break;
+		case DLB2_XSTATS_FN_PORT:
+			fn = get_port_stat;
+			break;
+		case DLB2_XSTATS_FN_QUEUE:
+			fn = get_queue_stat;
+			break;
+		default:
+			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs->fn_id);
+			return;
+		}
+
+		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
+		xs->reset_value = val;
+	}
+}
+
+static int
+dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
+			const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_QUEUE,
+					queue_id, ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	if (ids == NULL)
+		dlb2_xstats_reset_range(dlb2,
+			dlb2->xstats_offset_for_qid[queue_id],
+			dlb2->xstats_count_per_qid[queue_id]);
+
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
+		       const uint32_t ids[], uint32_t nb_ids)
+{
+	const uint32_t reset = 1;
+	int offset = dlb2->xstats_offset_for_port[port_id];
+	int nb_stat = dlb2->xstats_count_per_port[port_id];
+
+	if (ids) {
+		uint32_t nb_reset = dlb2_xstats_update(dlb2,
+					RTE_EVENT_DEV_XSTATS_PORT, port_id,
+					ids, NULL, nb_ids,
+					reset);
+		return nb_reset == nb_ids ? 0 : -EINVAL;
+	}
+
+	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
+	return 0;
+}
+
+static int
+dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
+		      uint32_t nb_ids)
+{
+	uint32_t i;
+
+	if (ids) {
+		for (i = 0; i < nb_ids; i++) {
+			uint32_t id = ids[i];
+
+			if (id >= dlb2->xstats_count_mode_dev)
+				return -EINVAL;
+			dlb2_xstats_reset_range(dlb2, id, 1);
+		}
+	} else {
+		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
+			dlb2_xstats_reset_range(dlb2, i, 1);
+	}
+
+	return 0;
+}
+
+int
+dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
+			   enum rte_event_dev_xstats_mode mode,
+			   int16_t queue_port_id,
+			   const uint32_t ids[],
+			   uint32_t nb_ids)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	uint32_t i;
+
+	/* handle -1 for queue_port_id here, looping over all ports/queues */
+	switch (mode) {
+	case RTE_EVENT_DEV_XSTATS_DEVICE:
+		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
+			return -EINVAL;
+		break;
+	case RTE_EVENT_DEV_XSTATS_PORT:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
+				if (dlb2_xstats_reset_port(dlb2, i,
+								ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
+			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
+							ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	case RTE_EVENT_DEV_XSTATS_QUEUE:
+		if (queue_port_id == -1) {
+			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
+				if (dlb2_xstats_reset_queue(dlb2, i,
+								ids, nb_ids))
+					return -EINVAL;
+			}
+		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
+			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
+								ids, nb_ids))
+				return -EINVAL;
+		}
+		break;
+	};
+
+	return 0;
+}
+
+void
+dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	int i;
+
+	if (!f) {
+		printf("Invalid file pointer\n");
+		return;
+	}
+
+	if (!dev) {
+		fprintf(f, "Invalid event device\n");
+		return;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (!dlb2) {
+		fprintf(f, "DLB2 Event device cannot be dumped!\n");
+		return;
+	}
+
+	if (!dlb2->configured)
+		fprintf(f, "DLB2 Event device is not configured\n");
+
+	handle = &dlb2->qm_instance;
+
+	fprintf(f, "================\n");
+	fprintf(f, "DLB2 Device Dump\n");
+	fprintf(f, "================\n");
+
+	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
+		dlb2->umwait_allowed ? "yes" : "no");
+
+	/* Generic top level device information */
+
+	fprintf(f, "device is configured and run state =");
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		fprintf(f, "STOPPED\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
+		fprintf(f, "STOPPING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
+		fprintf(f, "STARTING\n");
+	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
+		fprintf(f, "STARTED\n");
+	else
+		fprintf(f, "UNEXPECTED\n");
+
+	fprintf(f,
+		"dev ID=%d, dom ID=%u, name=%s, path=%s, sock=%u, evdev=%p\n",
+		handle->device_id, handle->domain_id, handle->device_name,
+		handle->device_path, handle->info.socket_id, dlb2->event_dev);
+
+	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
+		dlb2->num_dir_ports, dlb2->num_dir_queues);
+
+	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
+		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
+
+	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
+		handle->cfg.resources.num_atomic_inflights,
+		handle->cfg.resources.num_hist_list_entries);
+
+	fprintf(f, "results from most recent hw resource query:\n");
+
+	fprintf(f, "\tnum_sched_domains = %u\n",
+		dlb2->hw_rsrc_query_results.num_sched_domains);
+
+	fprintf(f, "\tnum_ldb_queues = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_queues);
+
+	fprintf(f, "\tnum_ldb_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_ports);
+
+	fprintf(f, "\tnum_dir_ports = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_ports);
+
+	fprintf(f, "\tnum_atomic_inflights = %u\n",
+		dlb2->hw_rsrc_query_results.num_atomic_inflights);
+
+	fprintf(f, "\tnum_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.num_hist_list_entries);
+
+	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
+		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
+
+	fprintf(f, "\tnum_ldb_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_ldb_credits);
+
+	fprintf(f, "\tnum_dir_credits = %u\n",
+		dlb2->hw_rsrc_query_results.num_dir_credits);
+
+	/* Port level information */
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
+		int j;
+
+		if (!p->enq_configured)
+			fprintf(f, "Port_%d is not configured\n", i);
+
+		fprintf(f, "Port_%d\n", i);
+		fprintf(f, "=======\n");
+
+		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
+			p->id, p->setup_done);
+
+		fprintf(f, "\tconfig state=%d, port state=%d\n",
+			p->qm_port.config_state, p->qm_port.state);
+
+		fprintf(f, "\tport is %s\n",
+			p->qm_port.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\toutstanding releases=%u\n",
+			p->outstanding_releases);
+
+		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
+			p->inflight_max, p->inflight_credits);
+
+		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
+			p->credit_update_quanta, p->implicit_release);
+
+		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (p->link[j].valid)
+				fprintf(f, "id=%u prio=%u ",
+					p->link[j].queue_id,
+					p->link[j].priority);
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
+
+		fprintf(f, "\tcached_ldb_credits=%u\n",
+			p->qm_port.cached_ldb_credits);
+
+		fprintf(f, "\tldb_credits = %u\n",
+			p->qm_port.ldb_credits);
+
+		fprintf(f, "\tcached_dir_credits = %u\n",
+			p->qm_port.cached_dir_credits);
+
+		fprintf(f, "\tdir_credits = %u\n",
+			p->qm_port.dir_credits);
+
+		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
+			p->qm_port.gen_bit,
+			p->qm_port.cq_idx,
+			p->qm_port.cq_depth);
+
+		fprintf(f, "\tinterrupt armed=%d\n",
+			p->qm_port.int_armed);
+
+		fprintf(f, "\tPort statistics\n");
+
+		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
+			p->stats.traffic.rx_ok);
+
+		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
+			p->stats.traffic.rx_drop);
+
+		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
+			p->stats.traffic.rx_interrupt_wait);
+
+		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
+			p->stats.traffic.rx_umonitor_umwait);
+
+		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
+			p->stats.traffic.tx_ok);
+
+		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
+			p->stats.traffic.total_polls);
+
+		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
+			p->stats.traffic.zero_polls);
+
+		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_ldb_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_dir_hw_credits);
+
+		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_max);
+
+		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_new_event_limit);
+
+		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
+			p->stats.traffic.tx_nospc_inflight_credits);
+
+		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
+
+		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
+
+		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
+			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
+
+		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
+			p->stats.tx_implicit_rel);
+
+		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
+			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
+			p->stats.tx_invalid);
+
+		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
+
+		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
+
+		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
+
+		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
+			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
+
+		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
+			p->stats.rx_sched_invalid);
+	}
+
+	/* Queue level information */
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
+		int j, k;
+
+		if (!q->setup_done)
+			fprintf(f, "Queue_%d is not configured\n", i);
+
+		fprintf(f, "Queue_%d\n", i);
+		fprintf(f, "========\n");
+
+		fprintf(f, "\tevqueue_%u is set up\n", q->id);
+
+		fprintf(f, "\tqueue is %s\n",
+			q->qm_queue.is_directed ? "directed" : "load balanced");
+
+		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
+
+		for (j = 0; j < dlb2->num_ports; j++) {
+			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
+
+			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; k++) {
+				if (p->link[k].valid &&
+				    p->link[k].queue_id == q->id)
+					fprintf(f, "id=%u prio=%u ",
+						p->id, p->link[k].priority);
+			}
+		}
+		fprintf(f, "\n");
+
+		fprintf(f, "\tcurrent depth: %u events\n",
+			dlb2_get_queue_depth(dlb2, q));
+
+		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
+			q->qm_queue.num_qid_inflights, q->qm_queue.sched_type);
+	}
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 557e3b4..492452e 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -3,6 +3,7 @@
 
 sources = files('dlb2.c',
 		'dlb2_iface.c',
+		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c'
-- 
2.6.4

Re: [dpdk-dev] [PATCH 07/22] event/dlb2: add xstats

[Chen, Mike Ximing]

<snip>
> +dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f) {
> +	struct dlb2_eventdev *dlb2;
> +	struct dlb2_hw_dev *handle;
> +	int i;
> +
> +	if (!f) {
> +		printf("Invalid file pointer\n");
> +		return;
> +	}
> +
> +	if (!dev) {
> +		fprintf(f, "Invalid event device\n");
> +		return;
> +	}
> +
> +	dlb2 = dlb2_pmd_priv(dev);
> +
> +	if (!dlb2) {
> +		fprintf(f, "DLB2 Event device cannot be dumped!\n");
> +		return;
> +	}
> +

Not sure if this is enforced. The DPDK coding style discourages using ! on pointers ( see section 1. 8.1 at https://doc.dpdk.org/guides/contributing/coding_style.html). 

Reviewed-by: Mike Ximing Chen <mike.ximing.chen@intel.com>

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Timothy McDaniel
> Sent: Friday, September 11, 2020 4:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [dpdk-dev] [PATCH 07/22] event/dlb2: add xstats
> 
> Add support for DLB2 xstats.  Perform initialization and add standard xstats entry
> points.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  drivers/event/dlb2/dlb2.c        |   35 +-
>  drivers/event/dlb2/dlb2_xstats.c | 1269
> ++++++++++++++++++++++++++++++++++++++
>  drivers/event/dlb2/meson.build   |    1 +
>  3 files changed, 1302 insertions(+), 3 deletions(-)  create mode 100644
> drivers/event/dlb2/dlb2_xstats.c
> 
> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c index
> 7ff7dac..0d6fea4 100644
> --- a/drivers/event/dlb2/dlb2.c
> +++ b/drivers/event/dlb2/dlb2.c
> @@ -77,6 +77,21 @@ static struct dlb2_port_low_level_io_functions
> qm_mmio_fns;  struct process_local_port_data
> dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
> 
> +/*
> + * DUMMY - added so that xstats path will compile/link.
> + * Will be replaced by real version in a subsequent
> + * patch.
> + */
> +uint32_t
> +dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
> +		     struct dlb2_eventdev_queue *queue) {
> +	RTE_SET_USED(dlb2);
> +	RTE_SET_USED(queue);
> +
> +	return 0;
> +}
> +
>  /* override defaults with value(s) provided on command line */  static void
> dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2, @@ -353,9
> +368,16 @@ set_qid_depth_thresh(const char *key __rte_unused,  static void
> dlb2_entry_points_init(struct rte_eventdev *dev)  {
> -	RTE_SET_USED(dev);
> -
> -	/* Eventdev PMD entry points */
> +	/* Expose PMD's eventdev interface */
> +	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
> +		.dump             = dlb2_eventdev_dump,
> +		.xstats_get       = dlb2_eventdev_xstats_get,
> +		.xstats_get_names = dlb2_eventdev_xstats_get_names,
> +		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
> +		.xstats_reset	    = dlb2_eventdev_xstats_reset,
> +	};
> +
> +	dev->dev_ops = &dlb2_eventdev_entry_ops;
>  }
> 
>  int
> @@ -411,6 +433,13 @@ dlb2_primary_eventdev_probe(struct rte_eventdev
> *dev,
>  		return err;
>  	}
> 
> +	/* Complete xtstats runtime initialization */
> +	err = dlb2_xstats_init(dlb2);
> +	if (err) {
> +		DLB2_LOG_ERR("dlb2: failed to init xstats, err=%d\n", err);
> +		return err;
> +	}
> +
>  	/* Initialize each port's token pop mode */
>  	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++)
>  		dlb2->ev_ports[i].qm_port.token_pop_mode = AUTO_POP; diff
> --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
> new file mode 100644
> index 0000000..9a69d78
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2_xstats.c
> @@ -0,0 +1,1269 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation  */
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <sys/mman.h>
> +#include <sys/fcntl.h>
> +#include <errno.h>
> +#include <assert.h>
> +#include <unistd.h>
> +#include <string.h>
> +
> +#include <rte_debug.h>
> +#include <rte_log.h>
> +#include <rte_dev.h>
> +#include <rte_mbuf.h>
> +#include <rte_ring.h>
> +#include <rte_errno.h>
> +#include <rte_kvargs.h>
> +#include <rte_malloc.h>
> +#include <rte_cycles.h>
> +#include <rte_io.h>
> +#include <rte_eventdev.h>
> +#include <rte_eventdev_pmd.h>
> +#include <rte_eventdev_pmd_vdev.h>
> +
> +#include "dlb2_priv.h"
> +#include "dlb2_inline_fns.h"
> +
> +enum dlb2_xstats_type {
> +	/* common to device and port */
> +	rx_ok,				/**< Receive an event */
> +	rx_drop,                        /**< Error bit set in received QE */
> +	rx_interrupt_wait,		/**< Wait on an interrupt */
> +	rx_umonitor_umwait,		/**< Block using umwait */
> +	tx_ok,				/**< Transmit an event */
> +	total_polls,			/**< Call dequeue_burst */
> +	zero_polls,			/**< Call dequeue burst and return 0 */
> +	tx_nospc_ldb_hw_credits,	/**< Insufficient LDB h/w credits */
> +	tx_nospc_dir_hw_credits,	/**< Insufficient DIR h/w credits */
> +	tx_nospc_inflight_max,		/**< Reach the new_event_threshold
> */
> +	tx_nospc_new_event_limit,	/**< Insufficient s/w credits */
> +	tx_nospc_inflight_credits,	/**< Port has too few s/w credits */
> +	/* device specific */
> +	nb_events_limit,
> +	inflight_events,
> +	ldb_pool_size,
> +	dir_pool_size,
> +	/* port specific */
> +	tx_new,				/**< Send an OP_NEW event
> */
> +	tx_fwd,				/**< Send an OP_FORWARD event */
> +	tx_rel,				/**< Send an OP_RELEASE event */
> +	tx_implicit_rel,		/**< Issue an implicit event release */
> +	tx_sched_ordered,		/**< Send a SCHED_TYPE_ORDERED
> event */
> +	tx_sched_unordered,		/**< Send a SCHED_TYPE_PARALLEL
> event */
> +	tx_sched_atomic,		/**< Send a SCHED_TYPE_ATOMIC
> event */
> +	tx_sched_directed,		/**< Send a directed event */
> +	tx_invalid,                     /**< Send an event with an invalid op */
> +	outstanding_releases,		/**< # of releases a port owes */
> +	max_outstanding_releases,	/**< max # of releases a port can owe
> */
> +	rx_sched_ordered,		/**< Dequeue an ordered event */
> +	rx_sched_unordered,		/**< Dequeue an unordered event */
> +	rx_sched_atomic,		/**< Dequeue an atomic event */
> +	rx_sched_directed,		/**< Dequeue an directed event */
> +	rx_sched_invalid,               /**< Dequeue event sched type invalid */
> +	/* common to port and queue */
> +	is_configured,			/**< Port is configured */
> +	is_load_balanced,		/**< Port is LDB */
> +	hw_id,				/**< Hardware ID */
> +	/* queue specific */
> +	num_links,			/**< Number of ports linked */
> +	sched_type,			/**< Queue sched type */
> +	enq_ok,				/**< # events enqueued to the
> queue */
> +	current_depth,			/**< Current queue depth */
> +	depth_threshold,		/**< Programmed depth threshold */
> +	depth_le50_threshold,
> +	/**< Depth LE to 50% of the configured hardware threshold */
> +	depth_gt50_le75_threshold,
> +	/**< Depth GT 50%, but LE to 75% of the configured hardware threshold
> */
> +	depth_gt75_le100_threshold,
> +	/**< Depth GT 75%. but LE to the configured hardware threshold */
> +	depth_gt100_threshold
> +	/**< Depth GT 100% of the configured hw threshold */ };
> +
> +typedef uint64_t (*dlb2_xstats_fn)(struct dlb2_eventdev *dlb2,
> +		uint16_t obj_idx, /* port or queue id */
> +		enum dlb2_xstats_type stat, int extra_arg);
> +
> +enum dlb2_xstats_fn_type {
> +	DLB2_XSTATS_FN_DEV,
> +	DLB2_XSTATS_FN_PORT,
> +	DLB2_XSTATS_FN_QUEUE
> +};
> +
> +struct dlb2_xstats_entry {
> +	struct rte_event_dev_xstats_name name;
> +	uint64_t reset_value; /* an offset to be taken away to emulate resets */
> +	enum dlb2_xstats_fn_type fn_id;
> +	enum dlb2_xstats_type stat;
> +	enum rte_event_dev_xstats_mode mode;
> +	int extra_arg;
> +	uint16_t obj_idx;
> +	uint8_t reset_allowed; /* when set, this value can be reset */ };
> +
> +/* Some device stats are simply a summation of the corresponding port
> +values */ static uint64_t dlb2_device_traffic_stat_get(struct
> +dlb2_eventdev *dlb2,
> +			     int which_stat)
> +{
> +	int i;
> +	uint64_t val = 0;
> +
> +	for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
> +		struct dlb2_eventdev_port *port = &dlb2->ev_ports[i];
> +
> +		if (!port->setup_done)
> +			continue;
> +
> +		switch (which_stat) {
> +		case rx_ok:
> +			val += port->stats.traffic.rx_ok;
> +			break;
> +		case rx_drop:
> +			val += port->stats.traffic.rx_drop;
> +			break;
> +		case rx_interrupt_wait:
> +			val += port->stats.traffic.rx_interrupt_wait;
> +			break;
> +		case rx_umonitor_umwait:
> +			val += port->stats.traffic.rx_umonitor_umwait;
> +			break;
> +		case tx_ok:
> +			val += port->stats.traffic.tx_ok;
> +			break;
> +		case total_polls:
> +			val += port->stats.traffic.total_polls;
> +			break;
> +		case zero_polls:
> +			val += port->stats.traffic.zero_polls;
> +			break;
> +		case tx_nospc_ldb_hw_credits:
> +			val += port->stats.traffic.tx_nospc_ldb_hw_credits;
> +			break;
> +		case tx_nospc_dir_hw_credits:
> +			val += port->stats.traffic.tx_nospc_dir_hw_credits;
> +			break;
> +		case tx_nospc_inflight_max:
> +			val += port->stats.traffic.tx_nospc_inflight_max;
> +			break;
> +		case tx_nospc_new_event_limit:
> +			val += port->stats.traffic.tx_nospc_new_event_limit;
> +			break;
> +		case tx_nospc_inflight_credits:
> +			val += port->stats.traffic.tx_nospc_inflight_credits;
> +			break;
> +		default:
> +			return -1;
> +		}
> +	}
> +	return val;
> +}
> +
> +static uint64_t
> +get_dev_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx __rte_unused,
> +	     enum dlb2_xstats_type type, int extra_arg __rte_unused) {
> +	switch (type) {
> +	case rx_ok:
> +	case rx_drop:
> +	case rx_interrupt_wait:
> +	case rx_umonitor_umwait:
> +	case tx_ok:
> +	case total_polls:
> +	case zero_polls:
> +	case tx_nospc_ldb_hw_credits:
> +	case tx_nospc_dir_hw_credits:
> +	case tx_nospc_inflight_max:
> +	case tx_nospc_new_event_limit:
> +	case tx_nospc_inflight_credits:
> +		return dlb2_device_traffic_stat_get(dlb2, type);
> +	case nb_events_limit:
> +		return dlb2->new_event_limit;
> +	case inflight_events:
> +		return __atomic_load_n(&dlb2->inflights, __ATOMIC_SEQ_CST);
> +	case ldb_pool_size:
> +		return dlb2->num_ldb_credits;
> +	case dir_pool_size:
> +		return dlb2->num_dir_credits;
> +	default: return -1;
> +	}
> +}
> +
> +static uint64_t
> +get_port_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
> +	      enum dlb2_xstats_type type, int extra_arg __rte_unused) {
> +	struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[obj_idx];
> +
> +	switch (type) {
> +	case rx_ok: return ev_port->stats.traffic.rx_ok;
> +
> +	case rx_drop: return ev_port->stats.traffic.rx_drop;
> +
> +	case rx_interrupt_wait: return
> +ev_port->stats.traffic.rx_interrupt_wait;
> +
> +	case rx_umonitor_umwait:
> +		return ev_port->stats.traffic.rx_umonitor_umwait;
> +
> +	case tx_ok: return ev_port->stats.traffic.tx_ok;
> +
> +	case total_polls: return ev_port->stats.traffic.total_polls;
> +
> +	case zero_polls: return ev_port->stats.traffic.zero_polls;
> +
> +	case tx_nospc_ldb_hw_credits:
> +		return ev_port->stats.traffic.tx_nospc_ldb_hw_credits;
> +
> +	case tx_nospc_dir_hw_credits:
> +		return ev_port->stats.traffic.tx_nospc_dir_hw_credits;
> +
> +	case tx_nospc_inflight_max:
> +		return ev_port->stats.traffic.tx_nospc_inflight_max;
> +
> +	case tx_nospc_new_event_limit:
> +		return ev_port->stats.traffic.tx_nospc_new_event_limit;
> +
> +	case tx_nospc_inflight_credits:
> +		return ev_port->stats.traffic.tx_nospc_inflight_credits;
> +
> +	case is_configured: return ev_port->setup_done;
> +
> +	case is_load_balanced: return !ev_port->qm_port.is_directed;
> +
> +	case hw_id: return ev_port->qm_port.id;
> +
> +	case tx_new: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW];
> +
> +	case tx_fwd: return ev_port-
> >stats.tx_op_cnt[RTE_EVENT_OP_FORWARD];
> +
> +	case tx_rel: return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE];
> +
> +	case tx_implicit_rel: return ev_port->stats.tx_implicit_rel;
> +
> +	case tx_sched_ordered:
> +		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ORDERED];
> +
> +	case tx_sched_unordered:
> +		return ev_port-
> >stats.tx_sched_cnt[DLB2_SCHED_UNORDERED];
> +
> +	case tx_sched_atomic:
> +		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC];
> +
> +	case tx_sched_directed:
> +		return ev_port->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED];
> +
> +	case tx_invalid: return ev_port->stats.tx_invalid;
> +
> +	case outstanding_releases: return ev_port->outstanding_releases;
> +
> +	case max_outstanding_releases:
> +		return DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
> +
> +	case rx_sched_ordered:
> +		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ORDERED];
> +
> +	case rx_sched_unordered:
> +		return ev_port-
> >stats.rx_sched_cnt[DLB2_SCHED_UNORDERED];
> +
> +	case rx_sched_atomic:
> +		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC];
> +
> +	case rx_sched_directed:
> +		return ev_port->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED];
> +
> +	case rx_sched_invalid: return ev_port->stats.rx_sched_invalid;
> +
> +	default: return -1;
> +	}
> +}
> +
> +static uint64_t
> +dlb2_get_threshold_stat(struct dlb2_eventdev *dlb2, int qid, int stat)
> +{
> +	int port = 0;
> +	uint64_t tally = 0;
> +
> +	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
> +		tally += dlb2->ev_ports[port].stats.queue[qid].qid_depth[stat];
> +
> +	return tally;
> +}
> +
> +static uint64_t
> +dlb2_get_enq_ok_stat(struct dlb2_eventdev *dlb2, int qid) {
> +	int port = 0;
> +	uint64_t enq_ok_tally = 0;
> +
> +	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++)
> +		enq_ok_tally += dlb2->ev_ports[port].stats.queue[qid].enq_ok;
> +
> +	return enq_ok_tally;
> +}
> +
> +static uint64_t
> +get_queue_stat(struct dlb2_eventdev *dlb2, uint16_t obj_idx,
> +	       enum dlb2_xstats_type type, int extra_arg __rte_unused) {
> +	struct dlb2_eventdev_queue *ev_queue =
> +		&dlb2->ev_queues[obj_idx];
> +
> +	switch (type) {
> +	case is_configured: return ev_queue->setup_done;
> +
> +	case is_load_balanced: return !ev_queue->qm_queue.is_directed;
> +
> +	case hw_id: return ev_queue->qm_queue.id;
> +
> +	case num_links: return ev_queue->num_links;
> +
> +	case sched_type: return ev_queue->qm_queue.sched_type;
> +
> +	case enq_ok: return dlb2_get_enq_ok_stat(dlb2, obj_idx);
> +
> +	case current_depth: return dlb2_get_queue_depth(dlb2, ev_queue);
> +
> +	case depth_threshold: return ev_queue->depth_threshold;
> +
> +	case depth_le50_threshold:
> +		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
> +					       DLB2_QID_DEPTH_LE50);
> +
> +	case depth_gt50_le75_threshold:
> +		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
> +					       DLB2_QID_DEPTH_GT50_LE75);
> +
> +	case depth_gt75_le100_threshold:
> +		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
> +					       DLB2_QID_DEPTH_GT75_LE100);
> +
> +	case depth_gt100_threshold:
> +		return dlb2_get_threshold_stat(dlb2, ev_queue->id,
> +					       DLB2_QID_DEPTH_GT100);
> +
> +	default: return -1;
> +	}
> +}
> +
> +int
> +dlb2_xstats_init(struct dlb2_eventdev *dlb2) {
> +	/*
> +	 * define the stats names and types. Used to build up the device
> +	 * xstats array
> +	 * There are multiple set of stats:
> +	 *   - device-level,
> +	 *   - per-port,
> +	 *   - per-qid,
> +	 *
> +	 * For each of these sets, we have three parallel arrays, one for the
> +	 * names, the other for the stat type parameter to be passed in the fn
> +	 * call to get that stat. The third array allows resetting or not.
> +	 * All these arrays must be kept in sync
> +	 */
> +	static const char * const dev_stats[] = {
> +		"rx_ok",
> +		"rx_drop",
> +		"rx_interrupt_wait",
> +		"rx_umonitor_umwait",
> +		"tx_ok",
> +		"total_polls",
> +		"zero_polls",
> +		"tx_nospc_ldb_hw_credits",
> +		"tx_nospc_dir_hw_credits",
> +		"tx_nospc_inflight_max",
> +		"tx_nospc_new_event_limit",
> +		"tx_nospc_inflight_credits",
> +		"nb_events_limit",
> +		"inflight_events",
> +		"ldb_pool_size",
> +		"dir_pool_size",
> +	};
> +	static const enum dlb2_xstats_type dev_types[] = {
> +		rx_ok,
> +		rx_drop,
> +		rx_interrupt_wait,
> +		rx_umonitor_umwait,
> +		tx_ok,
> +		total_polls,
> +		zero_polls,
> +		tx_nospc_ldb_hw_credits,
> +		tx_nospc_dir_hw_credits,
> +		tx_nospc_inflight_max,
> +		tx_nospc_new_event_limit,
> +		tx_nospc_inflight_credits,
> +		nb_events_limit,
> +		inflight_events,
> +		ldb_pool_size,
> +		dir_pool_size,
> +	};
> +	/* Note: generated device stats are not allowed to be reset. */
> +	static const uint8_t dev_reset_allowed[] = {
> +		0, /* rx_ok */
> +		0, /* rx_drop */
> +		0, /* rx_interrupt_wait */
> +		0, /* rx_umonitor_umwait */
> +		0, /* tx_ok */
> +		0, /* total_polls */
> +		0, /* zero_polls */
> +		0, /* tx_nospc_ldb_hw_credits */
> +		0, /* tx_nospc_dir_hw_credits */
> +		0, /* tx_nospc_inflight_max */
> +		0, /* tx_nospc_new_event_limit */
> +		0, /* tx_nospc_inflight_credits */
> +		0, /* nb_events_limit */
> +		0, /* inflight_events */
> +		0, /* ldb_pool_size */
> +		0, /* dir_pool_size */
> +	};
> +	static const char * const port_stats[] = {
> +		"is_configured",
> +		"is_load_balanced",
> +		"hw_id",
> +		"rx_ok",
> +		"rx_drop",
> +		"rx_interrupt_wait",
> +		"rx_umonitor_umwait",
> +		"tx_ok",
> +		"total_polls",
> +		"zero_polls",
> +		"tx_nospc_ldb_hw_credits",
> +		"tx_nospc_dir_hw_credits",
> +		"tx_nospc_inflight_max",
> +		"tx_nospc_new_event_limit",
> +		"tx_nospc_inflight_credits",
> +		"tx_new",
> +		"tx_fwd",
> +		"tx_rel",
> +		"tx_implicit_rel",
> +		"tx_sched_ordered",
> +		"tx_sched_unordered",
> +		"tx_sched_atomic",
> +		"tx_sched_directed",
> +		"tx_invalid",
> +		"outstanding_releases",
> +		"max_outstanding_releases",
> +		"rx_sched_ordered",
> +		"rx_sched_unordered",
> +		"rx_sched_atomic",
> +		"rx_sched_directed",
> +		"rx_sched_invalid"
> +	};
> +	static const enum dlb2_xstats_type port_types[] = {
> +		is_configured,
> +		is_load_balanced,
> +		hw_id,
> +		rx_ok,
> +		rx_drop,
> +		rx_interrupt_wait,
> +		rx_umonitor_umwait,
> +		tx_ok,
> +		total_polls,
> +		zero_polls,
> +		tx_nospc_ldb_hw_credits,
> +		tx_nospc_dir_hw_credits,
> +		tx_nospc_inflight_max,
> +		tx_nospc_new_event_limit,
> +		tx_nospc_inflight_credits,
> +		tx_new,
> +		tx_fwd,
> +		tx_rel,
> +		tx_implicit_rel,
> +		tx_sched_ordered,
> +		tx_sched_unordered,
> +		tx_sched_atomic,
> +		tx_sched_directed,
> +		tx_invalid,
> +		outstanding_releases,
> +		max_outstanding_releases,
> +		rx_sched_ordered,
> +		rx_sched_unordered,
> +		rx_sched_atomic,
> +		rx_sched_directed,
> +		rx_sched_invalid
> +	};
> +	static const uint8_t port_reset_allowed[] = {
> +		0, /* is_configured */
> +		0, /* is_load_balanced */
> +		0, /* hw_id */
> +		1, /* rx_ok */
> +		1, /* rx_drop */
> +		1, /* rx_interrupt_wait */
> +		1, /* rx_umonitor_umwait */
> +		1, /* tx_ok */
> +		1, /* total_polls */
> +		1, /* zero_polls */
> +		1, /* tx_nospc_ldb_hw_credits */
> +		1, /* tx_nospc_dir_hw_credits */
> +		1, /* tx_nospc_inflight_max */
> +		1, /* tx_nospc_new_event_limit */
> +		1, /* tx_nospc_inflight_credits */
> +		1, /* tx_new */
> +		1, /* tx_fwd */
> +		1, /* tx_rel */
> +		1, /* tx_implicit_rel */
> +		1, /* tx_sched_ordered */
> +		1, /* tx_sched_unordered */
> +		1, /* tx_sched_atomic */
> +		1, /* tx_sched_directed */
> +		1, /* tx_invalid */
> +		0, /* outstanding_releases */
> +		0, /* max_outstanding_releases */
> +		1, /* rx_sched_ordered */
> +		1, /* rx_sched_unordered */
> +		1, /* rx_sched_atomic */
> +		1, /* rx_sched_directed */
> +		1  /* rx_sched_invalid */
> +	};
> +
> +	/* QID specific stats */
> +	static const char * const qid_stats[] = {
> +		"is_configured",
> +		"is_load_balanced",
> +		"hw_id",
> +		"num_links",
> +		"sched_type",
> +		"enq_ok",
> +		"current_depth",
> +		"depth_threshold",
> +		"depth_le50_threshold",
> +		"depth_gt50_le75_threshold",
> +		"depth_gt75_le100_threshold",
> +		"depth_gt100_threshold",
> +	};
> +	static const enum dlb2_xstats_type qid_types[] = {
> +		is_configured,
> +		is_load_balanced,
> +		hw_id,
> +		num_links,
> +		sched_type,
> +		enq_ok,
> +		current_depth,
> +		depth_threshold,
> +		depth_le50_threshold,
> +		depth_gt50_le75_threshold,
> +		depth_gt75_le100_threshold,
> +		depth_gt100_threshold,
> +	};
> +	static const uint8_t qid_reset_allowed[] = {
> +		0, /* is_configured */
> +		0, /* is_load_balanced */
> +		0, /* hw_id */
> +		0, /* num_links */
> +		0, /* sched_type */
> +		1, /* enq_ok */
> +		0, /* current_depth */
> +		0, /* depth_threshold */
> +		1, /* depth_le50_threshold */
> +		1, /* depth_gt50_le75_threshold */
> +		1, /* depth_gt75_le100_threshold */
> +		1, /* depth_gt100_threshold */
> +	};
> +
> +	/* ---- end of stat definitions ---- */
> +
> +	/* check sizes, since a missed comma can lead to strings being
> +	 * joined by the compiler.
> +	 */
> +	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) != RTE_DIM(dev_types));
> +	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) != RTE_DIM(port_types));
> +	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) != RTE_DIM(qid_types));
> +
> +	RTE_BUILD_BUG_ON(RTE_DIM(dev_stats) !=
> RTE_DIM(dev_reset_allowed));
> +	RTE_BUILD_BUG_ON(RTE_DIM(port_stats) !=
> RTE_DIM(port_reset_allowed));
> +	RTE_BUILD_BUG_ON(RTE_DIM(qid_stats) !=
> RTE_DIM(qid_reset_allowed));
> +
> +	/* other vars */
> +	const unsigned int count = RTE_DIM(dev_stats) +
> +			DLB2_MAX_NUM_PORTS * RTE_DIM(port_stats) +
> +			DLB2_MAX_NUM_QUEUES * RTE_DIM(qid_stats);
> +	unsigned int i, port, qid, stat_id = 0;
> +
> +	dlb2->xstats = rte_zmalloc_socket(NULL,
> +			sizeof(dlb2->xstats[0]) * count, 0,
> +			dlb2->qm_instance.info.socket_id);
> +	if (dlb2->xstats == NULL)
> +		return -ENOMEM;
> +
> +#define sname dlb2->xstats[stat_id].name.name
> +	for (i = 0; i < RTE_DIM(dev_stats); i++, stat_id++) {
> +		dlb2->xstats[stat_id] = (struct dlb2_xstats_entry) {
> +			.fn_id = DLB2_XSTATS_FN_DEV,
> +			.stat = dev_types[i],
> +			.mode = RTE_EVENT_DEV_XSTATS_DEVICE,
> +			.reset_allowed = dev_reset_allowed[i],
> +		};
> +		snprintf(sname, sizeof(sname), "dev_%s", dev_stats[i]);
> +	}
> +	dlb2->xstats_count_mode_dev = stat_id;
> +
> +	for (port = 0; port < DLB2_MAX_NUM_PORTS; port++) {
> +		dlb2->xstats_offset_for_port[port] = stat_id;
> +
> +		uint32_t count_offset = stat_id;
> +
> +		for (i = 0; i < RTE_DIM(port_stats); i++, stat_id++) {
> +			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
> +				.fn_id = DLB2_XSTATS_FN_PORT,
> +				.obj_idx = port,
> +				.stat = port_types[i],
> +				.mode = RTE_EVENT_DEV_XSTATS_PORT,
> +				.reset_allowed = port_reset_allowed[i],
> +			};
> +			snprintf(sname, sizeof(sname), "port_%u_%s",
> +				 port, port_stats[i]);
> +		}
> +
> +		dlb2->xstats_count_per_port[port] = stat_id - count_offset;
> +	}
> +
> +	dlb2->xstats_count_mode_port = stat_id - dlb2-
> >xstats_count_mode_dev;
> +
> +	for (qid = 0; qid < DLB2_MAX_NUM_QUEUES; qid++) {
> +		uint32_t count_offset = stat_id;
> +
> +		dlb2->xstats_offset_for_qid[qid] = stat_id;
> +
> +		for (i = 0; i < RTE_DIM(qid_stats); i++, stat_id++) {
> +			dlb2->xstats[stat_id] = (struct dlb2_xstats_entry){
> +				.fn_id = DLB2_XSTATS_FN_QUEUE,
> +				.obj_idx = qid,
> +				.stat = qid_types[i],
> +				.mode = RTE_EVENT_DEV_XSTATS_QUEUE,
> +				.reset_allowed = qid_reset_allowed[i],
> +			};
> +			snprintf(sname, sizeof(sname), "qid_%u_%s",
> +				 qid, qid_stats[i]);
> +		}
> +
> +		dlb2->xstats_count_per_qid[qid] = stat_id - count_offset;
> +	}
> +
> +	dlb2->xstats_count_mode_queue = stat_id -
> +		(dlb2->xstats_count_mode_dev + dlb2-
> >xstats_count_mode_port); #undef
> +sname
> +
> +	dlb2->xstats_count = stat_id;
> +
> +	return 0;
> +}
> +
> +void
> +dlb2_xstats_uninit(struct dlb2_eventdev *dlb2) {
> +	rte_free(dlb2->xstats);
> +	dlb2->xstats_count = 0;
> +}
> +
> +int
> +dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
> +		enum rte_event_dev_xstats_mode mode, uint8_t
> queue_port_id,
> +		struct rte_event_dev_xstats_name *xstats_names,
> +		unsigned int *ids, unsigned int size) {
> +	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	unsigned int i;
> +	unsigned int xidx = 0;
> +
> +	RTE_SET_USED(mode);
> +	RTE_SET_USED(queue_port_id);
> +
> +	uint32_t xstats_mode_count = 0;
> +	uint32_t start_offset = 0;
> +
> +	switch (mode) {
> +	case RTE_EVENT_DEV_XSTATS_DEVICE:
> +		xstats_mode_count = dlb2->xstats_count_mode_dev;
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_PORT:
> +		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
> +			break;
> +		xstats_mode_count = dlb2-
> >xstats_count_per_port[queue_port_id];
> +		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_QUEUE:
> +#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
> +		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
> +			break;
> +#endif
> +		xstats_mode_count = dlb2-
> >xstats_count_per_qid[queue_port_id];
> +		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
> +		break;
> +	default:
> +		return -EINVAL;
> +	};
> +
> +	if (xstats_mode_count > size || !ids || !xstats_names)
> +		return xstats_mode_count;
> +
> +	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
> +		if (dlb2->xstats[i].mode != mode)
> +			continue;
> +
> +		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
> +		    queue_port_id != dlb2->xstats[i].obj_idx)
> +			continue;
> +
> +		xstats_names[xidx] = dlb2->xstats[i].name;
> +		if (ids)
> +			ids[xidx] = start_offset + xidx;
> +		xidx++;
> +	}
> +	return xidx;
> +}
> +
> +static int
> +dlb2_xstats_update(struct dlb2_eventdev *dlb2,
> +		enum rte_event_dev_xstats_mode mode,
> +		uint8_t queue_port_id, const unsigned int ids[],
> +		uint64_t values[], unsigned int n, const uint32_t reset) {
> +	unsigned int i;
> +	unsigned int xidx = 0;
> +
> +	RTE_SET_USED(mode);
> +	RTE_SET_USED(queue_port_id);
> +
> +	uint32_t xstats_mode_count = 0;
> +
> +	switch (mode) {
> +	case RTE_EVENT_DEV_XSTATS_DEVICE:
> +		xstats_mode_count = dlb2->xstats_count_mode_dev;
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_PORT:
> +		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
> +			goto invalid_value;
> +		xstats_mode_count = dlb2-
> >xstats_count_per_port[queue_port_id];
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_QUEUE:
> +#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */
> +		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
> +			goto invalid_value;
> +#endif
> +		xstats_mode_count = dlb2-
> >xstats_count_per_qid[queue_port_id];
> +		break;
> +	default:
> +		goto invalid_value;
> +	};
> +
> +	for (i = 0; i < n && xidx < xstats_mode_count; i++) {
> +		struct dlb2_xstats_entry *xs = &dlb2->xstats[ids[i]];
> +		dlb2_xstats_fn fn;
> +
> +		if (ids[i] > dlb2->xstats_count || xs->mode != mode)
> +			continue;
> +
> +		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
> +		    queue_port_id != xs->obj_idx)
> +			continue;
> +
> +		switch (xs->fn_id) {
> +		case DLB2_XSTATS_FN_DEV:
> +			fn = get_dev_stat;
> +			break;
> +		case DLB2_XSTATS_FN_PORT:
> +			fn = get_port_stat;
> +			break;
> +		case DLB2_XSTATS_FN_QUEUE:
> +			fn = get_queue_stat;
> +			break;
> +		default:
> +			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs-
> >fn_id);
> +			goto invalid_value;
> +		}
> +
> +		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat,
> +				  xs->extra_arg) - xs->reset_value;
> +
> +		if (values)
> +			values[xidx] = val;
> +
> +		if (xs->reset_allowed && reset)
> +			xs->reset_value += val;
> +
> +		xidx++;
> +	}
> +
> +	return xidx;
> +
> +invalid_value:
> +	return -EINVAL;
> +}
> +
> +int
> +dlb2_eventdev_xstats_get(const struct rte_eventdev *dev,
> +		enum rte_event_dev_xstats_mode mode, uint8_t
> queue_port_id,
> +		const unsigned int ids[], uint64_t values[], unsigned int n) {
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	const uint32_t reset = 0;
> +
> +	return dlb2_xstats_update(dlb2, mode, queue_port_id, ids, values, n,
> +				  reset);
> +}
> +
> +uint64_t
> +dlb2_eventdev_xstats_get_by_name(const struct rte_eventdev *dev,
> +				 const char *name, unsigned int *id) {
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	unsigned int i;
> +	dlb2_xstats_fn fn;
> +
> +	for (i = 0; i < dlb2->xstats_count; i++) {
> +		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
> +
> +		if (strncmp(xs->name.name, name,
> +			    RTE_EVENT_DEV_XSTATS_NAME_SIZE) == 0){
> +			if (id != NULL)
> +				*id = i;
> +
> +			switch (xs->fn_id) {
> +			case DLB2_XSTATS_FN_DEV:
> +				fn = get_dev_stat;
> +				break;
> +			case DLB2_XSTATS_FN_PORT:
> +				fn = get_port_stat;
> +				break;
> +			case DLB2_XSTATS_FN_QUEUE:
> +				fn = get_queue_stat;
> +				break;
> +			default:
> +				DLB2_LOG_ERR("Unexpected xstat fn_id %d\n",
> +					  xs->fn_id);
> +				return (uint64_t)-1;
> +			}
> +
> +			return fn(dlb2, xs->obj_idx, xs->stat,
> +				  xs->extra_arg) - xs->reset_value;
> +		}
> +	}
> +	if (id != NULL)
> +		*id = (uint32_t)-1;
> +	return (uint64_t)-1;
> +}
> +
> +static void
> +dlb2_xstats_reset_range(struct dlb2_eventdev *dlb2, uint32_t start,
> +			uint32_t num)
> +{
> +	uint32_t i;
> +	dlb2_xstats_fn fn;
> +
> +	for (i = start; i < start + num; i++) {
> +		struct dlb2_xstats_entry *xs = &dlb2->xstats[i];
> +
> +		if (!xs->reset_allowed)
> +			continue;
> +
> +		switch (xs->fn_id) {
> +		case DLB2_XSTATS_FN_DEV:
> +			fn = get_dev_stat;
> +			break;
> +		case DLB2_XSTATS_FN_PORT:
> +			fn = get_port_stat;
> +			break;
> +		case DLB2_XSTATS_FN_QUEUE:
> +			fn = get_queue_stat;
> +			break;
> +		default:
> +			DLB2_LOG_ERR("Unexpected xstat fn_id %d\n", xs-
> >fn_id);
> +			return;
> +		}
> +
> +		uint64_t val = fn(dlb2, xs->obj_idx, xs->stat, xs->extra_arg);
> +		xs->reset_value = val;
> +	}
> +}
> +
> +static int
> +dlb2_xstats_reset_queue(struct dlb2_eventdev *dlb2, uint8_t queue_id,
> +			const uint32_t ids[], uint32_t nb_ids) {
> +	const uint32_t reset = 1;
> +
> +	if (ids) {
> +		uint32_t nb_reset = dlb2_xstats_update(dlb2,
> +					RTE_EVENT_DEV_XSTATS_QUEUE,
> +					queue_id, ids, NULL, nb_ids,
> +					reset);
> +		return nb_reset == nb_ids ? 0 : -EINVAL;
> +	}
> +
> +	if (ids == NULL)
> +		dlb2_xstats_reset_range(dlb2,
> +			dlb2->xstats_offset_for_qid[queue_id],
> +			dlb2->xstats_count_per_qid[queue_id]);
> +
> +	return 0;
> +}
> +
> +static int
> +dlb2_xstats_reset_port(struct dlb2_eventdev *dlb2, uint8_t port_id,
> +		       const uint32_t ids[], uint32_t nb_ids) {
> +	const uint32_t reset = 1;
> +	int offset = dlb2->xstats_offset_for_port[port_id];
> +	int nb_stat = dlb2->xstats_count_per_port[port_id];
> +
> +	if (ids) {
> +		uint32_t nb_reset = dlb2_xstats_update(dlb2,
> +					RTE_EVENT_DEV_XSTATS_PORT,
> port_id,
> +					ids, NULL, nb_ids,
> +					reset);
> +		return nb_reset == nb_ids ? 0 : -EINVAL;
> +	}
> +
> +	dlb2_xstats_reset_range(dlb2, offset, nb_stat);
> +	return 0;
> +}
> +
> +static int
> +dlb2_xstats_reset_dev(struct dlb2_eventdev *dlb2, const uint32_t ids[],
> +		      uint32_t nb_ids)
> +{
> +	uint32_t i;
> +
> +	if (ids) {
> +		for (i = 0; i < nb_ids; i++) {
> +			uint32_t id = ids[i];
> +
> +			if (id >= dlb2->xstats_count_mode_dev)
> +				return -EINVAL;
> +			dlb2_xstats_reset_range(dlb2, id, 1);
> +		}
> +	} else {
> +		for (i = 0; i < dlb2->xstats_count_mode_dev; i++)
> +			dlb2_xstats_reset_range(dlb2, i, 1);
> +	}
> +
> +	return 0;
> +}
> +
> +int
> +dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
> +			   enum rte_event_dev_xstats_mode mode,
> +			   int16_t queue_port_id,
> +			   const uint32_t ids[],
> +			   uint32_t nb_ids)
> +{
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	uint32_t i;
> +
> +	/* handle -1 for queue_port_id here, looping over all ports/queues */
> +	switch (mode) {
> +	case RTE_EVENT_DEV_XSTATS_DEVICE:
> +		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
> +			return -EINVAL;
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_PORT:
> +		if (queue_port_id == -1) {
> +			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
> +				if (dlb2_xstats_reset_port(dlb2, i,
> +								ids, nb_ids))
> +					return -EINVAL;
> +			}
> +		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
> +			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
> +							ids, nb_ids))
> +				return -EINVAL;
> +		}
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_QUEUE:
> +		if (queue_port_id == -1) {
> +			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
> +				if (dlb2_xstats_reset_queue(dlb2, i,
> +								ids, nb_ids))
> +					return -EINVAL;
> +			}
> +		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
> +			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
> +								ids, nb_ids))
> +				return -EINVAL;
> +		}
> +		break;
> +	};
> +
> +	return 0;
> +}
> +
> +void
> +dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f) {
> +	struct dlb2_eventdev *dlb2;
> +	struct dlb2_hw_dev *handle;
> +	int i;
> +
> +	if (!f) {
> +		printf("Invalid file pointer\n");
> +		return;
> +	}
> +
> +	if (!dev) {
> +		fprintf(f, "Invalid event device\n");
> +		return;
> +	}
> +
> +	dlb2 = dlb2_pmd_priv(dev);
> +
> +	if (!dlb2) {
> +		fprintf(f, "DLB2 Event device cannot be dumped!\n");
> +		return;
> +	}
> +
> +	if (!dlb2->configured)
> +		fprintf(f, "DLB2 Event device is not configured\n");
> +
> +	handle = &dlb2->qm_instance;
> +
> +	fprintf(f, "================\n");
> +	fprintf(f, "DLB2 Device Dump\n");
> +	fprintf(f, "================\n");
> +
> +	fprintf(f, "Processor supports umonitor/umwait instructions = %s\n",
> +		dlb2->umwait_allowed ? "yes" : "no");
> +
> +	/* Generic top level device information */
> +
> +	fprintf(f, "device is configured and run state =");
> +	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
> +		fprintf(f, "STOPPED\n");
> +	else if (dlb2->run_state == DLB2_RUN_STATE_STOPPING)
> +		fprintf(f, "STOPPING\n");
> +	else if (dlb2->run_state == DLB2_RUN_STATE_STARTING)
> +		fprintf(f, "STARTING\n");
> +	else if (dlb2->run_state == DLB2_RUN_STATE_STARTED)
> +		fprintf(f, "STARTED\n");
> +	else
> +		fprintf(f, "UNEXPECTED\n");
> +
> +	fprintf(f,
> +		"dev ID=%d, dom ID=%u, name=%s, path=%s, sock=%u,
> evdev=%p\n",
> +		handle->device_id, handle->domain_id, handle->device_name,
> +		handle->device_path, handle->info.socket_id, dlb2->event_dev);
> +
> +	fprintf(f, "num dir ports=%u, num dir queues=%u\n",
> +		dlb2->num_dir_ports, dlb2->num_dir_queues);
> +
> +	fprintf(f, "num ldb ports=%u, num ldb queues=%u\n",
> +		dlb2->num_ldb_ports, dlb2->num_ldb_queues);
> +
> +	fprintf(f, "num atomic inflights=%u, hist list entries=%u\n",
> +		handle->cfg.resources.num_atomic_inflights,
> +		handle->cfg.resources.num_hist_list_entries);
> +
> +	fprintf(f, "results from most recent hw resource query:\n");
> +
> +	fprintf(f, "\tnum_sched_domains = %u\n",
> +		dlb2->hw_rsrc_query_results.num_sched_domains);
> +
> +	fprintf(f, "\tnum_ldb_queues = %u\n",
> +		dlb2->hw_rsrc_query_results.num_ldb_queues);
> +
> +	fprintf(f, "\tnum_ldb_ports = %u\n",
> +		dlb2->hw_rsrc_query_results.num_ldb_ports);
> +
> +	fprintf(f, "\tnum_dir_ports = %u\n",
> +		dlb2->hw_rsrc_query_results.num_dir_ports);
> +
> +	fprintf(f, "\tnum_atomic_inflights = %u\n",
> +		dlb2->hw_rsrc_query_results.num_atomic_inflights);
> +
> +	fprintf(f, "\tnum_hist_list_entries = %u\n",
> +		dlb2->hw_rsrc_query_results.num_hist_list_entries);
> +
> +	fprintf(f, "\tmax_contiguous_hist_list_entries = %u\n",
> +		dlb2->hw_rsrc_query_results.max_contiguous_hist_list_entries);
> +
> +	fprintf(f, "\tnum_ldb_credits = %u\n",
> +		dlb2->hw_rsrc_query_results.num_ldb_credits);
> +
> +	fprintf(f, "\tnum_dir_credits = %u\n",
> +		dlb2->hw_rsrc_query_results.num_dir_credits);
> +
> +	/* Port level information */
> +
> +	for (i = 0; i < dlb2->num_ports; i++) {
> +		struct dlb2_eventdev_port *p = &dlb2->ev_ports[i];
> +		int j;
> +
> +		if (!p->enq_configured)
> +			fprintf(f, "Port_%d is not configured\n", i);
> +
> +		fprintf(f, "Port_%d\n", i);
> +		fprintf(f, "=======\n");
> +
> +		fprintf(f, "\tevport_%u is configured, setup done=%d\n",
> +			p->id, p->setup_done);
> +
> +		fprintf(f, "\tconfig state=%d, port state=%d\n",
> +			p->qm_port.config_state, p->qm_port.state);
> +
> +		fprintf(f, "\tport is %s\n",
> +			p->qm_port.is_directed ? "directed" : "load balanced");
> +
> +		fprintf(f, "\toutstanding releases=%u\n",
> +			p->outstanding_releases);
> +
> +		fprintf(f, "\tinflight max=%u, inflight credits=%u\n",
> +			p->inflight_max, p->inflight_credits);
> +
> +		fprintf(f, "\tcredit update quanta=%u, implicit release =%u\n",
> +			p->credit_update_quanta, p->implicit_release);
> +
> +		fprintf(f, "\tnum_links=%d, queues -> ", p->num_links);
> +
> +		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
> +			if (p->link[j].valid)
> +				fprintf(f, "id=%u prio=%u ",
> +					p->link[j].queue_id,
> +					p->link[j].priority);
> +		}
> +		fprintf(f, "\n");
> +
> +		fprintf(f, "\thardware port id=%u\n", p->qm_port.id);
> +
> +		fprintf(f, "\tcached_ldb_credits=%u\n",
> +			p->qm_port.cached_ldb_credits);
> +
> +		fprintf(f, "\tldb_credits = %u\n",
> +			p->qm_port.ldb_credits);
> +
> +		fprintf(f, "\tcached_dir_credits = %u\n",
> +			p->qm_port.cached_dir_credits);
> +
> +		fprintf(f, "\tdir_credits = %u\n",
> +			p->qm_port.dir_credits);
> +
> +		fprintf(f, "\tgenbit=%d, cq_idx=%d, cq_depth=%d\n",
> +			p->qm_port.gen_bit,
> +			p->qm_port.cq_idx,
> +			p->qm_port.cq_depth);
> +
> +		fprintf(f, "\tinterrupt armed=%d\n",
> +			p->qm_port.int_armed);
> +
> +		fprintf(f, "\tPort statistics\n");
> +
> +		fprintf(f, "\t\trx_ok %" PRIu64 "\n",
> +			p->stats.traffic.rx_ok);
> +
> +		fprintf(f, "\t\trx_drop %" PRIu64 "\n",
> +			p->stats.traffic.rx_drop);
> +
> +		fprintf(f, "\t\trx_interrupt_wait %" PRIu64 "\n",
> +			p->stats.traffic.rx_interrupt_wait);
> +
> +		fprintf(f, "\t\trx_umonitor_umwait %" PRIu64 "\n",
> +			p->stats.traffic.rx_umonitor_umwait);
> +
> +		fprintf(f, "\t\ttx_ok %" PRIu64 "\n",
> +			p->stats.traffic.tx_ok);
> +
> +		fprintf(f, "\t\ttotal_polls %" PRIu64 "\n",
> +			p->stats.traffic.total_polls);
> +
> +		fprintf(f, "\t\tzero_polls %" PRIu64 "\n",
> +			p->stats.traffic.zero_polls);
> +
> +		fprintf(f, "\t\ttx_nospc_ldb_hw_credits %" PRIu64 "\n",
> +			p->stats.traffic.tx_nospc_ldb_hw_credits);
> +
> +		fprintf(f, "\t\ttx_nospc_dir_hw_credits %" PRIu64 "\n",
> +			p->stats.traffic.tx_nospc_dir_hw_credits);
> +
> +		fprintf(f, "\t\ttx_nospc_inflight_max %" PRIu64 "\n",
> +			p->stats.traffic.tx_nospc_inflight_max);
> +
> +		fprintf(f, "\t\ttx_nospc_new_event_limit %" PRIu64 "\n",
> +			p->stats.traffic.tx_nospc_new_event_limit);
> +
> +		fprintf(f, "\t\ttx_nospc_inflight_credits %" PRIu64 "\n",
> +			p->stats.traffic.tx_nospc_inflight_credits);
> +
> +		fprintf(f, "\t\ttx_new %" PRIu64 "\n",
> +			p->stats.tx_op_cnt[RTE_EVENT_OP_NEW]);
> +
> +		fprintf(f, "\t\ttx_fwd %" PRIu64 "\n",
> +			p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD]);
> +
> +		fprintf(f, "\t\ttx_rel %" PRIu64 "\n",
> +			p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE]);
> +
> +		fprintf(f, "\t\ttx_implicit_rel %" PRIu64 "\n",
> +			p->stats.tx_implicit_rel);
> +
> +		fprintf(f, "\t\ttx_sched_ordered %" PRIu64 "\n",
> +			p->stats.tx_sched_cnt[DLB2_SCHED_ORDERED]);
> +
> +		fprintf(f, "\t\ttx_sched_unordered %" PRIu64 "\n",
> +			p->stats.tx_sched_cnt[DLB2_SCHED_UNORDERED]);
> +
> +		fprintf(f, "\t\ttx_sched_atomic %" PRIu64 "\n",
> +			p->stats.tx_sched_cnt[DLB2_SCHED_ATOMIC]);
> +
> +		fprintf(f, "\t\ttx_sched_directed %" PRIu64 "\n",
> +			p->stats.tx_sched_cnt[DLB2_SCHED_DIRECTED]);
> +
> +		fprintf(f, "\t\ttx_invalid %" PRIu64 "\n",
> +			p->stats.tx_invalid);
> +
> +		fprintf(f, "\t\trx_sched_ordered %" PRIu64 "\n",
> +			p->stats.rx_sched_cnt[DLB2_SCHED_ORDERED]);
> +
> +		fprintf(f, "\t\trx_sched_unordered %" PRIu64 "\n",
> +			p->stats.rx_sched_cnt[DLB2_SCHED_UNORDERED]);
> +
> +		fprintf(f, "\t\trx_sched_atomic %" PRIu64 "\n",
> +			p->stats.rx_sched_cnt[DLB2_SCHED_ATOMIC]);
> +
> +		fprintf(f, "\t\trx_sched_directed %" PRIu64 "\n",
> +			p->stats.rx_sched_cnt[DLB2_SCHED_DIRECTED]);
> +
> +		fprintf(f, "\t\trx_sched_invalid %" PRIu64 "\n",
> +			p->stats.rx_sched_invalid);
> +	}
> +
> +	/* Queue level information */
> +
> +	for (i = 0; i < dlb2->num_queues; i++) {
> +		struct dlb2_eventdev_queue *q = &dlb2->ev_queues[i];
> +		int j, k;
> +
> +		if (!q->setup_done)
> +			fprintf(f, "Queue_%d is not configured\n", i);
> +
> +		fprintf(f, "Queue_%d\n", i);
> +		fprintf(f, "========\n");
> +
> +		fprintf(f, "\tevqueue_%u is set up\n", q->id);
> +
> +		fprintf(f, "\tqueue is %s\n",
> +			q->qm_queue.is_directed ? "directed" : "load
> balanced");
> +
> +		fprintf(f, "\tnum_links=%d, ports -> ", q->num_links);
> +
> +		for (j = 0; j < dlb2->num_ports; j++) {
> +			struct dlb2_eventdev_port *p = &dlb2->ev_ports[j];
> +
> +			for (k = 0; k < DLB2_MAX_NUM_QIDS_PER_LDB_CQ;
> k++) {
> +				if (p->link[k].valid &&
> +				    p->link[k].queue_id == q->id)
> +					fprintf(f, "id=%u prio=%u ",
> +						p->id, p->link[k].priority);
> +			}
> +		}
> +		fprintf(f, "\n");
> +
> +		fprintf(f, "\tcurrent depth: %u events\n",
> +			dlb2_get_queue_depth(dlb2, q));
> +
> +		fprintf(f, "\tnum qid inflights=%u, sched_type=%d\n",
> +			q->qm_queue.num_qid_inflights, q-
> >qm_queue.sched_type);
> +	}
> +}
> diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
> index 557e3b4..492452e 100644
> --- a/drivers/event/dlb2/meson.build
> +++ b/drivers/event/dlb2/meson.build
> @@ -3,6 +3,7 @@
> 
>  sources = files('dlb2.c',
>  		'dlb2_iface.c',
> +		'dlb2_xstats.c',
>  		'pf/dlb2_main.c',
>  		'pf/dlb2_pf.c',
>  		'pf/base/dlb2_resource.c'
> --
> 2.6.4

Re: [dpdk-dev] [PATCH 07/22] event/dlb2: add xstats

[McDaniel, Timothy]

> -----Original Message-----
> From: Chen, Mike Ximing <mike.ximing.chen@intel.com>
> Sent: Thursday, September 17, 2020 3:58 PM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dev@dpdk.org; Carrillo, Erik G <Erik.G.Carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: RE: [dpdk-dev] [PATCH 07/22] event/dlb2: add xstats
> 
> <snip>
> > +dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f) {
> > +	struct dlb2_eventdev *dlb2;
> > +	struct dlb2_hw_dev *handle;
> > +	int i;
> > +
> > +	if (!f) {
> > +		printf("Invalid file pointer\n");
> > +		return;
> > +	}
> > +
> > +	if (!dev) {
> > +		fprintf(f, "Invalid event device\n");
> > +		return;
> > +	}
> > +
> > +	dlb2 = dlb2_pmd_priv(dev);
> > +
> > +	if (!dlb2) {
> > +		fprintf(f, "DLB2 Event device cannot be dumped!\n");
> > +		return;
> > +	}
> > +
> 
> Not sure if this is enforced. The DPDK coding style discourages using ! on
> pointers ( see section 1. 8.1 at
> https://doc.dpdk.org/guides/contributing/coding_style.html).
> 

I see !ptr used in many other dpdk components, and it is not flagged by checkpatch either.

Re: [dpdk-dev] [PATCH 07/22] event/dlb2: add xstats

[Chen, Mike Ximing]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Thursday, September 17, 2020 5:26 PM
> To: Chen, Mike Ximing <mike.ximing.chen@intel.com>
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: RE: [dpdk-dev] [PATCH 07/22] event/dlb2: add xstats
> 
> 
> 
> > -----Original Message-----
> > From: Chen, Mike Ximing <mike.ximing.chen@intel.com>
> > Sent: Thursday, September 17, 2020 3:58 PM
> > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > Cc: dev@dpdk.org; Carrillo, Erik G <Erik.G.Carrillo@intel.com>; Eads,
> > Gage <gage.eads@intel.com>; Van Haaren, Harry
> > <harry.van.haaren@intel.com>; jerinj@marvell.com
> > Subject: RE: [dpdk-dev] [PATCH 07/22] event/dlb2: add xstats
> >
> > <snip>
> > > +dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f) {
> > > +	struct dlb2_eventdev *dlb2;
> > > +	struct dlb2_hw_dev *handle;
> > > +	int i;
> > > +
> > > +	if (!f) {
> > > +		printf("Invalid file pointer\n");
> > > +		return;
> > > +	}
> > > +
> > > +	if (!dev) {
> > > +		fprintf(f, "Invalid event device\n");
> > > +		return;
> > > +	}
> > > +
> > > +	dlb2 = dlb2_pmd_priv(dev);
> > > +
> > > +	if (!dlb2) {
> > > +		fprintf(f, "DLB2 Event device cannot be dumped!\n");
> > > +		return;
> > > +	}
> > > +
> >
> > Not sure if this is enforced. The DPDK coding style discourages using
> > ! on pointers ( see section 1. 8.1 at
> > https://doc.dpdk.org/guides/contributing/coding_style.html).
> >
> 
> I see !ptr used in many other dpdk components, and it is not flagged by
> checkpatch either.
> 
Yes, I do see !ptr in dpdk functions. I guess the rule is not enforced.

Re: [dpdk-dev] [PATCH 07/22] event/dlb2: add xstats

[Bruce Richardson]

On Fri, Sep 18, 2020 at 12:37:58AM +0000, Chen, Mike Ximing wrote:
> 
> 
> > -----Original Message-----
> > From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > Sent: Thursday, September 17, 2020 5:26 PM
> > To: Chen, Mike Ximing <mike.ximing.chen@intel.com>
> > Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> > <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> > jerinj@marvell.com
> > Subject: RE: [dpdk-dev] [PATCH 07/22] event/dlb2: add xstats
> > 
> > 
> > 
> > > -----Original Message-----
> > > From: Chen, Mike Ximing <mike.ximing.chen@intel.com>
> > > Sent: Thursday, September 17, 2020 3:58 PM
> > > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > > Cc: dev@dpdk.org; Carrillo, Erik G <Erik.G.Carrillo@intel.com>; Eads,
> > > Gage <gage.eads@intel.com>; Van Haaren, Harry
> > > <harry.van.haaren@intel.com>; jerinj@marvell.com
> > > Subject: RE: [dpdk-dev] [PATCH 07/22] event/dlb2: add xstats
> > >
> > > <snip>
> > > > +dlb2_eventdev_dump(struct rte_eventdev *dev, FILE *f) {
> > > > +	struct dlb2_eventdev *dlb2;
> > > > +	struct dlb2_hw_dev *handle;
> > > > +	int i;
> > > > +
> > > > +	if (!f) {
> > > > +		printf("Invalid file pointer\n");
> > > > +		return;
> > > > +	}
> > > > +
> > > > +	if (!dev) {
> > > > +		fprintf(f, "Invalid event device\n");
> > > > +		return;
> > > > +	}
> > > > +
> > > > +	dlb2 = dlb2_pmd_priv(dev);
> > > > +
> > > > +	if (!dlb2) {
> > > > +		fprintf(f, "DLB2 Event device cannot be dumped!\n");
> > > > +		return;
> > > > +	}
> > > > +
> > >
> > > Not sure if this is enforced. The DPDK coding style discourages using
> > > ! on pointers ( see section 1. 8.1 at
> > > https://doc.dpdk.org/guides/contributing/coding_style.html).
> > >
> > 
> > I see !ptr used in many other dpdk components, and it is not flagged by
> > checkpatch either.
> > 
> Yes, I do see !ptr in dpdk functions. I guess the rule is not enforced.

Not strictly enforced, no, but for new code we should always endeavour to
follow the standards. If a new revision of this patch will be done, please
have all pointer comparisons checking for NULL explicitly in that version.

Thanks,
/Bruce

Re: [dpdk-dev] [PATCH 07/22] event/dlb2: add xstats

[Eads, Gage]

> diff --git a/drivers/event/dlb2/dlb2_xstats.c b/drivers/event/dlb2/dlb2_xstats.c
> new file mode 100644
> index 0000000..9a69d78
> --- /dev/null
> +++ b/drivers/event/dlb2/dlb2_xstats.c
> @@ -0,0 +1,1269 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2016-2020 Intel Corporation
> + */
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <sys/mman.h>
> +#include <sys/fcntl.h>
> +#include <errno.h>
> +#include <assert.h>
> +#include <unistd.h>
> +#include <string.h>

+inttypes.h for the printf format specifiers (PRIu64, etc.)

> +
> +#include <rte_debug.h>
> +#include <rte_log.h>
> +#include <rte_dev.h>
> +#include <rte_mbuf.h>
> +#include <rte_ring.h>
> +#include <rte_errno.h>
> +#include <rte_kvargs.h>
> +#include <rte_malloc.h>
> +#include <rte_cycles.h>
> +#include <rte_io.h>
> +#include <rte_eventdev.h>
> +#include <rte_eventdev_pmd.h>
> +#include <rte_eventdev_pmd_vdev.h>

Besides rte_malloc.h and rte_eventdev.h, I suspect the rest are unnecessary.

[...]

> +int
> +dlb2_eventdev_xstats_get_names(const struct rte_eventdev *dev,
> +		enum rte_event_dev_xstats_mode mode, uint8_t
> queue_port_id,
> +		struct rte_event_dev_xstats_name *xstats_names,
> +		unsigned int *ids, unsigned int size)
> +{
> +	const struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	unsigned int i;
> +	unsigned int xidx = 0;
> +
> +	RTE_SET_USED(mode);
> +	RTE_SET_USED(queue_port_id);

Not needed, these variables are used below.

> +
> +	uint32_t xstats_mode_count = 0;
> +	uint32_t start_offset = 0;
> +
> +	switch (mode) {
> +	case RTE_EVENT_DEV_XSTATS_DEVICE:
> +		xstats_mode_count = dlb2->xstats_count_mode_dev;
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_PORT:
> +		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
> +			break;
> +		xstats_mode_count = dlb2-
> >xstats_count_per_port[queue_port_id];
> +		start_offset = dlb2->xstats_offset_for_port[queue_port_id];
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_QUEUE:
> +#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */

Looks like this macro is tied to the h/w definition, is it subject to grow?

> +		if (queue_port_id >= DLB2_MAX_NUM_QUEUES)
> +			break;
> +#endif
> +		xstats_mode_count = dlb2-
> >xstats_count_per_qid[queue_port_id];
> +		start_offset = dlb2->xstats_offset_for_qid[queue_port_id];
> +		break;
> +	default:
> +		return -EINVAL;
> +	};
> +
> +	if (xstats_mode_count > size || !ids || !xstats_names)
> +		return xstats_mode_count;
> +
> +	for (i = 0; i < dlb2->xstats_count && xidx < size; i++) {
> +		if (dlb2->xstats[i].mode != mode)
> +			continue;
> +
> +		if (mode != RTE_EVENT_DEV_XSTATS_DEVICE &&
> +		    queue_port_id != dlb2->xstats[i].obj_idx)
> +			continue;
> +
> +		xstats_names[xidx] = dlb2->xstats[i].name;
> +		if (ids)
> +			ids[xidx] = start_offset + xidx;
> +		xidx++;
> +	}
> +	return xidx;
> +}
> +
> +static int
> +dlb2_xstats_update(struct dlb2_eventdev *dlb2,
> +		enum rte_event_dev_xstats_mode mode,
> +		uint8_t queue_port_id, const unsigned int ids[],
> +		uint64_t values[], unsigned int n, const uint32_t reset)
> +{
> +	unsigned int i;
> +	unsigned int xidx = 0;
> +
> +	RTE_SET_USED(mode);
> +	RTE_SET_USED(queue_port_id);
> +

Not needed, these variables are used below.

> +	uint32_t xstats_mode_count = 0;
> +
> +	switch (mode) {
> +	case RTE_EVENT_DEV_XSTATS_DEVICE:
> +		xstats_mode_count = dlb2->xstats_count_mode_dev;
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_PORT:
> +		if (queue_port_id >= DLB2_MAX_NUM_PORTS)
> +			goto invalid_value;
> +		xstats_mode_count = dlb2-
> >xstats_count_per_port[queue_port_id];
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_QUEUE:
> +#if (DLB2_MAX_NUM_QUEUES <= 255) /* max 8 bit value */

(Same comment as above)

[...]

> +int
> +dlb2_eventdev_xstats_reset(struct rte_eventdev *dev,
> +			   enum rte_event_dev_xstats_mode mode,
> +			   int16_t queue_port_id,
> +			   const uint32_t ids[],
> +			   uint32_t nb_ids)
> +{
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	uint32_t i;
> +
> +	/* handle -1 for queue_port_id here, looping over all ports/queues */
> +	switch (mode) {
> +	case RTE_EVENT_DEV_XSTATS_DEVICE:
> +		if (dlb2_xstats_reset_dev(dlb2, ids, nb_ids))
> +			return -EINVAL;
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_PORT:
> +		if (queue_port_id == -1) {
> +			for (i = 0; i < DLB2_MAX_NUM_PORTS; i++) {
> +				if (dlb2_xstats_reset_port(dlb2, i,
> +								ids, nb_ids))

Nit: argument alignment (occurs below as well)

> +					return -EINVAL;
> +			}
> +		} else if (queue_port_id < DLB2_MAX_NUM_PORTS) {
> +			if (dlb2_xstats_reset_port(dlb2, queue_port_id,
> +							ids, nb_ids))
> +				return -EINVAL;
> +		}
> +		break;
> +	case RTE_EVENT_DEV_XSTATS_QUEUE:
> +		if (queue_port_id == -1) {
> +			for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++) {
> +				if (dlb2_xstats_reset_queue(dlb2, i,
> +								ids, nb_ids))
> +					return -EINVAL;
> +			}
> +		} else if (queue_port_id < DLB2_MAX_NUM_QUEUES) {
> +			if (dlb2_xstats_reset_queue(dlb2, queue_port_id,
> +								ids, nb_ids))
> +				return -EINVAL;
> +		}
> +		break;
> +	};
> +
> +	return 0;
> +}

Thanks,
Gage

[dpdk-dev] [PATCH 08/22] event/dlb2: add infos get and configure

[Timothy McDaniel]

Add support for configuring the DLB hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  |  334 +++
 drivers/event/dlb2/dlb2_iface.c            |    7 +-
 drivers/event/dlb2/dlb2_iface.h            |    5 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 3234 ++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |   14 +
 drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
 6 files changed, 3637 insertions(+), 1 deletion(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 0d6fea4..58e953b 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -92,6 +92,28 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
 	return 0;
 }
 
+static void
+dlb2_free_qe_mem(struct dlb2_port *qm_port)
+{
+	if (qm_port == NULL)
+		return;
+
+	if (qm_port->qe4) {
+		rte_free(qm_port->qe4);
+		qm_port->qe4 = NULL;
+	}
+
+	if (qm_port->int_arm_qe) {
+		rte_free(qm_port->int_arm_qe);
+		qm_port->int_arm_qe = NULL;
+	}
+
+	if (qm_port->consume_qe) {
+		rte_free(qm_port->consume_qe);
+		qm_port->consume_qe = NULL;
+	}
+}
+
 /* override defaults with value(s) provided on command line */
 static void
 dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
@@ -366,10 +388,322 @@ set_qid_depth_thresh(const char *key __rte_unused,
 }
 
 static void
+dlb2_eventdev_info_get(struct rte_eventdev *dev,
+		       struct rte_event_dev_info *dev_info)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret;
+
+	ret = dlb2_hw_query_resources(dlb2);
+	if (ret) {
+		const struct rte_eventdev_data *data = dev->data;
+
+		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+			     ret, data->dev_id);
+		/* fn is void, so fall through and return values set up in
+		 * probe
+		 */
+	}
+
+	/* Add num resources currently owned by this domain.
+	 * These would become available if the scheduling domain were reset due
+	 * to the application recalling eventdev_configure to *reconfigure* the
+	 * domain.
+	 */
+	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
+	evdev_dlb2_default_info.max_event_queues += dlb2->num_ldb_queues;
+	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
+
+	evdev_dlb2_default_info.max_event_queues =
+		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
+			RTE_EVENT_MAX_QUEUES_PER_DEV);
+
+	evdev_dlb2_default_info.max_num_events =
+		RTE_MIN(evdev_dlb2_default_info.max_num_events,
+			dlb2->max_num_events_override);
+
+	*dev_info = evdev_dlb2_default_info;
+}
+
+static int
+dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
+			    const struct dlb2_hw_rsrcs *resources_asked)
+{
+	int ret = 0;
+	struct dlb2_create_sched_domain_args *config_params;
+
+	if (resources_asked == NULL) {
+		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
+		ret = EINVAL;
+		goto error_exit;
+	}
+
+	/* Map generic qm resources to dlb2 resources */
+	config_params = &handle->cfg.resources;
+
+	/* DIR ports and queues */
+
+	config_params->num_dir_ports =
+		resources_asked->num_dir_ports;
+
+	config_params->num_dir_credits =
+		resources_asked->num_dir_credits;
+
+	/* LDB queues */
+
+	config_params->num_ldb_queues =
+		resources_asked->num_ldb_queues;
+
+	/* LDB ports */
+
+	config_params->cos_strict = 0; /* Best effort */
+	config_params->num_cos_ldb_ports[0] = 0;
+	config_params->num_cos_ldb_ports[1] = 0;
+	config_params->num_cos_ldb_ports[2] = 0;
+	config_params->num_cos_ldb_ports[3] = 0;
+
+	switch (handle->cos_id) {
+	case DLB2_COS_0:
+		config_params->num_ldb_ports = 0; /* no don't care ports */
+		config_params->num_cos_ldb_ports[0] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_1:
+		config_params->num_ldb_ports = 0; /* no don't care ports */
+		config_params->num_cos_ldb_ports[1] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_2:
+		config_params->num_ldb_ports = 0; /* no don't care ports */
+		config_params->num_cos_ldb_ports[2] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_3:
+		config_params->num_ldb_ports = 0; /* no don't care ports */
+		config_params->num_cos_ldb_ports[3] =
+			resources_asked->num_ldb_ports;
+		break;
+	case DLB2_COS_DEFAULT:
+		/* all ldb ports are don't care ports from a cos perspective */
+		config_params->num_ldb_ports =
+			resources_asked->num_ldb_ports;
+		break;
+	}
+
+	config_params->num_ldb_credits =
+		resources_asked->num_ldb_credits;
+
+	config_params->num_atomic_inflights =
+		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
+		config_params->num_ldb_queues;
+
+	config_params->num_hist_list_entries = resources_asked->num_ldb_ports *
+		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d\n",
+		     config_params->num_ldb_queues,
+		     resources_asked->num_ldb_ports,
+		     config_params->num_dir_ports,
+		     config_params->num_atomic_inflights,
+		     config_params->num_hist_list_entries,
+		     config_params->num_ldb_credits,
+		     config_params->num_dir_credits);
+
+	/* Configure the QM */
+
+	ret = dlb2_iface_sched_domain_create(handle, config_params);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: domain create failed, device_id = %d, (driver ret = %d, extra status: %s)\n",
+			     handle->device_id,
+			     ret,
+			     dlb2_error_strings
+				[config_params->response.status]);
+
+		goto error_exit;
+	}
+
+	handle->domain_id = config_params->response.id;
+	handle->domain_id_valid = 1;
+	handle->cfg.configured = true;
+
+error_exit:
+
+	return ret;
+}
+
+static void
+dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	enum dlb2_configuration_state config_state;
+	int i, j;
+
+	dlb2_iface_domain_reset(dlb2);
+
+	/* Free all dynamically allocated port memory */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
+
+	/* If reconfiguring, mark the device's queues and ports as "previously
+	 * configured." If the user doesn't reconfigure them, the PMD will
+	 * reapply their previous configuration when the device is started.
+	 */
+	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
+		DLB2_NOT_CONFIGURED;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		dlb2->ev_ports[i].qm_port.config_state = config_state;
+		/* Reset setup_done so ports can be reconfigured */
+		dlb2->ev_ports[i].setup_done = false;
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			dlb2->ev_ports[i].link[j].mapped = false;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++)
+		dlb2->ev_queues[i].qm_queue.config_state = config_state;
+
+	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
+		dlb2->ev_queues[i].setup_done = false;
+
+	dlb2->num_ports = 0;
+	dlb2->num_ldb_ports = 0;
+	dlb2->num_dir_ports = 0;
+	dlb2->num_queues = 0;
+	dlb2->num_ldb_queues = 0;
+	dlb2->num_dir_queues = 0;
+	dlb2->configured = false;
+}
+
+/* Note: 1 QM instance per QM device, QM instance/device == event device */
+static int
+dlb2_eventdev_configure(const struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
+	const struct rte_eventdev_data *data = dev->data;
+	const struct rte_event_dev_config *config = &data->dev_conf;
+	int ret;
+
+	/* If this eventdev is already configured, we must release the current
+	 * scheduling domain before attempting to configure a new one.
+	 */
+	if (dlb2->configured) {
+		dlb2_hw_reset_sched_domain(dev, true);
+
+		ret = dlb2_hw_query_resources(dlb2);
+		if (ret) {
+			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
+				     ret, data->dev_id);
+			return ret;
+		}
+	}
+
+	if (config->nb_event_queues > rsrcs->num_queues) {
+		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_queues,
+			     rsrcs->num_queues);
+		return -EINVAL;
+	}
+	if (config->nb_event_ports > (rsrcs->num_ldb_ports
+			+ rsrcs->num_dir_ports)) {
+		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_event_ports,
+			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
+		return -EINVAL;
+	}
+	if (config->nb_events_limit > rsrcs->nb_events_limit) {
+		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n",
+			     config->nb_events_limit,
+			     rsrcs->nb_events_limit);
+		return -EINVAL;
+	}
+
+	if (config->event_dev_cfg & RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT) {
+		dlb2->global_dequeue_wait = false;
+	} else {
+		uint32_t timeout32;
+
+		dlb2->global_dequeue_wait = true;
+
+		/* Craziness here is due to size mismatch in eventdev lib.
+		 * TODO: Submit patch so dequeue API and config use same bit
+		 * width timeout value and same units or time, instead of one
+		 * being 32b ns and the other being 64b ticks.
+		 */
+
+		timeout32 = config->dequeue_timeout_ns;
+
+		dlb2->global_dequeue_wait_ticks =
+			timeout32 * (rte_get_timer_hz() / 1E9);
+	}
+
+	/* Does this platform support umonitor/umwait? */
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_UMWAIT)) {
+		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
+		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
+			DLB2_LOG_ERR("invalid value (%d) for RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
+				     RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
+			return -EINVAL;
+		}
+		dlb2->umwait_allowed = true;
+	}
+
+	/* FIXME: DLB should revert to load-balanced ports if dir are not
+	 * available
+	 */
+
+	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
+	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs->num_dir_ports;
+	/* 1 dir queue per dir port */
+	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports;
+
+	/* Scale down nb_events_limit by 4 for directed credits, since there
+	 * are 4x as many load-balanced credits.
+	 */
+	rsrcs->num_ldb_credits = 0;
+	rsrcs->num_dir_credits = 0;
+
+	if (rsrcs->num_ldb_queues)
+		rsrcs->num_ldb_credits = config->nb_events_limit;
+	if (rsrcs->num_dir_ports)
+		rsrcs->num_dir_credits = config->nb_events_limit / 4;
+	if (dlb2->num_dir_credits_override != -1)
+		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
+
+	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
+		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
+		return -ENODEV;
+	}
+
+	dlb2->new_event_limit = config->nb_events_limit;
+	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
+
+
+	/* Save number of ports/queues for this event dev */
+	dlb2->num_ports = config->nb_event_ports;
+	dlb2->num_queues = config->nb_event_queues;
+	dlb2->num_dir_ports = rsrcs->num_dir_ports;
+	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
+	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
+	dlb2->num_dir_queues = dlb2->num_dir_ports;
+	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
+	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
+	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
+	dlb2->max_dir_credits = rsrcs->num_dir_credits;
+
+	dlb2->configured = true;
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
+		.dev_infos_get    = dlb2_eventdev_info_get,
+		.dev_configure    = dlb2_eventdev_configure,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index fefdf78..5c11736 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -39,4 +39,9 @@ int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 				   enum dlb2_cq_poll_modes *mode);
 
 int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
-				    struct dlb2_get_num_resources_args *rsrcs);
+				struct dlb2_get_num_resources_args *rsrcs);
+
+int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				struct dlb2_create_sched_domain_args *args);
+
+void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 4fb416e..576c1c3 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -26,4 +26,9 @@ extern int (*dlb2_iface_get_cq_poll_mode)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_num_resources)(struct dlb2_hw_dev *handle,
 				struct dlb2_get_num_resources_args *rsrcs);
 
+extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
+				 struct dlb2_create_sched_domain_args *args);
+
+extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 6de8b95..f83f8a1 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -12,6 +12,24 @@
 #include "dlb2_regs.h"
 #include "dlb2_resource.h"
 
+#define DLB2_DOM_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+	DLB2_LIST_HEAD((head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, domain_list, iter)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr, iter) \
+	DLB2_LIST_FOR_EACH(head, ptr, func_list, iter)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, domain_list, it, it_tmp)
+
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp, it, it_tmp) \
+	DLB2_LIST_FOR_EACH_SAFE((head), ptr, ptr_tmp, func_list, it, it_tmp)
+
 static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
 {
 	int i;
@@ -272,3 +290,3219 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
 
 	DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
 }
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+	union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+	r0.field.count = domain->num_ldb_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+	r1.field.count = domain->num_dir_credits;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+		       struct dlb2_function_resources *rsrcs,
+		       u32 domain_id,
+		       u32 cos_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	RTE_SET_USED(iter);
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port, iter) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+				   typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				   struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_ports,
+				   u32 cos_id,
+				   struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_ldb_port *port;
+
+		port = dlb2_get_next_ldb_port(hw, rsrcs,
+					      domain->id.phys_id, cos_id);
+		if (!port) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_ports[cos_id],
+			      &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_ports[cos_id],
+			      &port->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_create_sched_domain_args *args,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      num,
+						      i,
+						      resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+					ret = __dlb2_attach_ldb_ports(hw,
+								      rsrcs,
+								      domain,
+								      1,
+								      cos_id,
+								      resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret < 0)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+			ret = __dlb2_attach_ldb_ports(hw,
+						      rsrcs,
+						      domain,
+						      1,
+						      j,
+						      resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+				 struct dlb2_function_resources *rsrcs,
+				 struct dlb2_hw_domain *domain,
+				 u32 num_ports,
+				 struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+					   typeof(*port));
+		if (!port) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_dir_pq_pairs, &port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		dlb2_list_add(&domain->avail_dir_pq_pairs, &port->domain_list);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+				   struct dlb2_hw_domain *domain,
+				   u32 num_credits,
+				   struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+					struct dlb2_hw_domain *domain,
+					u32 num_atomic_inflights,
+					struct dlb2_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+				     struct dlb2_hw_domain *domain,
+				     u32 num_hist_list_entries,
+				     struct dlb2_cmd_response *resp)
+{
+	struct dlb2_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb2_bitmap_find_set_bit_range(bitmap,
+						      num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+				  struct dlb2_function_resources *rsrcs,
+				  struct dlb2_hw_domain *domain,
+				  u32 num_queues,
+				  struct dlb2_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb2_ldb_queue *queue;
+
+		queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+					    typeof(*queue));
+		if (!queue) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: domain validation failed\n",
+				    __func__);
+			return -EFAULT;
+		}
+
+		dlb2_list_del(&rsrcs->avail_ldb_queues, &queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		dlb2_list_add(&domain->avail_ldb_queues, &queue->domain_list);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+			     struct dlb2_function_resources *rsrcs,
+			     struct dlb2_hw_domain *domain,
+			     struct dlb2_create_sched_domain_args *args,
+			     struct dlb2_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb2_attach_ldb_queues(hw,
+				     rsrcs,
+				     domain,
+				     args->num_ldb_queues,
+				     resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_ports(hw,
+				    rsrcs,
+				    domain,
+				    args,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_ports(hw,
+				    rsrcs,
+				    domain,
+				    args->num_dir_ports,
+				    resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_ldb_credits(rsrcs,
+				      domain,
+				      args->num_ldb_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_dir_credits(rsrcs,
+				      domain,
+				      args->num_dir_credits,
+				      resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+						   domain,
+						   args->num_hist_list_entries,
+						   resp);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_attach_atomic_inflights(rsrcs,
+					   domain,
+					   args->num_atomic_inflights,
+					   resp);
+	if (ret < 0)
+		return ret;
+
+	dlb2_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+				  struct dlb2_create_sched_domain_args *args,
+				  struct dlb2_cmd_response *resp)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb2_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+				  struct dlb2_create_sched_domain_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		    args->num_ldb_queues);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		    args->num_ldb_ports);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		    args->num_cos_ldb_ports[0]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		    args->num_cos_ldb_ports[1]);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		    args->cos_strict);
+	DLB2_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		    args->num_dir_ports);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		    args->num_atomic_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		    args->num_hist_list_entries);
+	DLB2_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		    args->num_ldb_credits);
+	DLB2_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		    args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ *	domain and its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+				struct dlb2_create_sched_domain_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+	if (ret)
+		return ret;
+
+	domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+	if (!domain) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available domains\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (domain->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_domains contains configured domains.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	dlb2_init_domain_rsrc_lists(domain);
+
+	ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to verify args.\n",
+			    __func__);
+
+		return ret;
+	}
+
+	dlb2_list_del(&rsrcs->avail_domains, &domain->func_list);
+
+	dlb2_list_add(&rsrcs->used_domains, &domain->func_list);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+	DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+			     struct dlb2_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb2_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		os_enqueue_four_hcws(hw, hcw, pp_addr);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *port)
+{
+	union dlb2_lsp_cq_dir_dsbl reg;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	int ret;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_disable(hw, port);
+
+		ret = dlb2_drain_dir_cq(hw, port);
+		if (ret < 0)
+			return ret;
+
+		if (toggle_port)
+			dlb2_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_dir_pq_pair *queue)
+{
+	union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_dir_pq_pair *queue)
+{
+	return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+				    struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	reg.field.disabled = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+				     struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_dsbl reg;
+
+	reg.field.disabled = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+				      struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb2_hcw hcw_mem[8], *hcw;
+		void  *pp_addr;
+
+		pp_addr = os_map_producer_port(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		os_enqueue_four_hcws(hw, hcw, pp_addr);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			os_enqueue_four_hcws(hw, hcw, pp_addr);
+
+		os_fence_hcw(hw, pp_addr);
+
+		os_unmap_producer_port(hw, pp_addr);
+	}
+
+	return 0;
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int ret, i;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			if (toggle_port)
+				dlb2_ldb_port_cq_disable(hw, port);
+
+			ret = dlb2_drain_ldb_cq(hw, port);
+			if (ret < 0)
+				return ret;
+
+			if (toggle_port)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+
+	return 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+				struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_atm_active r1;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+				    struct dlb2_ldb_queue *queue)
+{
+	return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb2_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+					   struct dlb2_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to unmap domain queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+		if (ret < 0)
+			return ret;
+
+		if (dlb2_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: failed to empty queues\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = true;
+
+			dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+			   u32 id,
+			   bool vdev_req,
+			   unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iter1) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter2)
+			if (queue->id.virt_id == id)
+				return queue;
+	}
+
+	DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue, iter1)
+		if (queue->id.virt_id == id)
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+						      u32 id,
+						      bool vdev_req,
+						      unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iteration;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_hw_domain *domain;
+	RTE_SET_USED(iteration);
+
+	if (id >= DLB2_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain, iteration)
+		if (domain->id.virt_id == id)
+			return domain;
+
+	return NULL;
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot,
+					   enum dlb2_qid_map_state new_state)
+{
+	enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb2_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (!domain) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB2_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB2_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB2_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB2_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB2_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB2_HW_DBG(hw,
+		    "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB2_HW_ERR(hw,
+		    "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		    __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+				enum dlb2_qid_map_state state,
+				int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+				      enum dlb2_qid_map_state state,
+				      struct dlb2_ldb_queue *queue,
+				      int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+					      struct dlb2_hw_domain *domain,
+					      struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain,
+					     struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int slot, i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+						struct dlb2_ldb_port *port,
+						int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 0;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+	r0.field.cq = port->id.phys_id;
+	r0.field.qidix = slot;
+	r0.field.value = 1;
+	r0.field.inflight_ok_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+					struct dlb2_ldb_port *p,
+					struct dlb2_ldb_queue *q,
+					u8 priority)
+{
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_lsp_cq2qid0 r1;
+	union dlb2_atm_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix_00 r3;
+	union dlb2_lsp_qid2cqidix2_00 r4;
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+	    !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+	r0.field.v |= 1 << i;
+	r0.field.prio |= (priority & 0x7) << i * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+	else
+		r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		r1.field.qid_p0 = q->id.phys_id;
+	if (i == 1 || i == 5)
+		r1.field.qid_p1 = q->id.phys_id;
+	if (i == 2 || i == 6)
+		r1.field.qid_p2 = q->id.phys_id;
+	if (i == 3 || i == 7)
+		r1.field.qid_p3 = q->id.phys_id;
+
+	if (i < 4)
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+	else
+		DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	r4.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+						  p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		r2.field.cq_p0 |= 1 << i;
+		r3.field.cq_p0 |= 1 << i;
+		r4.field.cq_p0 |= 1 << i;
+		break;
+
+	case 1:
+		r2.field.cq_p1 |= 1 << i;
+		r3.field.cq_p1 |= 1 << i;
+		r4.field.cq_p1 |= 1 << i;
+		break;
+
+	case 2:
+		r2.field.cq_p2 |= 1 << i;
+		r3.field.cq_p2 |= 1 << i;
+		r4.field.cq_p2 |= 1 << i;
+		break;
+
+	case 3:
+		r2.field.cq_p3 |= 1 << i;
+		r3.field.cq_p3 |= 1 << i;
+		r4.field.cq_p3 |= 1 << i;
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		    r3.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		    r4.val);
+
+	dlb2_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB2_QUEUE_MAPPED;
+
+	return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+					   struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int slot)
+{
+	union dlb2_lsp_qid_aqed_active_cnt r0;
+	union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	/* Set the atomic scheduling haswork bit */
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.rlist_haswork_v = r0.field.count > 0;
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 1;
+	r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      u8 slot)
+{
+	union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.rlist_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	memset(&r2, 0, sizeof(r2));
+
+	r2.field.cq = port->id.phys_id;
+	r2.field.qidix = slot;
+	r2.field.value = 0;
+	r2.field.nalb_haswork_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+	dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+					      struct dlb2_ldb_queue *queue)
+{
+	union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+	r0.field.limit = queue->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+						struct dlb2_ldb_queue *queue)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+						struct dlb2_hw_domain *domain,
+						struct dlb2_ldb_port *port,
+						struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_lsp_qid_ldb_infl_cnt r0;
+	enum dlb2_qid_map_state state;
+	int slot, ret, i;
+	u8 prio;
+	RTE_SET_USED(iter);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: non-zero QID inflight count\n",
+			    __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			state = DLB2_QUEUE_MAPPED;
+			if (!dlb2_port_find_slot_queue(port, state,
+						       queue, &slot))
+				continue;
+
+			dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+					 struct dlb2_ldb_port *port,
+					 struct dlb2_ldb_queue *queue,
+					 u8 priority)
+{
+	union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	int domain_id, slot, ret;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+	if (!domain) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: unable to find domain %d\n",
+			    __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+		DLB2_HW_ERR(hw,
+			    "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	r0.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (r0.field.count) {
+		if (port->enabled)
+			dlb2_ldb_port_cq_enable(hw, port);
+
+		dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		if (!os_worker_active(hw))
+			os_schedule_work(hw);
+
+		return 1;
+	}
+
+	return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain,
+					struct dlb2_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		union dlb2_lsp_qid_ldb_infl_cnt r0;
+		struct dlb2_ldb_queue *queue;
+		int qid;
+
+		if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (!queue) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: unable to find queue %d\n",
+				    __func__, qid);
+			continue;
+		}
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count)
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb2_ldb_port_cq_disable(hw, port);
+
+		dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+		if (r0.field.count) {
+			if (port->enabled)
+				dlb2_ldb_port_cq_enable(hw, port);
+
+			dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+				   struct dlb2_ldb_port *port,
+				   struct dlb2_ldb_queue *queue)
+{
+	enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+	union dlb2_lsp_cq2priov r0;
+	union dlb2_atm_qid2cqidix_00 r1;
+	union dlb2_lsp_qid2cqidix_00 r2;
+	union dlb2_lsp_qid2cqidix2_00 r3;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB2_QUEUE_MAPPED;
+	in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: QID %d isn't mapped\n",
+			    __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+	r0.field.v &= ~(1 << i);
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+	r1.val = DLB2_CSR_RD(hw,
+			     DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	r2.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	r3.val = DLB2_CSR_RD(hw,
+			     DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		r1.field.cq_p0 &= ~(1 << i);
+		r2.field.cq_p0 &= ~(1 << i);
+		r3.field.cq_p0 &= ~(1 << i);
+		break;
+
+	case 1:
+		r1.field.cq_p1 &= ~(1 << i);
+		r2.field.cq_p1 &= ~(1 << i);
+		r3.field.cq_p1 &= ~(1 << i);
+		break;
+
+	case 2:
+		r1.field.cq_p2 &= ~(1 << i);
+		r2.field.cq_p2 &= ~(1 << i);
+		r3.field.cq_p2 &= ~(1 << i);
+		break;
+
+	case 3:
+		r1.field.cq_p3 &= ~(1 << i);
+		r2.field.cq_p3 &= ~(1 << i);
+		r3.field.cq_p3 &= ~(1 << i);
+		break;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+		    r1.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+		    r2.val);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+		    r3.val);
+
+	dlb2_flush_csr(hw);
+
+	unmapped = DLB2_QUEUE_UNMAPPED;
+
+	return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+				 struct dlb2_hw_domain *domain,
+				 struct dlb2_ldb_port *port,
+				 struct dlb2_ldb_queue *queue,
+				 u8 prio)
+{
+	if (domain->started)
+		return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   int slot)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb2_ldb_port_qid_map *map;
+		struct dlb2_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain,
+					  struct dlb2_ldb_port *port)
+{
+	union dlb2_lsp_cq_ldb_infl_cnt r0;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (r0.field.count > 0)
+		return false;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			dlb2_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			port->enabled = false;
+
+			dlb2_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_dir_vpp_v r1;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		unsigned int offs;
+		u32 virt_id;
+
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+	}
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+					 struct dlb2_hw_domain *domain,
+					 unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_vf_ldb_vpp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.vpp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			unsigned int offs;
+			u32 virt_id;
+
+			if (hw->virt_mode == DLB2_VIRT_SRIOV)
+				virt_id = port->id.virt_id;
+			else
+				virt_id = port->id.phys_id;
+
+			offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+			DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+	union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+				    r0.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+				    r1.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+	union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	r0.field.en_tim = 0;
+	r0.field.en_depth = 0;
+
+	r1.field.wd_enable = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+			    r0.val);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+			    r1.val);
+	}
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+		union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+		union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+				    r1.val);
+
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID_V(idx),
+				    r2.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_LDB_VQID2QID(idx),
+				    r3.val);
+		}
+	}
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+					  struct dlb2_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+		int idx;
+
+		idx = domain_offset + queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+		if (queue->id.vdev_owned) {
+			idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+				queue->id.virt_id;
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID_V(idx),
+				    r1.val);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_VF_DIR_VQID2QID(idx),
+				    r2.val);
+		}
+	}
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+					       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_chp_sn_chk_enbl r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.en = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter) {
+			int i;
+
+			for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+				if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					    __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter) {
+		port->enabled = false;
+
+		dlb2_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	union dlb2_sys_dir_pp_v r1;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+			    r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	union dlb2_sys_ldb_pp_v r1;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	r1.field.pp_v = 0;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			DLB2_CSR_WR(hw,
+				    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+				    r1.val);
+	}
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_ldb_queue *queue;
+	int i;
+	RTE_SET_USED(iter);
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty ldb queue %d\n",
+				    __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port, iter) {
+			if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb2_ldb_cq_token_count(hw, ldb_port)) {
+				DLB2_HW_ERR(hw,
+					    "[%s()] Internal error: failed to empty ldb port %d\n",
+					    __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port, iter) {
+		if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir queue %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb2_dir_cq_token_count(hw, dir_port)) {
+			DLB2_HW_ERR(hw,
+				    "[%s()] Internal error: failed to empty dir port %d\n",
+				    __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						   struct dlb2_ldb_port *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_LDB_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP2PP(offs),
+			    DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_LDB_VPP_V(offs),
+			    DLB2_SYS_VF_LDB_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    DLB2_SYS_LDB_PP_V_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_DSBL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_LIM_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_BASE_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID0(port->id.phys_id),
+		    DLB2_LSP_CQ2QID0_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2QID1(port->id.phys_id),
+		    DLB2_LSP_CQ2QID1_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+		    DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			__dlb2_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+				       struct dlb2_dir_pq_pair *port)
+{
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_DSBL_RST);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ISR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_AT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_PASID_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ_FMT_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		    DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		    DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ2VAS_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+		    DLB2_SYS_DIR_PP2VDEV_RST);
+
+	if (port->id.vdev_owned) {
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP2PP(offs),
+			    DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_VF_DIR_VPP_V(offs),
+			    DLB2_SYS_VF_DIR_VPP_V_RST);
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+						 struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		__dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			    DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			    DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+			    DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			    DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			    DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_ITS(queue_id),
+			    DLB2_SYS_LDB_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN(queue_id),
+			    DLB2_CHP_ORD_QID_SN_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+			    DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_V(queue_id),
+			    DLB2_SYS_LDB_QID_V_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID_CFG_V(queue_id),
+			    DLB2_SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB2_CSR_WR(hw,
+				    offs[queue->sn_group],
+				    DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_LSP_QID2CQIDIX2(queue_id, i),
+				    DLB2_LSP_QID2CQIDIX2_00_RST);
+
+			DLB2_CSR_WR(hw,
+				    DLB2_ATM_QID2CQIDIX(queue_id, i),
+				    DLB2_ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+						  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *queue;
+	RTE_SET_USED(iter);
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue, iter) {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			    DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_ITS_RST);
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+			    DLB2_SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+					struct dlb2_hw_domain *domain)
+{
+	dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb2_domain_reset_dir_port_registers(hw, domain);
+
+	dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		    DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_dir_pq_pair *tmp_dir_port;
+	struct dlb2_ldb_queue *tmp_ldb_queue;
+	struct dlb2_ldb_port *tmp_ldb_port;
+	struct dlb2_list_entry *iter1;
+	struct dlb2_list_entry *iter2;
+	struct dlb2_function_resources *rsrcs;
+	struct dlb2_dir_pq_pair *dir_port;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_ldb_port *ldb_port;
+	struct dlb2_list_head *list;
+	int ret, i;
+	RTE_SET_USED(tmp_dir_port);
+	RTE_SET_USED(tmp_ldb_queue);
+	RTE_SET_USED(tmp_ldb_port);
+	RTE_SET_USED(iter1);
+	RTE_SET_USED(iter2);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list = &domain->used_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb2_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		dlb2_list_del(&domain->used_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list = &domain->avail_ldb_queues;
+	DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue, iter1, iter2) {
+		ldb_queue->owned = false;
+
+		dlb2_list_del(&domain->avail_ldb_queues,
+			      &ldb_queue->domain_list);
+		dlb2_list_add(&rsrcs->avail_ldb_queues,
+			      &ldb_queue->func_list);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		list = &domain->used_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB2_QUEUE_UNMAPPED;
+
+			dlb2_list_del(&domain->used_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list = &domain->avail_ldb_ports[i];
+		DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port,
+				       iter1, iter2) {
+			ldb_port->owned = false;
+
+			dlb2_list_del(&domain->avail_ldb_ports[i],
+				      &ldb_port->domain_list);
+			dlb2_list_add(&rsrcs->avail_ldb_ports[i],
+				      &ldb_port->func_list);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list = &domain->used_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		dlb2_list_del(&domain->used_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list = &domain->avail_dir_pq_pairs;
+	DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port, iter1, iter2) {
+		dir_port->owned = false;
+
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &dir_port->domain_list);
+
+		dlb2_list_add(&rsrcs->avail_dir_pq_pairs,
+			      &dir_port->func_list);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				    domain->hist_list_entry_base,
+				    domain->total_hist_list_entries);
+	if (ret) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			    __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	dlb2_list_del(&rsrcs->used_domains, &domain->func_list);
+	dlb2_list_add(&rsrcs->avail_domains, &domain->func_list);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+					    struct dlb2_hw_domain *domain,
+					    struct dlb2_ldb_queue *queue)
+{
+	struct dlb2_ldb_port *port;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		if (!dlb2_list_empty(&domain->used_ldb_ports[i]))
+			break;
+	}
+
+	if (i == DLB2_NUM_COS_DOMAINS) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: No configured LDB ports\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb2_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+					     struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+	RTE_SET_USED(iter);
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+	 * prior to this in dlb2_domain_drain_mapped_queues().
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter) {
+		if (queue->num_mappings != 0 ||
+		    dlb2_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ *	hardware resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain || !domain->configured)
+		return -EINVAL;
+
+	/* Disable VPPs */
+	if (vdev_req) {
+		dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+		dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+	}
+
+	/* Disable CQ interrupts */
+	dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+	dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+	dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb2_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb2_domain_drain_mapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+	if (ret < 0)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb2_domain_disable_ldb_cqs(hw, domain);
+
+	dlb2_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb2_domain_disable_dir_cqs(hw, domain);
+
+	/* Disable PPs */
+	dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+	ret = dlb2_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Reset the QID and port state. */
+	dlb2_domain_reset_registers(hw, domain);
+
+	/* Hardware reset complete. Reset the domain's software state */
+	ret = dlb2_domain_reset_software_state(hw, domain);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
+unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+		struct dlb2_hw_domain *domain = &hw->domains[i];
+
+		num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 1c275ff..ca1ad69 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -618,3 +618,17 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
 /****** Device configuration ******/
 /**********************************/
 
+int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+			    struct dlb2_create_sched_domain_args *args,
+			    struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_sched_domain(hw, args, resp, NOT_VF_REQ,
+					   PF_ID_ZERO);
+}
+
+int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 8c5ec20..21f28a4 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -114,16 +114,60 @@ dlb2_pf_get_cq_poll_mode(struct dlb2_hw_dev *handle,
 	return 0;
 }
 
+static int
+dlb2_pf_sched_domain_create(struct dlb2_hw_dev *handle,
+			    struct dlb2_create_sched_domain_args *arg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (dlb2_dev->domain_reset_failed) {
+		response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto done;
+	}
+
+	ret = dlb2_pf_create_sched_domain(&dlb2_dev->hw, arg, &response);
+	if (ret)
+		goto done;
+
+done:
+
+	arg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static void
+dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
+{
+	struct dlb2_dev *dlb2_dev;
+	int ret;
+
+	dlb2_dev = (struct dlb2_dev *)dlb2->qm_instance.pf_dev;
+	ret = dlb2_pf_reset_domain(&dlb2_dev->hw, dlb2->qm_instance.domain_id);
+	if (ret)
+		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
 
 	dlb2_iface_low_level_io_init = dlb2_pf_low_level_io_init;
 	dlb2_iface_open = dlb2_pf_open;
+	dlb2_iface_domain_reset = dlb2_pf_domain_reset;
 	dlb2_iface_get_device_version = dlb2_pf_get_device_version;
 	dlb2_iface_hardware_init = dlb2_pf_hardware_init;
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
+	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

Re: [dpdk-dev] [PATCH 08/22] event/dlb2: add infos get and configure

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 08/22] event/dlb2: add infos get and configure
> 
> Add support for configuring the DLB hardware.

This is a large patch, even if most of the code is in the low-level/hardware-specific
base directory, and could use more explanation in the commit message.

> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  drivers/event/dlb2/dlb2.c                  |  334 +++
>  drivers/event/dlb2/dlb2_iface.c            |    7 +-
>  drivers/event/dlb2/dlb2_iface.h            |    5 +
>  drivers/event/dlb2/pf/base/dlb2_resource.c | 3234
> ++++++++++++++++++++++++++++
>  drivers/event/dlb2/pf/dlb2_main.c          |   14 +
>  drivers/event/dlb2/pf/dlb2_pf.c            |   44 +
>  6 files changed, 3637 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> index 0d6fea4..58e953b 100644
> --- a/drivers/event/dlb2/dlb2.c
> +++ b/drivers/event/dlb2/dlb2.c
> @@ -92,6 +92,28 @@ dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
>  	return 0;
>  }
> 
> +static void
> +dlb2_free_qe_mem(struct dlb2_port *qm_port)
> +{
> +	if (qm_port == NULL)
> +		return;
> +
> +	if (qm_port->qe4) {
> +		rte_free(qm_port->qe4);
> +		qm_port->qe4 = NULL;
> +	}
> +
> +	if (qm_port->int_arm_qe) {
> +		rte_free(qm_port->int_arm_qe);
> +		qm_port->int_arm_qe = NULL;
> +	}
> +
> +	if (qm_port->consume_qe) {
> +		rte_free(qm_port->consume_qe);
> +		qm_port->consume_qe = NULL;
> +	}

You can pass a NULL pointer to rte_free() -- it's guaranteed to do nothing in that case.

> +}
> +
>  /* override defaults with value(s) provided on command line */
>  static void
>  dlb2_init_queue_depth_thresholds(struct dlb2_eventdev *dlb2,
> @@ -366,10 +388,322 @@ set_qid_depth_thresh(const char *key __rte_unused,
>  }
> 
>  static void
> +dlb2_eventdev_info_get(struct rte_eventdev *dev,
> +		       struct rte_event_dev_info *dev_info)
> +{
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	int ret;
> +
> +	ret = dlb2_hw_query_resources(dlb2);
> +	if (ret) {
> +		const struct rte_eventdev_data *data = dev->data;
> +
> +		DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
> +			     ret, data->dev_id);
> +		/* fn is void, so fall through and return values set up in
> +		 * probe
> +		 */
> +	}
> +
> +	/* Add num resources currently owned by this domain.
> +	 * These would become available if the scheduling domain were reset
> due
> +	 * to the application recalling eventdev_configure to *reconfigure* the
> +	 * domain.
> +	 */
> +	evdev_dlb2_default_info.max_event_ports += dlb2->num_ldb_ports;
> +	evdev_dlb2_default_info.max_event_queues += dlb2-
> >num_ldb_queues;
> +	evdev_dlb2_default_info.max_num_events += dlb2->max_ldb_credits;
> +
> +	evdev_dlb2_default_info.max_event_queues =
> +		RTE_MIN(evdev_dlb2_default_info.max_event_queues,
> +			RTE_EVENT_MAX_QUEUES_PER_DEV);
> +
> +	evdev_dlb2_default_info.max_num_events =
> +		RTE_MIN(evdev_dlb2_default_info.max_num_events,
> +			dlb2->max_num_events_override);
> +
> +	*dev_info = evdev_dlb2_default_info;
> +}
> +
> +static int
> +dlb2_hw_create_sched_domain(struct dlb2_hw_dev *handle,
> +			    const struct dlb2_hw_rsrcs *resources_asked)
> +{
> +	int ret = 0;
> +	struct dlb2_create_sched_domain_args *config_params;
> +
> +	if (resources_asked == NULL) {
> +		DLB2_LOG_ERR("dlb2: dlb2_create NULL parameter\n");
> +		ret = EINVAL;
> +		goto error_exit;
> +	}
> +
> +	/* Map generic qm resources to dlb2 resources */
> +	config_params = &handle->cfg.resources;
> +
> +	/* DIR ports and queues */
> +
> +	config_params->num_dir_ports =
> +		resources_asked->num_dir_ports;
> +
> +	config_params->num_dir_credits =
> +		resources_asked->num_dir_credits;
> +
> +	/* LDB queues */
> +
> +	config_params->num_ldb_queues =
> +		resources_asked->num_ldb_queues;
> +
> +	/* LDB ports */
> +
> +	config_params->cos_strict = 0; /* Best effort */
> +	config_params->num_cos_ldb_ports[0] = 0;
> +	config_params->num_cos_ldb_ports[1] = 0;
> +	config_params->num_cos_ldb_ports[2] = 0;
> +	config_params->num_cos_ldb_ports[3] = 0;
> +
> +	switch (handle->cos_id) {
> +	case DLB2_COS_0:
> +		config_params->num_ldb_ports = 0; /* no don't care ports */
> +		config_params->num_cos_ldb_ports[0] =
> +			resources_asked->num_ldb_ports;
> +		break;
> +	case DLB2_COS_1:
> +		config_params->num_ldb_ports = 0; /* no don't care ports */
> +		config_params->num_cos_ldb_ports[1] =
> +			resources_asked->num_ldb_ports;
> +		break;
> +	case DLB2_COS_2:
> +		config_params->num_ldb_ports = 0; /* no don't care ports */
> +		config_params->num_cos_ldb_ports[2] =
> +			resources_asked->num_ldb_ports;
> +		break;
> +	case DLB2_COS_3:
> +		config_params->num_ldb_ports = 0; /* no don't care ports */
> +		config_params->num_cos_ldb_ports[3] =
> +			resources_asked->num_ldb_ports;
> +		break;
> +	case DLB2_COS_DEFAULT:
> +		/* all ldb ports are don't care ports from a cos perspective */
> +		config_params->num_ldb_ports =
> +			resources_asked->num_ldb_ports;
> +		break;
> +	}
> +
> +	config_params->num_ldb_credits =
> +		resources_asked->num_ldb_credits;
> +
> +	config_params->num_atomic_inflights =
> +		DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE *
> +		config_params->num_ldb_queues;
> +
> +	config_params->num_hist_list_entries = resources_asked-
> >num_ldb_ports *
> +		DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
> +
> +	DLB2_LOG_DBG("sched domain create - ldb_qs=%d, ldb_ports=%d,
> dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d,
> dir_credits=%d\n",
> +		     config_params->num_ldb_queues,
> +		     resources_asked->num_ldb_ports,
> +		     config_params->num_dir_ports,
> +		     config_params->num_atomic_inflights,
> +		     config_params->num_hist_list_entries,
> +		     config_params->num_ldb_credits,
> +		     config_params->num_dir_credits);
> +
> +	/* Configure the QM */
> +
> +	ret = dlb2_iface_sched_domain_create(handle, config_params);
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2: domain create failed, device_id = %d,
> (driver ret = %d, extra status: %s)\n",
> +			     handle->device_id,
> +			     ret,
> +			     dlb2_error_strings
> +				[config_params->response.status]);

Perhaps rename/shorten config_params so this array+index can fit on one line, or
assign config_params->response.status to a local variable? 

> +
> +		goto error_exit;
> +	}
> +
> +	handle->domain_id = config_params->response.id;
> +	handle->domain_id_valid = 1;
> +	handle->cfg.configured = true;
> +
> +error_exit:
> +
> +	return ret;
> +}
> +
> +static void
> +dlb2_hw_reset_sched_domain(const struct rte_eventdev *dev, bool reconfig)
> +{
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	enum dlb2_configuration_state config_state;
> +	int i, j;
> +
> +	dlb2_iface_domain_reset(dlb2);
> +
> +	/* Free all dynamically allocated port memory */
> +	for (i = 0; i < dlb2->num_ports; i++)
> +		dlb2_free_qe_mem(&dlb2->ev_ports[i].qm_port);
> +
> +	/* If reconfiguring, mark the device's queues and ports as "previously
> +	 * configured." If the user doesn't reconfigure them, the PMD will
> +	 * reapply their previous configuration when the device is started.
> +	 */
> +	config_state = (reconfig) ? DLB2_PREV_CONFIGURED :
> +		DLB2_NOT_CONFIGURED;
> +
> +	for (i = 0; i < dlb2->num_ports; i++) {
> +		dlb2->ev_ports[i].qm_port.config_state = config_state;
> +		/* Reset setup_done so ports can be reconfigured */
> +		dlb2->ev_ports[i].setup_done = false;
> +		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
> +			dlb2->ev_ports[i].link[j].mapped = false;
> +	}
> +
> +	for (i = 0; i < dlb2->num_queues; i++)
> +		dlb2->ev_queues[i].qm_queue.config_state = config_state;
> +
> +	for (i = 0; i < DLB2_MAX_NUM_QUEUES; i++)
> +		dlb2->ev_queues[i].setup_done = false;
> +
> +	dlb2->num_ports = 0;
> +	dlb2->num_ldb_ports = 0;
> +	dlb2->num_dir_ports = 0;
> +	dlb2->num_queues = 0;
> +	dlb2->num_ldb_queues = 0;
> +	dlb2->num_dir_queues = 0;
> +	dlb2->configured = false;
> +}
> +
> +/* Note: 1 QM instance per QM device, QM instance/device == event device */
> +static int
> +dlb2_eventdev_configure(const struct rte_eventdev *dev)
> +{
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
> +	struct dlb2_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max;
> +	const struct rte_eventdev_data *data = dev->data;
> +	const struct rte_event_dev_config *config = &data->dev_conf;
> +	int ret;
> +
> +	/* If this eventdev is already configured, we must release the current
> +	 * scheduling domain before attempting to configure a new one.
> +	 */
> +	if (dlb2->configured) {
> +		dlb2_hw_reset_sched_domain(dev, true);
> +
> +		ret = dlb2_hw_query_resources(dlb2);
> +		if (ret) {
> +			DLB2_LOG_ERR("get resources err=%d, devid=%d\n",
> +				     ret, data->dev_id);
> +			return ret;
> +		}
> +	}
> +
> +	if (config->nb_event_queues > rsrcs->num_queues) {
> +		DLB2_LOG_ERR("nb_event_queues parameter (%d) exceeds the
> QM device's capabilities (%d).\n",
> +			     config->nb_event_queues,
> +			     rsrcs->num_queues);
> +		return -EINVAL;
> +	}
> +	if (config->nb_event_ports > (rsrcs->num_ldb_ports
> +			+ rsrcs->num_dir_ports)) {
> +		DLB2_LOG_ERR("nb_event_ports parameter (%d) exceeds the
> QM device's capabilities (%d).\n",
> +			     config->nb_event_ports,
> +			     (rsrcs->num_ldb_ports + rsrcs->num_dir_ports));
> +		return -EINVAL;
> +	}
> +	if (config->nb_events_limit > rsrcs->nb_events_limit) {
> +		DLB2_LOG_ERR("nb_events_limit parameter (%d) exceeds the
> QM device's capabilities (%d).\n",
> +			     config->nb_events_limit,
> +			     rsrcs->nb_events_limit);
> +		return -EINVAL;
> +	}
> +
> +	if (config->event_dev_cfg &
> RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT) {
> +		dlb2->global_dequeue_wait = false;

Nit -- coding style discourages braces on single-statement conditionals, even if
It's followed by a multi-statement else block.

https://doc.dpdk.org/guides/contributing/coding_style.html#control-statements-and-loops

> +	} else {
> +		uint32_t timeout32;
> +
> +		dlb2->global_dequeue_wait = true;
> +
> +		/* Craziness here is due to size mismatch in eventdev lib.
> +		 * TODO: Submit patch so dequeue API and config use same bit
> +		 * width timeout value and same units or time, instead of one
> +		 * being 32b ns and the other being 64b ticks.

Leftover TODO

> +		 */
> +
> +		timeout32 = config->dequeue_timeout_ns;
> +
> +		dlb2->global_dequeue_wait_ticks =
> +			timeout32 * (rte_get_timer_hz() / 1E9);
> +	}
> +
> +	/* Does this platform support umonitor/umwait? */
> +	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_UMWAIT)) {
> +		if (RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 0 &&
> +		    RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE != 1) {
> +			DLB2_LOG_ERR("invalid value (%d) for
> RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE, must be 0 or 1.\n",
> +
> RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE);
> +			return -EINVAL;
> +		}
> +		dlb2->umwait_allowed = true;
> +	}
> +
> +	/* FIXME: DLB should revert to load-balanced ports if dir are not
> +	 * available
> +	 */

Leftover FIXME

> +
> +	rsrcs->num_dir_ports = config->nb_single_link_event_port_queues;
> +	rsrcs->num_ldb_ports  = config->nb_event_ports - rsrcs-
> >num_dir_ports;
> +	/* 1 dir queue per dir port */
> +	rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs-
> >num_dir_ports;
> +
> +	/* Scale down nb_events_limit by 4 for directed credits, since there
> +	 * are 4x as many load-balanced credits.
> +	 */
> +	rsrcs->num_ldb_credits = 0;
> +	rsrcs->num_dir_credits = 0;
> +
> +	if (rsrcs->num_ldb_queues)
> +		rsrcs->num_ldb_credits = config->nb_events_limit;
> +	if (rsrcs->num_dir_ports)
> +		rsrcs->num_dir_credits = config->nb_events_limit / 4;
> +	if (dlb2->num_dir_credits_override != -1)
> +		rsrcs->num_dir_credits = dlb2->num_dir_credits_override;
> +
> +	if (dlb2_hw_create_sched_domain(handle, rsrcs) < 0) {
> +		DLB2_LOG_ERR("dlb2_hw_create_sched_domain failed\n");
> +		return -ENODEV;
> +	}
> +
> +	dlb2->new_event_limit = config->nb_events_limit;
> +	__atomic_store_n(&dlb2->inflights, 0, __ATOMIC_SEQ_CST);
> +
> +

Nit: extra whitespace

> +	/* Save number of ports/queues for this event dev */
> +	dlb2->num_ports = config->nb_event_ports;
> +	dlb2->num_queues = config->nb_event_queues;
> +	dlb2->num_dir_ports = rsrcs->num_dir_ports;
> +	dlb2->num_ldb_ports = dlb2->num_ports - dlb2->num_dir_ports;
> +	dlb2->num_ldb_queues = dlb2->num_queues - dlb2->num_dir_ports;
> +	dlb2->num_dir_queues = dlb2->num_dir_ports;
> +	dlb2->ldb_credit_pool = rsrcs->num_ldb_credits;
> +	dlb2->max_ldb_credits = rsrcs->num_ldb_credits;
> +	dlb2->dir_credit_pool = rsrcs->num_dir_credits;
> +	dlb2->max_dir_credits = rsrcs->num_dir_credits;
> +
> +	dlb2->configured = true;
> +
> +	return 0;
> +}

Thanks,
Gage

[dpdk-dev] [PATCH 09/22] event/dlb2: add queue and port default conf

[Timothy McDaniel]

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 58e953b..9ff371a 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -698,12 +698,45 @@ dlb2_eventdev_configure(const struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
+				    uint8_t port_id,
+				    struct rte_event_port_conf *port_conf)
+{
+	RTE_SET_USED(port_id);
+	struct dlb2_eventdev *dlb2;
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	/* FIXME: Arbitrary values */
+	port_conf->new_event_threshold = dlb2->new_event_limit;
+	port_conf->dequeue_depth = 32;
+	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
+	port_conf->event_port_cfg = 0;
+}
+
+static void
+dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
+				     uint8_t queue_id,
+				     struct rte_event_queue_conf *queue_conf)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(queue_id);
+	/* FIXME: Arbitrary values */
+	queue_conf->nb_atomic_flows = 1024;
+	queue_conf->nb_atomic_order_sequences = 64;
+	queue_conf->event_queue_cfg = 0;
+	queue_conf->priority = 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

Re: [dpdk-dev] [PATCH 09/22] event/dlb2: add queue and port default conf

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 09/22] event/dlb2: add queue and port default conf
> 
> Add support for getting the queue and port default configuration.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  drivers/event/dlb2/dlb2.c | 33 +++++++++++++++++++++++++++++++++
>  1 file changed, 33 insertions(+)
> 
> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> index 58e953b..9ff371a 100644
> --- a/drivers/event/dlb2/dlb2.c
> +++ b/drivers/event/dlb2/dlb2.c
> @@ -698,12 +698,45 @@ dlb2_eventdev_configure(const struct rte_eventdev
> *dev)
>  }
> 
>  static void
> +dlb2_eventdev_port_default_conf_get(struct rte_eventdev *dev,
> +				    uint8_t port_id,
> +				    struct rte_event_port_conf *port_conf)
> +{
> +	RTE_SET_USED(port_id);
> +	struct dlb2_eventdev *dlb2;
> +
> +	dlb2 = dlb2_pmd_priv(dev);
> +
> +	/* FIXME: Arbitrary values */

Leftover FIXME. Hopefully not too arbitrary :p.

> +	port_conf->new_event_threshold = dlb2->new_event_limit;
> +	port_conf->dequeue_depth = 32;
> +	port_conf->enqueue_depth = DLB2_MAX_ENQUEUE_DEPTH;
> +	port_conf->event_port_cfg = 0;
> +}
> +
> +static void
> +dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
> +				     uint8_t queue_id,
> +				     struct rte_event_queue_conf *queue_conf)
> +{
> +	RTE_SET_USED(dev);
> +	RTE_SET_USED(queue_id);
> +	/* FIXME: Arbitrary values */

Ditto

> +	queue_conf->nb_atomic_flows = 1024;
> +	queue_conf->nb_atomic_order_sequences = 64;
> +	queue_conf->event_queue_cfg = 0;
> +	queue_conf->priority = 0;
> +}

Thanks,
Gage

[dpdk-dev] [PATCH 10/22] event/dlb2: add queue setup

[Timothy McDaniel]

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 316 ++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |  12 +
 drivers/event/dlb2/dlb2_iface.h            |  12 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 464 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  82 +++++
 6 files changed, 896 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 9ff371a..366e194 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -728,6 +728,321 @@ dlb2_eventdev_queue_default_conf_get(struct rte_eventdev *dev,
 	queue_conf->priority = 0;
 }
 
+static int32_t
+dlb2_get_sn_allocation(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_allocation_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_set_sn_allocation(struct dlb2_eventdev *dlb2, int group, int num)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_set_sn_allocation_args cfg;
+	int ret;
+
+	cfg.num = num;
+	cfg.group = group;
+
+	ret = dlb2_iface_set_sn_allocation(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: set_sn_allocation ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return ret;
+}
+
+static int32_t
+dlb2_get_sn_occupancy(struct dlb2_eventdev *dlb2, int group)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_sn_occupancy_args cfg;
+	int ret;
+
+	cfg.group = group;
+
+	ret = dlb2_iface_get_sn_occupancy(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_sn_occupancy ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+/* Query the current sequence number allocations and, if they conflict with the
+ * requested LDB queue configuration, attempt to re-allocate sequence numbers.
+ * This is best-effort; if it fails, the PMD will attempt to configure the
+ * load-balanced queue and return an error.
+ */
+static void
+dlb2_program_sn_allocation(struct dlb2_eventdev *dlb2,
+			   const struct rte_event_queue_conf *queue_conf)
+{
+	int grp_occupancy[DLB2_NUM_SN_GROUPS];
+	int grp_alloc[DLB2_NUM_SN_GROUPS];
+	int i, sequence_numbers;
+
+	sequence_numbers = (int)queue_conf->nb_atomic_order_sequences;
+
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		int total_slots;
+
+		grp_alloc[i] = dlb2_get_sn_allocation(dlb2, i);
+		if (grp_alloc[i] < 0)
+			return;
+
+		total_slots = DLB2_MAX_LDB_SN_ALLOC / grp_alloc[i];
+
+		grp_occupancy[i] = dlb2_get_sn_occupancy(dlb2, i);
+		if (grp_occupancy[i] < 0)
+			return;
+
+		/* DLB has at least one available slot for the requested
+		 * sequence numbers, so no further configuration required.
+		 */
+		if (grp_alloc[i] == sequence_numbers &&
+		    grp_occupancy[i] < total_slots)
+			return;
+	}
+
+	/* None of the sequence number groups are configured for the requested
+	 * sequence numbers, so we have to reconfigure one of them. This is
+	 * only possible if a group is not in use.
+	 */
+	for (i = 0; i < DLB2_NUM_SN_GROUPS; i++) {
+		if (grp_occupancy[i] == 0)
+			break;
+	}
+
+	if (i == DLB2_NUM_SN_GROUPS) {
+		printf("[%s()] No groups with %d sequence_numbers are available or have free slots\n",
+		       __func__, sequence_numbers);
+		return;
+	}
+
+	/* Attempt to configure slot i with the requested number of sequence
+	 * numbers. Ignore the return value -- if this fails, the error will be
+	 * caught during subsequent queue configuration.
+	 */
+	dlb2_set_sn_allocation(dlb2, i, sequence_numbers);
+}
+
+static int32_t
+dlb2_hw_create_ldb_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 const struct rte_event_queue_conf *evq_conf)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_queue *queue = &ev_queue->qm_queue;
+	struct dlb2_create_ldb_queue_args cfg;
+	int32_t ret;
+	uint32_t qm_qid;
+	int sched_type = -1;
+
+	if (evq_conf == NULL)
+		return -EINVAL;
+
+	if (evq_conf->event_queue_cfg & RTE_EVENT_QUEUE_CFG_ALL_TYPES) {
+		if (evq_conf->nb_atomic_order_sequences != 0)
+			sched_type = RTE_SCHED_TYPE_ORDERED;
+		else
+			sched_type = RTE_SCHED_TYPE_PARALLEL;
+	} else {
+		sched_type = evq_conf->schedule_type;
+	}
+
+	cfg.num_atomic_inflights = DLB2_NUM_ATOMIC_INFLIGHTS_PER_QUEUE;
+	cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences;
+	cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences;
+
+	if (sched_type != RTE_SCHED_TYPE_ORDERED) {
+		cfg.num_sequence_numbers = 0;
+		cfg.num_qid_inflights = 2048;
+	}
+
+	/* App should set this to the number of hardware flows they want, not
+	 * the overall number of flows they're going to use. E.g. if app is
+	 * using 64 flows and sets compression to 64, best-case they'll get
+	 * 64 unique hashed flows in hardware.
+	 */
+	switch (evq_conf->nb_atomic_flows) {
+	/* Valid DLB2 compression levels */
+	case 64:
+	case 128:
+	case 256:
+	case 512:
+	case (1 * 1024): /* 1K */
+	case (2 * 1024): /* 2K */
+	case (4 * 1024): /* 4K */
+	case (64 * 1024): /* 64K */
+		cfg.lock_id_comp_level = evq_conf->nb_atomic_flows;
+		break;
+	default:
+		/* Invalid compression level */
+		cfg.lock_id_comp_level = 0; /* no compression */
+	}
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else {
+		cfg.depth_threshold = ev_queue->depth_threshold;
+	}
+
+	ret = dlb2_iface_ldb_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create LB event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	qm_qid = cfg.response.id;
+
+	/* Save off queue config for debug, resource lookups, and reconfig */
+	queue->num_qid_inflights = cfg.num_qid_inflights;
+	queue->num_atm_inflights = cfg.num_atomic_inflights;
+
+	queue->sched_type = sched_type;
+	queue->config_state = DLB2_CONFIGURED;
+
+	DLB2_LOG_DBG("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n",
+		     qm_qid,
+		     cfg.num_atomic_inflights,
+		     cfg.num_sequence_numbers,
+		     cfg.num_qid_inflights);
+
+	return qm_qid;
+}
+
+static int
+dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int32_t qm_qid;
+
+	if (queue_conf->nb_atomic_order_sequences)
+		dlb2_program_sn_allocation(dlb2, queue_conf);
+
+	qm_qid = dlb2_hw_create_ldb_queue(dlb2,
+					  ev_queue,
+					  queue_conf);
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the load-balanced queue\n");
+
+		return qm_qid;
+	}
+
+	dlb2->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int dlb2_num_dir_queues_setup(struct dlb2_eventdev *dlb2)
+{
+	int i, num = 0;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].setup_done &&
+		    dlb2->ev_queues[i].qm_queue.is_directed)
+			num++;
+	}
+
+	return num;
+}
+
+static void
+dlb2_queue_link_teardown(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue)
+{
+	struct dlb2_eventdev_port *ev_port;
+	int i, j;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			if (!ev_port->link[j].valid ||
+			    ev_port->link[j].queue_id != ev_queue->id)
+				continue;
+
+			ev_port->link[j].valid = false;
+			ev_port->num_links--;
+		}
+	}
+
+	ev_queue->num_links = 0;
+}
+
+static int
+dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
+			  uint8_t ev_qid,
+			  const struct rte_event_queue_conf *queue_conf)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_queue *ev_queue;
+	int ret;
+
+	if (!queue_conf)
+		return -EINVAL;
+
+	if (ev_qid >= dlb2->num_queues)
+		return -EINVAL;
+
+	ev_queue = &dlb2->ev_queues[ev_qid];
+
+	ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg &
+		RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+	ev_queue->id = ev_qid;
+	ev_queue->conf = *queue_conf;
+
+	if (!ev_queue->qm_queue.is_directed) {
+		ret = dlb2_eventdev_ldb_queue_setup(dev, ev_queue, queue_conf);
+	} else {
+		/* The directed queue isn't setup until link time, at which
+		 * point we know its directed port ID. Directed queue setup
+		 * will only fail if this queue is already setup or there are
+		 * no directed queues left to configure.
+		 */
+		ret = 0;
+
+		ev_queue->qm_queue.config_state = DLB2_NOT_CONFIGURED;
+
+		if (ev_queue->setup_done ||
+		    dlb2_num_dir_queues_setup(dlb2) == dlb2->num_dir_queues)
+			ret = -EINVAL;
+	}
+
+	/* Tear down pre-existing port->queue links */
+	if (!ret && dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_queue_link_teardown(dlb2, ev_queue);
+
+	if (!ret)
+		ev_queue->setup_done = true;
+
+	return ret;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -736,6 +1051,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
+		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 5c11736..f50a918 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -45,3 +45,15 @@ int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 				struct dlb2_create_sched_domain_args *args);
 
 void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
+
+int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_ldb_queue_args *cfg);
+
+int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_get_sn_allocation_args *args);
+
+int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				    struct dlb2_set_sn_allocation_args *args);
+
+int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_occupancy_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 576c1c3..c1ef7c2 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -31,4 +31,16 @@ extern int (*dlb2_iface_sched_domain_create)(struct dlb2_hw_dev *handle,
 
 extern void (*dlb2_iface_domain_reset)(struct dlb2_eventdev *dlb2);
 
+extern int (*dlb2_iface_ldb_queue_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_queue_args *cfg);
+
+extern int (*dlb2_iface_get_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_get_sn_allocation_args *args);
+
+extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
+				   struct dlb2_set_sn_allocation_args *args);
+
+extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
+				  struct dlb2_get_sn_occupancy_args *args);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index f83f8a1..dc9d19a 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3506,3 +3506,467 @@ unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
 
 	return num;
 }
+
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_ldb_queue *queue,
+				     struct dlb2_create_ldb_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+	union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+	union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+	union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+	union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+	union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+	union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+	union dlb2_sys_ldb_qid_its r7 = { {0} };
+	union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+	union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+	union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+	union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+	union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+	union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+	struct dlb2_sn_group *sn_group;
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r3.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	r4.field.limit = args->num_qid_inflights;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+	r5.field.limit = queue->aqed_limit;
+
+	if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+		r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+		    r5.val);
+
+	switch (args->lock_id_comp_level) {
+	case 64:
+		r6.field.compress_code = 1;
+		break;
+	case 128:
+		r6.field.compress_code = 2;
+		break;
+	case 256:
+		r6.field.compress_code = 3;
+		break;
+	case 512:
+		r6.field.compress_code = 4;
+		break;
+	case 1024:
+		r6.field.compress_code = 5;
+		break;
+	case 2048:
+		r6.field.compress_code = 6;
+		break;
+	case 4096:
+		r6.field.compress_code = 7;
+		break;
+	case 0:
+	case 65536:
+		r6.field.compress_code = 0;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+		    r6.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r7.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+		    r7.val);
+
+	r8.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+		    r8.val);
+
+	r9.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+		    r9.val);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	r10.field.qid_fid_limit = 512;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+		    r10.val);
+
+	/* Configure SNs */
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	r11.field.mode = sn_group->mode;
+	r11.field.slot = queue->sn_slot;
+	r11.field.grp  = sn_group->id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+	r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+	r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		r0.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+		r1.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+		r2.field.vqid = queue->id.virt_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+			    r2.val);
+	}
+
+	r13.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+				  struct dlb2_ldb_queue *queue,
+				  struct dlb2_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb2_sn_group_full(group)) {
+			slot = dlb2_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no sequence number slots available\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+				struct dlb2_hw_domain *domain,
+				struct dlb2_ldb_queue *queue,
+				struct dlb2_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_ldb_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (dlb2_list_empty(&domain->avail_ldb_queues)) {
+		resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb2_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_ldb_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		    args->num_sequence_numbers);
+	DLB2_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		    args->num_qid_inflights);
+	DLB2_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		    args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_ldb_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	int ret;
+
+	dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (!domain) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+	if (!queue) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret < 0) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			    __func__, __LINE__);
+		return ret;
+	}
+
+	dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_queues, &queue->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_queues, &queue->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+					     unsigned int group_id)
+{
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+						unsigned int group_id,
+						unsigned long val)
+{
+	DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+	DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+	DLB2_HW_DBG(hw, "\tValue:    %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+				    unsigned int group_id,
+				    unsigned long val)
+{
+	u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+	union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+	struct dlb2_sn_group *group;
+	int mode;
+
+	if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+		return -EINVAL;
+
+	group = &hw->rsrcs.sn_groups[group_id];
+
+	/*
+	 * Once the first load-balanced queue using an SN group is configured,
+	 * the group cannot be changed.
+	 */
+	if (group->slot_use_bitmap != 0)
+		return -EPERM;
+
+	for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+		if (val == valid_allocations[mode])
+			break;
+
+	if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+		return -EINVAL;
+
+	group->mode = mode;
+	group->sequence_numbers_per_queue = val;
+
+	r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+	r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+	DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+	dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index ca1ad69..b8e32ab 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -632,3 +632,13 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
 {
 	return dlb2_reset_domain(hw, id, NOT_VF_REQ, PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_ldb_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 21f28a4..dea70e6 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -156,6 +156,84 @@ dlb2_pf_domain_reset(struct dlb2_eventdev *dlb2)
 		DLB2_LOG_ERR("dlb2_pf_reset_domain err %d", ret);
 }
 
+static int
+dlb2_pf_ldb_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_ldb_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_ldb_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw_dev *handle,
+			 struct dlb2_get_sn_occupancy_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_number_occupancy(&dlb2_dev->hw,
+						       args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_get_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_get_group_sequence_numbers(&dlb2_dev->hw, args->group);
+
+	response.id = ret;
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
+			  struct dlb2_set_sn_allocation_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	ret = dlb2_set_group_sequence_numbers(&dlb2_dev->hw, args->group,
+					      args->num);
+
+	response.status = 0;
+
+	args->response = response;
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -168,6 +246,10 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_num_resources = dlb2_pf_get_num_resources;
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
+	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
+	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
+	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
 }
 
 /* PCI DEV HOOKS */
-- 
2.6.4

Re: [dpdk-dev] [PATCH 10/22] event/dlb2: add queue setup

[Eads, Gage]

> +static int
> +dlb2_eventdev_ldb_queue_setup(struct rte_eventdev *dev,
> +			      struct dlb2_eventdev_queue *ev_queue,
> +			      const struct rte_event_queue_conf *queue_conf)
> +{
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	int32_t qm_qid;
> +
> +	if (queue_conf->nb_atomic_order_sequences)
> +		dlb2_program_sn_allocation(dlb2, queue_conf);
> +
> +	qm_qid = dlb2_hw_create_ldb_queue(dlb2,
> +					  ev_queue,
> +					  queue_conf);

Nit: this can fit on one line

Thanks,
Gage

[dpdk-dev] [PATCH 11/22] event/dlb2: add port setup

[Timothy McDaniel]

Configure the load balanded (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 527 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   9 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 921 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  28 +
 drivers/event/dlb2/pf/dlb2_pf.c            | 179 ++++++
 6 files changed, 1672 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 366e194..a4c8833 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1043,6 +1043,532 @@ dlb2_eventdev_queue_setup(struct rte_eventdev *dev,
 	return ret;
 }
 
+static int
+dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_cq_pop_qe *qe;
+
+	qe = rte_malloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_cq_pop_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL)	{
+		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->consume_qe = qe;
+
+	memset(qe, 0, DLB2_NUM_QES_PER_CACHE_LINE *
+	       sizeof(struct dlb2_cq_pop_qe));
+
+	qe->qe_valid = 0;
+	qe->qe_frag = 0;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
+	 * and so on.
+	 */
+	qe->tokens = 0;	/* set at run time */
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
+{
+	struct dlb2_enqueue_qe *qe;
+
+	qe = rte_malloc(mz_name,
+			DLB2_NUM_QES_PER_CACHE_LINE *
+				sizeof(struct dlb2_enqueue_qe),
+			RTE_CACHE_LINE_SIZE);
+
+	if (qe == NULL) {
+		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
+		return -ENOMEM;
+	}
+	qm_port->int_arm_qe = qe;
+
+	memset(qe, 0, DLB2_NUM_QES_PER_CACHE_LINE *
+	       sizeof(struct dlb2_enqueue_qe));
+
+	/* V2 - INT ARM is CQ_TOKEN + FRAG */
+	qe->qe_valid = 0;
+	qe->qe_frag = 1;
+	qe->qe_comp = 0;
+	qe->cq_token = 1;
+	qe->meas_lat = 0;
+	qe->no_dec = 0;
+	/* Completion IDs are disabled */
+	qe->cmp_id = 0;
+
+	return 0;
+}
+
+static int
+dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
+{
+	int ret, sz;
+
+	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct dlb2_enqueue_qe);
+
+	qm_port->qe4 = rte_malloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
+
+	if (qm_port->qe4 == NULL) {
+		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
+		ret = -ENOMEM;
+		goto error_exit;
+	}
+
+	memset(qm_port->qe4, 0, sz);
+
+	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n",
+			     ret);
+		goto error_exit;
+	}
+
+	ret = dlb2_init_consume_qe(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n",
+			     ret);
+		goto error_exit;
+	}
+
+	return 0;
+
+error_exit:
+
+	dlb2_free_qe_mem(qm_port);
+
+	return ret;
+}
+
+static int
+dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_ldb_port_args cfg = {0};
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH ||
+	    dequeue_depth > DLB2_MAX_INPUT_QUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least %d\n",
+			     DLB2_MIN_ENQUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* TODO - additional parameter validation */
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	cfg.cq_history_list_size = DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
+
+	if (handle->cos_id == DLB2_COS_DEFAULT)
+		cfg.cos_id = 0;
+	else
+		cfg.cos_id = handle->cos_id;
+
+	cfg.cos_strict = 0;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* If there are no directed ports, the kernel driver will ignore this
+	 * port's directed credit settings. Don't use enqueue_depth if it would
+	 * require more directed credits than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	/* DEBUG
+	 * DLB2_LOG_ERR("create ldb port - grp=%d, devId=%d\n",
+	 * handle->cfg.domain_id, handle->device_id);
+	 */
+
+	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2; /* back ptr */
+	/*
+	 * Allocate and init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "%s_ldb_port%d",
+		 handle->device_name,
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+	qm_port->token_pop_thresh = dequeue_depth;
+
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(cfg));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     qm_port->ldb_credits,
+		     qm_port->dir_credits);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
+
+	return ret;
+}
+
+static void
+dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port)
+{
+	struct dlb2_eventdev_queue *ev_queue;
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (!ev_port->link[i].valid)
+			continue;
+
+		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
+
+		ev_port->link[i].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+}
+
+static int
+dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
+			struct dlb2_eventdev_port *ev_port,
+			uint32_t dequeue_depth,
+			uint32_t enqueue_depth)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_port_args cfg = {0};
+	int ret;
+	struct dlb2_port *qm_port = NULL;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t qm_port_id;
+	uint16_t ldb_credit_high_watermark;
+	uint16_t dir_credit_high_watermark;
+
+	if (dlb2 == NULL || handle == NULL)
+		return -EINVAL;
+
+	if (dequeue_depth < DLB2_MIN_CQ_DEPTH ||
+	    dequeue_depth > DLB2_MAX_INPUT_QUEUE_DEPTH) {
+		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-%d\n",
+			     DLB2_MIN_CQ_DEPTH, DLB2_MAX_INPUT_QUEUE_DEPTH);
+		return -EINVAL;
+	}
+
+	rte_spinlock_lock(&handle->resource_lock);
+
+	/* Directed queues are configured at link time. */
+	cfg.queue_id = -1;
+
+	/* We round up to the next power of 2 if necessary */
+	cfg.cq_depth = rte_align32pow2(dequeue_depth);
+	cfg.cq_depth_threshold = 1;
+
+	/* User controls the LDB high watermark via enqueue depth. The DIR high
+	 * watermark is equal, unless the directed credit pool is too small.
+	 */
+	ldb_credit_high_watermark = enqueue_depth;
+
+	/* Don't use enqueue_depth if it would require more directed credits
+	 * than are available.
+	 */
+	dir_credit_high_watermark =
+		RTE_MIN(enqueue_depth,
+			handle->cfg.num_dir_credits / dlb2->num_ports);
+
+	/* Per QM values */
+
+	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		goto error_exit;
+	}
+
+	qm_port_id = cfg.response.id;
+
+	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
+		     ev_port->id, qm_port_id);
+
+	qm_port = &ev_port->qm_port;
+	qm_port->ev_port = ev_port; /* back ptr */
+	qm_port->dlb2 = dlb2;  /* back ptr */
+
+	/*
+	 * Init local qe struct(s).
+	 * Note: MOVDIR64 requires the enqueue QE to be aligned
+	 */
+
+	snprintf(mz_name, sizeof(mz_name), "%s_dir_port%d",
+		 handle->device_name,
+		 ev_port->id);
+
+	ret = dlb2_init_qe_mem(qm_port, mz_name);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
+		goto error_exit;
+	}
+
+	qm_port->id = qm_port_id;
+
+	qm_port->cached_ldb_credits = 0;
+	qm_port->cached_dir_credits = 0;
+	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
+	 * the effective depth is smaller.
+	 */
+	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
+	qm_port->cq_idx = 0;
+	qm_port->cq_idx_unmasked = 0;
+
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
+		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
+	else
+		qm_port->cq_depth_mask = cfg.cq_depth - 1;
+
+	qm_port->gen_bit_shift = __builtin_popcount(qm_port->cq_depth_mask);
+	/* starting value of gen bit - it toggles at wrap time */
+	qm_port->gen_bit = 1;
+
+	qm_port->int_armed = false;
+
+	/* Save off for later use in info and lookup APIs. */
+	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
+
+	qm_port->dequeue_depth = dequeue_depth;
+
+	/* Directed ports are auto-pop, by default. */
+	qm_port->token_pop_mode = AUTO_POP;
+	qm_port->owed_tokens = 0;
+	qm_port->issued_releases = 0;
+
+	/* Save config message too. */
+	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(cfg));
+
+	/* update state */
+	qm_port->state = PORT_STARTED; /* enabled at create time */
+	qm_port->config_state = DLB2_CONFIGURED;
+
+	qm_port->dir_credits = dir_credit_high_watermark;
+	qm_port->ldb_credits = ldb_credit_high_watermark;
+	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
+	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
+
+	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
+		     qm_port_id,
+		     dequeue_depth,
+		     dir_credit_high_watermark,
+		     ldb_credit_high_watermark);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	return 0;
+
+error_exit:
+
+	if (qm_port)
+		dlb2_free_qe_mem(qm_port);
+
+	rte_spinlock_unlock(&handle->resource_lock);
+
+	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_setup(struct rte_eventdev *dev,
+			 uint8_t ev_port_id,
+			 const struct rte_event_port_conf *port_conf)
+{
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_eventdev_port *ev_port;
+	int ret;
+
+	if (dev == NULL || port_conf == NULL) {
+		DLB2_LOG_ERR("Null parameter\n");
+		return -EINVAL;
+	}
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
+		return -EINVAL;
+
+	if (port_conf->dequeue_depth >
+		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
+	    port_conf->enqueue_depth >
+		evdev_dlb2_default_info.max_event_port_enqueue_depth)
+		return -EINVAL;
+
+	ev_port = &dlb2->ev_ports[ev_port_id];
+	/* configured? */
+	if (ev_port->setup_done) {
+		DLB2_LOG_ERR("evport %d is already configured\n", ev_port_id);
+		return -EINVAL;
+	}
+
+	/* The reserved token interrupt arming scheme requires that one or more
+	 * CQ tokens be reserved by the PMD. This limits the amount of CQ space
+	 * usable by the DLB, so in order to give an *effective* CQ depth equal
+	 * to the user-requested value, we double CQ depth and reserve half of
+	 * its tokens. If the user requests the max CQ depth (256) then we
+	 * cannot double it, so we reserve one token and give an effective
+	 * depth of 255 entries.
+	 */
+
+	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
+		RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (!ev_port->qm_port.is_directed) {
+		ret = dlb2_hw_create_ldb_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the lB port ve portId=%d\n",
+				     ev_port_id);
+
+			return ret;
+		}
+	} else {
+		ret = dlb2_hw_create_dir_port(dlb2,
+					      ev_port,
+					      port_conf->dequeue_depth,
+					      port_conf->enqueue_depth);
+		if (ret < 0) {
+			DLB2_LOG_ERR("Failed to create the DIR port\n");
+			return ret;
+		}
+	}
+
+	/* Save off port config for reconfig */
+	dlb2->ev_ports[ev_port_id].conf = *port_conf;
+
+	dlb2->ev_ports[ev_port_id].id = ev_port_id;
+	dlb2->ev_ports[ev_port_id].enq_configured = true;
+	dlb2->ev_ports[ev_port_id].setup_done = true;
+	dlb2->ev_ports[ev_port_id].inflight_max =
+		port_conf->new_event_threshold;
+	dlb2->ev_ports[ev_port_id].implicit_release =
+		!(port_conf->event_port_cfg &
+		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
+	dlb2->ev_ports[ev_port_id].outstanding_releases = 0;
+	dlb2->ev_ports[ev_port_id].inflight_credits = 0;
+	dlb2->ev_ports[ev_port_id].credit_update_quanta =
+		RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
+	dlb2->ev_ports[ev_port_id].dlb2 = dlb2; /* reverse link */
+
+	/* Tear down pre-existing port->queue links */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
+
+	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1053,6 +1579,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
+		.port_setup       = dlb2_eventdev_port_setup,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index f50a918..84cbf25 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -57,3 +57,12 @@ int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				   struct dlb2_get_sn_occupancy_args *args);
+
+int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_ldb_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
+int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+				  struct dlb2_create_dir_port_args *cfg,
+				  enum dlb2_cq_poll_modes poll_mode);
+
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index c1ef7c2..a090a54 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -43,4 +43,12 @@ extern int (*dlb2_iface_set_sn_allocation)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_get_sn_occupancy)(struct dlb2_hw_dev *handle,
 				  struct dlb2_get_sn_occupancy_args *args);
 
+extern int (*dlb2_iface_ldb_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_ldb_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
+extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
+					 struct dlb2_create_dir_port_args *cfg,
+					 enum dlb2_cq_poll_modes poll_mode);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index dc9d19a..bb8683c 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -3970,3 +3970,924 @@ int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_ldb_port *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_pp2vas r0 = { {0} };
+	union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+		union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_ldb_port *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_ldb_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+	union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+	union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+	union dlb2_chp_hist_list_lim r5 = { {0} };
+	union dlb2_chp_hist_list_base r6 = { {0} };
+	union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+	union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+	union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+	union dlb2_sys_ldb_cq_at r10 = { {0} };
+	union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+	union dlb2_chp_ldb_cq2vas r12 = { {0} };
+	union dlb2_lsp_cq2priov r13 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r14.field.token_count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    r14.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_LDB_CQ_WPTR_RST);
+
+	r5.field.limit = port->hist_list_entry_limit - 1;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+	r6.field.base = port->hist_list_entry_base;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	r7.field.limit = args->cq_history_list_size;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+	r8.field.push_ptr = r6.field.base;
+	r8.field.generation = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		    r8.val);
+
+	r9.field.pop_ptr = r6.field.base;
+	r9.field.generation = 0;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+	/* Disable the port's QID mappings */
+	r13.field.v = 0;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+	return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_ldb_port *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_ldb_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb2_ldb_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+	if (ret < 0)
+		return ret;
+
+	dlb2_ldb_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_ldb_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		    args->cq_history_list_size);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+	DLB2_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB2_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		    args->cos_strict);
+}
+
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_ldb_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	int i;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+		resp->status = DLB2_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		if (dlb2_list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	} else {
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			if (!dlb2_list_empty(&domain->avail_ldb_ports[i]))
+				break;
+		}
+
+		if (i == DLB2_NUM_COS_DOMAINS) {
+			resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ *	its resources.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_ldb_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	int ret, cos_id, i;
+
+	dlb2_log_create_ldb_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_ldb_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (!domain) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->cos_strict) {
+		cos_id = args->cos_id;
+
+		port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+					  typeof(*port));
+	} else {
+		int idx;
+
+		for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+			idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+			port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+						  typeof(*port));
+			if (port)
+				break;
+		}
+
+		cos_id = idx;
+	}
+
+	if (!port) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available ldb ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (port->configured) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_ldb_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	dlb2_list_del(&domain->avail_ldb_ports[cos_id], &port->domain_list);
+
+	dlb2_list_add(&domain->used_ldb_ports[cos_id], &port->domain_list);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+			      u32 domain_id,
+			      uintptr_t cq_dma_base,
+			      struct dlb2_create_dir_port_args *args,
+			      bool vdev_req,
+			      unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		    args->cq_depth);
+	DLB2_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		    cq_dma_base);
+}
+
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+			    bool vdev_req,
+			    struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *port;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port, iter)
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+
+	return NULL;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+				 u32 domain_id,
+				 uintptr_t cq_dma_base,
+				 struct dlb2_create_dir_port_args *args,
+				 struct dlb2_cmd_response *resp,
+				 bool vdev_req,
+				 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the queue is already configured, validate
+	 * the queue ID, its domain, and whether the queue is configured.
+	 */
+	if (args->queue_id != -1) {
+		struct dlb2_dir_pq_pair *queue;
+
+		queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+						    vdev_req,
+						    domain);
+
+		if (!queue || queue->domain_id.phys_id != domain->id.phys_id ||
+		    !queue->queue_configured) {
+			resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the port's queue is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->queue_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* Check cache-line alignment */
+	if ((cq_dma_base & 0x3F) != 0) {
+		resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (args->cq_depth != 1 &&
+	    args->cq_depth != 2 &&
+	    args->cq_depth != 4 &&
+	    args->cq_depth != 8 &&
+	    args->cq_depth != 16 &&
+	    args->cq_depth != 32 &&
+	    args->cq_depth != 64 &&
+	    args->cq_depth != 128 &&
+	    args->cq_depth != 256 &&
+	    args->cq_depth != 512 &&
+	    args->cq_depth != 1024) {
+		resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+				       struct dlb2_hw_domain *domain,
+				       struct dlb2_dir_pq_pair *port,
+				       bool vdev_req,
+				       unsigned int vdev_id)
+{
+	union dlb2_sys_dir_pp2vas r0 = { {0} };
+	union dlb2_sys_dir_pp_v r4 = { {0} };
+
+	r0.field.vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+		union dlb2_sys_dir_pp2vdev r2 = { {0} };
+		union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+		unsigned int offs;
+		u32 virt_id;
+
+		/*
+		 * DLB uses producer port address bits 17:12 to determine the
+		 * producer port ID. In Scalable IOV mode, PP accesses come
+		 * through the PF MMIO window for the physical producer port,
+		 * so for translation purposes the virtual and physical port
+		 * IDs are equal.
+		 */
+		if (hw->virt_mode == DLB2_VIRT_SRIOV)
+			virt_id = port->id.virt_id;
+		else
+			virt_id = port->id.phys_id;
+
+		r1.field.pp = port->id.phys_id;
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+		r2.field.vdev = vdev_id;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+			    r2.val);
+
+		r3.field.vpp_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+	}
+
+	r4.field.pp_v = 1;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_PP_V(port->id.phys_id),
+		    r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+				      struct dlb2_hw_domain *domain,
+				      struct dlb2_dir_pq_pair *port,
+				      uintptr_t cq_dma_base,
+				      struct dlb2_create_dir_port_args *args,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+	union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+	union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+	union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+	union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+	union dlb2_sys_dir_cq_fmt r9 = { {0} };
+	union dlb2_sys_dir_cq_at r10 = { {0} };
+	union dlb2_sys_dir_cq_pasid r11 = { {0} };
+	union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	r0.field.addr_l = cq_dma_base >> 6;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+	r1.field.addr_u = cq_dma_base >> 32;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB2 to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	r2.field.vf = vdev_id;
+	r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+	r2.field.ro = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+	if (args->cq_depth <= 8) {
+		r3.field.token_depth_select = 1;
+	} else if (args->cq_depth == 16) {
+		r3.field.token_depth_select = 2;
+	} else if (args->cq_depth == 32) {
+		r3.field.token_depth_select = 3;
+	} else if (args->cq_depth == 64) {
+		r3.field.token_depth_select = 4;
+	} else if (args->cq_depth == 128) {
+		r3.field.token_depth_select = 5;
+	} else if (args->cq_depth == 256) {
+		r3.field.token_depth_select = 6;
+	} else if (args->cq_depth == 512) {
+		r3.field.token_depth_select = 7;
+	} else if (args->cq_depth == 1024) {
+		r3.field.token_depth_select = 8;
+	} else {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: invalid CQ depth\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		    r3.val);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		r13.field.count = port->init_tkn_cnt;
+
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    r13.val);
+	} else {
+		DLB2_CSR_WR(hw,
+			    DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			    DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	r4.field.token_depth_select = r3.field.token_depth_select;
+	r4.field.disable_wb_opt = 0;
+	r4.field.ignore_depth = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		    r4.val);
+
+	/* Reset the CQ write pointer */
+	DLB2_CSR_WR(hw,
+		    DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+		    DLB2_CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	r9.field.keep_pf_ppid = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	r10.field.cq_at = 0;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+	if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+		r11.field.pasid = hw->pasid[vdev_id];
+		r11.field.fmt2 = 1;
+	}
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+		    r11.val);
+
+	r12.field.cq2vas = domain->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+	return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+				   struct dlb2_hw_domain *domain,
+				   struct dlb2_dir_pq_pair *port,
+				   uintptr_t cq_dma_base,
+				   struct dlb2_create_dir_port_args *args,
+				   bool vdev_req,
+				   unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb2_dir_port_configure_cq(hw,
+					 domain,
+					 port,
+					 cq_dma_base,
+					 args,
+					 vdev_req,
+					 vdev_id);
+
+	if (ret < 0)
+		return ret;
+
+	dlb2_dir_port_configure_pp(hw,
+				   domain,
+				   port,
+				   vdev_req,
+				   vdev_id);
+
+	dlb2_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ *	and queue. The port/queue pair have the same ID and name.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+			    u32 domain_id,
+			    struct dlb2_create_dir_port_args *args,
+			    uintptr_t cq_dma_base,
+			    struct dlb2_cmd_response *resp,
+			    bool vdev_req,
+			    unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *port;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_port_args(hw,
+				      domain_id,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_port_args(hw,
+					       domain_id,
+					       cq_dma_base,
+					       args,
+					       resp,
+					       vdev_req,
+					       vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (args->queue_id != -1)
+		port = dlb2_get_domain_used_dir_pq(args->queue_id,
+						   vdev_req,
+						   domain);
+	else
+		port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					  typeof(*port));
+	if (!port) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir ports\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	ret = dlb2_configure_dir_port(hw,
+				      domain,
+				      port,
+				      cq_dma_base,
+				      args,
+				      vdev_req,
+				      vdev_id);
+	if (ret < 0)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs, &port->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs, &port->domain_list);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index b8e32ab..582ef53 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -642,3 +642,31 @@ dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_ldb_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_ldb_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_ldb_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
+
+int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+			u32 id,
+			struct dlb2_create_dir_port_args *args,
+			uintptr_t cq_dma_base,
+			struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_port(hw, id, args,
+				       cq_dma_base,
+				       resp,
+				       NOT_VF_REQ,
+				       PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index dea70e6..a6824b1 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -234,6 +234,183 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static void *
+dlb2_alloc_coherent_aligned(uintptr_t *phys, size_t size, int align)
+{
+	const struct rte_memzone *mz;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	uint32_t core_id = rte_lcore_id();
+	unsigned int socket_id;
+
+	snprintf(mz_name, sizeof(mz_name) - 1, "%lx",
+		 (unsigned long)rte_get_timer_cycles());
+	if (core_id == (unsigned int)LCORE_ID_ANY)
+		core_id = rte_get_master_lcore();
+	socket_id = rte_lcore_to_socket_id(core_id);
+	mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
+					 RTE_MEMZONE_IOVA_CONTIG, align);
+	if (!mz) {
+		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu bytes - %s\n",
+			     size, rte_strerror(rte_errno));
+		*phys = 0;
+		return NULL;
+	}
+	*phys = mz->iova;
+	return mz->addr;
+}
+
+static int
+dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_ldb_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret, cq_alloc_depth;
+	uint8_t *port_base;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = false;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	cq_alloc_depth = RTE_MAX(cfg->cq_depth, DLB2_MIN_HARDWARE_CQ_DEPTH);
+	alloc_sz = cq_alloc_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&cq_base,
+						alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
+		(void *)(uintptr_t)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_LDB_PORT].cq_base =
+		(void *)(uintptr_t)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+	dlb2_list_init_head(&port_memory.list);
+
+	/* Fill out the per-port memory tracking structure */
+	dlb2_dev->ldb_port_pages[response.id].valid = true;
+	dlb2_list_splice(&port_memory.list,
+			 &dlb2_dev->ldb_port_pages[response.id].list);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+create_port_err:
+
+	return ret;
+}
+
+static int
+dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
+			struct dlb2_create_dir_port_args *cfg,
+			enum dlb2_cq_poll_modes poll_mode)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	struct dlb2_port_memory port_memory;
+	int ret;
+	uint8_t *port_base;
+	int alloc_sz, qe_sz;
+	phys_addr_t cq_base;
+	phys_addr_t pp_base;
+	int is_dir = true;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
+		qe_sz = sizeof(struct dlb2_dequeue_qe);
+	else
+		qe_sz = RTE_CACHE_LINE_SIZE;
+
+	/* Calculate the port memory required, and round up to the nearest
+	 * cache line.
+	 */
+	alloc_sz = cfg->cq_depth * qe_sz;
+	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
+
+	port_base = dlb2_alloc_coherent_aligned(&cq_base,
+						alloc_sz,
+						PAGE_SIZE);
+	if (port_base == NULL)
+		return -ENOMEM;
+
+	/* Lock the page in memory */
+	ret = rte_mem_lock_page(port_base);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device i/o\n");
+		goto create_port_err;
+	}
+
+	memset(port_base, 0, alloc_sz);
+
+	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
+				      handle->domain_id,
+				      cfg,
+				      cq_base,
+				      &response);
+	if (ret)
+		goto create_port_err;
+
+	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
+	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
+		(void *)(uintptr_t)(pp_base + (PAGE_SIZE * response.id));
+
+	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
+		(void *)(uintptr_t)(port_base);
+	memset(&port_memory, 0, sizeof(port_memory));
+	dlb2_list_init_head(&port_memory.list);
+
+	/* Fill out the per-port memory tracking structure */
+	dlb2_dev->dir_port_pages[response.id].valid = true;
+	dlb2_list_splice(&port_memory.list,
+			 &dlb2_dev->dir_port_pages[response.id].list);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+create_port_err:
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -247,6 +424,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
+	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

Re: [dpdk-dev] [PATCH 11/22] event/dlb2: add port setup

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 11/22] event/dlb2: add port setup
> 
> Configure the load balanded (ldb) or directed (dir) port.
> The consumer queue (CQ) and producer port (PP) are also
> set up here.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  drivers/event/dlb2/dlb2.c                  | 527 +++++++++++++++++
>  drivers/event/dlb2/dlb2_iface.c            |   9 +
>  drivers/event/dlb2/dlb2_iface.h            |   8 +
>  drivers/event/dlb2/pf/base/dlb2_resource.c | 921
> +++++++++++++++++++++++++++++
>  drivers/event/dlb2/pf/dlb2_main.c          |  28 +
>  drivers/event/dlb2/pf/dlb2_pf.c            | 179 ++++++
>  6 files changed, 1672 insertions(+)
> 
> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> index 366e194..a4c8833 100644
> --- a/drivers/event/dlb2/dlb2.c
> +++ b/drivers/event/dlb2/dlb2.c
> @@ -1043,6 +1043,532 @@ dlb2_eventdev_queue_setup(struct rte_eventdev
> *dev,
>  	return ret;
>  }
> 
> +static int
> +dlb2_init_consume_qe(struct dlb2_port *qm_port, char *mz_name)
> +{
> +	struct dlb2_cq_pop_qe *qe;
> +
> +	qe = rte_malloc(mz_name,
> +			DLB2_NUM_QES_PER_CACHE_LINE *
> +				sizeof(struct dlb2_cq_pop_qe),
> +			RTE_CACHE_LINE_SIZE);
> +
> +	if (qe == NULL)	{
> +		DLB2_LOG_ERR("dlb2: no memory for consume_qe\n");
> +		return -ENOMEM;
> +	}
> +	qm_port->consume_qe = qe;
> +
> +	memset(qe, 0, DLB2_NUM_QES_PER_CACHE_LINE *
> +	       sizeof(struct dlb2_cq_pop_qe));

You can use rte_zmalloc() instead (applies to other init_*_qe functions below too), and
no need to zero-init the fields again after the memset.

> +
> +	qe->qe_valid = 0;
> +	qe->qe_frag = 0;
> +	qe->qe_comp = 0;
> +	qe->cq_token = 1;
> +	/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2,
> +	 * and so on.
> +	 */
> +	qe->tokens = 0;	/* set at run time */
> +	qe->meas_lat = 0;
> +	qe->no_dec = 0;
> +	/* Completion IDs are disabled */
> +	qe->cmp_id = 0;
> +
> +	return 0;
> +}
> +
> +static int
> +dlb2_init_int_arm_qe(struct dlb2_port *qm_port, char *mz_name)
> +{
> +	struct dlb2_enqueue_qe *qe;
> +
> +	qe = rte_malloc(mz_name,
> +			DLB2_NUM_QES_PER_CACHE_LINE *
> +				sizeof(struct dlb2_enqueue_qe),
> +			RTE_CACHE_LINE_SIZE);
> +
> +	if (qe == NULL) {
> +		DLB2_LOG_ERR("dlb2: no memory for complete_qe\n");
> +		return -ENOMEM;
> +	}
> +	qm_port->int_arm_qe = qe;
> +
> +	memset(qe, 0, DLB2_NUM_QES_PER_CACHE_LINE *
> +	       sizeof(struct dlb2_enqueue_qe));
> +
> +	/* V2 - INT ARM is CQ_TOKEN + FRAG */
> +	qe->qe_valid = 0;
> +	qe->qe_frag = 1;
> +	qe->qe_comp = 0;
> +	qe->cq_token = 1;
> +	qe->meas_lat = 0;
> +	qe->no_dec = 0;
> +	/* Completion IDs are disabled */
> +	qe->cmp_id = 0;
> +
> +	return 0;
> +}
> +
> +static int
> +dlb2_init_qe_mem(struct dlb2_port *qm_port, char *mz_name)
> +{
> +	int ret, sz;
> +
> +	sz = DLB2_NUM_QES_PER_CACHE_LINE * sizeof(struct
> dlb2_enqueue_qe);
> +
> +	qm_port->qe4 = rte_malloc(mz_name, sz, RTE_CACHE_LINE_SIZE);
> +
> +	if (qm_port->qe4 == NULL) {
> +		DLB2_LOG_ERR("dlb2: no qe4 memory\n");
> +		ret = -ENOMEM;
> +		goto error_exit;
> +	}
> +
> +	memset(qm_port->qe4, 0, sz);
> +
> +	ret = dlb2_init_int_arm_qe(qm_port, mz_name);
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2: dlb2_init_int_arm_qe ret=%d\n",
> +			     ret);

This can fit on one line

> +		goto error_exit;
> +	}
> +
> +	ret = dlb2_init_consume_qe(qm_port, mz_name);
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2: dlb2_init_consume_qe ret=%d\n",
> +			     ret);

This can fit on one line

> +		goto error_exit;
> +	}
> +
> +	return 0;
> +
> +error_exit:
> +
> +	dlb2_free_qe_mem(qm_port);
> +
> +	return ret;
> +}
> +
> +static int
> +dlb2_hw_create_ldb_port(struct dlb2_eventdev *dlb2,
> +			struct dlb2_eventdev_port *ev_port,
> +			uint32_t dequeue_depth,
> +			uint32_t enqueue_depth)
> +{
> +	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
> +	struct dlb2_create_ldb_port_args cfg = {0};
> +	int ret;
> +	struct dlb2_port *qm_port = NULL;
> +	char mz_name[RTE_MEMZONE_NAMESIZE];
> +	uint32_t qm_port_id;
> +	uint16_t ldb_credit_high_watermark;
> +	uint16_t dir_credit_high_watermark;
> +
> +	if (handle == NULL)
> +		return -EINVAL;
> +
> +	if (dequeue_depth < DLB2_MIN_CQ_DEPTH ||
> +	    dequeue_depth > DLB2_MAX_INPUT_QUEUE_DEPTH) {
> +		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-
> %d\n",
> +			     DLB2_MIN_CQ_DEPTH,
> DLB2_MAX_INPUT_QUEUE_DEPTH);
> +		return -EINVAL;
> +	}
> +
> +	if (enqueue_depth < DLB2_MIN_ENQUEUE_DEPTH) {
> +		DLB2_LOG_ERR("dlb2: invalid enqueue_depth, must be at least
> %d\n",
> +			     DLB2_MIN_ENQUEUE_DEPTH);
> +		return -EINVAL;
> +	}

The dequeue and enqueue depth checks are suspiciously inconsistent -- only the dequeue
depth is compared against an upper bound. I suspect the enqueue upper bound check is missing
because it's already checked in both dlb2_eventdev_port_setup() and
rte_event_port_setup()...if that's the case, can the dequeue depth max check be dropped as well?

> +
> +	rte_spinlock_lock(&handle->resource_lock);
> +
> +	/* TODO - additional parameter validation */

Leftover TODO

> +	/* We round up to the next power of 2 if necessary */
> +	cfg.cq_depth = rte_align32pow2(dequeue_depth);
> +	cfg.cq_depth_threshold = 1;
> +
> +	cfg.cq_history_list_size =
> DLB2_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT;
> +
> +	if (handle->cos_id == DLB2_COS_DEFAULT)
> +		cfg.cos_id = 0;
> +	else
> +		cfg.cos_id = handle->cos_id;
> +
> +	cfg.cos_strict = 0;
> +
> +	/* User controls the LDB high watermark via enqueue depth. The DIR
> high
> +	 * watermark is equal, unless the directed credit pool is too small.
> +	 */
> +	ldb_credit_high_watermark = enqueue_depth;
> +
> +	/* If there are no directed ports, the kernel driver will ignore this
> +	 * port's directed credit settings. Don't use enqueue_depth if it would
> +	 * require more directed credits than are available.
> +	 */
> +	dir_credit_high_watermark =
> +		RTE_MIN(enqueue_depth,
> +			handle->cfg.num_dir_credits / dlb2->num_ports);
> +
> +	/* Per QM values */
> +
> +	/* DEBUG
> +	 * DLB2_LOG_ERR("create ldb port - grp=%d, devId=%d\n",
> +	 * handle->cfg.domain_id, handle->device_id);
> +	 */

Leftover debug code

> +
> +	ret = dlb2_iface_ldb_port_create(handle, &cfg,  dlb2->poll_mode);
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2: dlb2_ldb_port_create error, ret=%d
> (driver status: %s)\n",
> +			     ret, dlb2_error_strings[cfg.response.status]);
> +		goto error_exit;
> +	}
> +
> +	qm_port_id = cfg.response.id;
> +
> +	DLB2_LOG_DBG("dlb2: ev_port %d uses qm LB port %d <<<<<\n",
> +		     ev_port->id, qm_port_id);
> +
> +	qm_port = &ev_port->qm_port;
> +	qm_port->ev_port = ev_port; /* back ptr */
> +	qm_port->dlb2 = dlb2; /* back ptr */
> +	/*
> +	 * Allocate and init local qe struct(s).
> +	 * Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned.
> +	 */
> +
> +	snprintf(mz_name, sizeof(mz_name), "%s_ldb_port%d",
> +		 handle->device_name,
> +		 ev_port->id);
> +

I see handle->device_name getting read here, in dlb2_hw_create_dir_port(),
and also in dlb2_eventdev_dump(), but I don't see it written anywhere?

> +	ret = dlb2_init_qe_mem(qm_port, mz_name);
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
> +		goto error_exit;
> +	}
> +
> +	qm_port->id = qm_port_id;
> +
> +	qm_port->cached_ldb_credits = 0;
> +	qm_port->cached_dir_credits = 0;
> +	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
> +	 * the effective depth is smaller.
> +	 */
> +	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
> +	qm_port->cq_idx = 0;
> +	qm_port->cq_idx_unmasked = 0;
> +
> +	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
> +		qm_port->cq_depth_mask = (qm_port->cq_depth * 4) - 1;
> +	else
> +		qm_port->cq_depth_mask = qm_port->cq_depth - 1;
> +
> +	qm_port->gen_bit_shift = __builtin_popcount(qm_port-
> >cq_depth_mask);
> +	/* starting value of gen bit - it toggles at wrap time */
> +	qm_port->gen_bit = 1;
> +
> +	qm_port->int_armed = false;
> +
> +	/* Save off for later use in info and lookup APIs. */
> +	qm_port->qid_mappings = &dlb2->qm_ldb_to_ev_queue_id[0];
> +
> +	qm_port->dequeue_depth = dequeue_depth;
> +	qm_port->token_pop_thresh = dequeue_depth;
> +
> +	qm_port->owed_tokens = 0;
> +	qm_port->issued_releases = 0;
> +
> +	/* Save config message too. */
> +	rte_memcpy(&qm_port->cfg.ldb, &cfg, sizeof(cfg));

I know qm_port->cfg.ldb and cfg are the same type, but just for safety in case that
ever changes in the future...probably better to use sizeof() on the destination rather
than the source.

> +
> +	/* update state */
> +	qm_port->state = PORT_STARTED; /* enabled at create time */
> +	qm_port->config_state = DLB2_CONFIGURED;
> +
> +	qm_port->dir_credits = dir_credit_high_watermark;
> +	qm_port->ldb_credits = ldb_credit_high_watermark;
> +	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
> +	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
> +
> +	DLB2_LOG_DBG("dlb2: created ldb port %d, depth = %d, ldb credits=%d,
> dir credits=%d\n",
> +		     qm_port_id,
> +		     dequeue_depth,
> +		     qm_port->ldb_credits,
> +		     qm_port->dir_credits);
> +
> +	rte_spinlock_unlock(&handle->resource_lock);
> +
> +	return 0;
> +
> +error_exit:
> +
> +	if (qm_port)
> +		dlb2_free_qe_mem(qm_port);
> +
> +	rte_spinlock_unlock(&handle->resource_lock);
> +
> +	DLB2_LOG_ERR("dlb2: create ldb port failed!\n");
> +
> +	return ret;
> +}
> +
> +static void
> +dlb2_port_link_teardown(struct dlb2_eventdev *dlb2,
> +			struct dlb2_eventdev_port *ev_port)
> +{
> +	struct dlb2_eventdev_queue *ev_queue;
> +	int i;
> +
> +	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
> +		if (!ev_port->link[i].valid)
> +			continue;
> +
> +		ev_queue = &dlb2->ev_queues[ev_port->link[i].queue_id];
> +
> +		ev_port->link[i].valid = false;
> +		ev_port->num_links--;
> +		ev_queue->num_links--;
> +	}
> +}
> +
> +static int
> +dlb2_hw_create_dir_port(struct dlb2_eventdev *dlb2,
> +			struct dlb2_eventdev_port *ev_port,
> +			uint32_t dequeue_depth,
> +			uint32_t enqueue_depth)
> +{
> +	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
> +	struct dlb2_create_dir_port_args cfg = {0};
> +	int ret;
> +	struct dlb2_port *qm_port = NULL;
> +	char mz_name[RTE_MEMZONE_NAMESIZE];
> +	uint32_t qm_port_id;
> +	uint16_t ldb_credit_high_watermark;
> +	uint16_t dir_credit_high_watermark;
> +
> +	if (dlb2 == NULL || handle == NULL)
> +		return -EINVAL;
> +
> +	if (dequeue_depth < DLB2_MIN_CQ_DEPTH ||
> +	    dequeue_depth > DLB2_MAX_INPUT_QUEUE_DEPTH) {
> +		DLB2_LOG_ERR("dlb2: invalid dequeue_depth, must be %d-
> %d\n",
> +			     DLB2_MIN_CQ_DEPTH,
> DLB2_MAX_INPUT_QUEUE_DEPTH);
> +		return -EINVAL;
> +	}

Enqueue depth check needed?

> +
> +	rte_spinlock_lock(&handle->resource_lock);
> +
> +	/* Directed queues are configured at link time. */
> +	cfg.queue_id = -1;
> +
> +	/* We round up to the next power of 2 if necessary */
> +	cfg.cq_depth = rte_align32pow2(dequeue_depth);
> +	cfg.cq_depth_threshold = 1;
> +
> +	/* User controls the LDB high watermark via enqueue depth. The DIR
> high
> +	 * watermark is equal, unless the directed credit pool is too small.
> +	 */
> +	ldb_credit_high_watermark = enqueue_depth;
> +
> +	/* Don't use enqueue_depth if it would require more directed credits
> +	 * than are available.
> +	 */
> +	dir_credit_high_watermark =
> +		RTE_MIN(enqueue_depth,
> +			handle->cfg.num_dir_credits / dlb2->num_ports);
> +
> +	/* Per QM values */
> +
> +	ret = dlb2_iface_dir_port_create(handle, &cfg,  dlb2->poll_mode);
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2: dlb2_dir_port_create error, ret=%d (driver
> status: %s)\n",
> +			     ret, dlb2_error_strings[cfg.response.status]);
> +		goto error_exit;
> +	}
> +
> +	qm_port_id = cfg.response.id;
> +
> +	DLB2_LOG_DBG("dlb2: ev_port %d uses qm DIR port %d <<<<<\n",
> +		     ev_port->id, qm_port_id);
> +
> +	qm_port = &ev_port->qm_port;
> +	qm_port->ev_port = ev_port; /* back ptr */
> +	qm_port->dlb2 = dlb2;  /* back ptr */
> +
> +	/*
> +	 * Init local qe struct(s).
> +	 * Note: MOVDIR64 requires the enqueue QE to be aligned
> +	 */
> +
> +	snprintf(mz_name, sizeof(mz_name), "%s_dir_port%d",
> +		 handle->device_name,

(See device_name comment above)

> +		 ev_port->id);
> +
> +	ret = dlb2_init_qe_mem(qm_port, mz_name);
> +
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2: init_qe_mem failed, ret=%d\n", ret);
> +		goto error_exit;
> +	}
> +
> +	qm_port->id = qm_port_id;
> +
> +	qm_port->cached_ldb_credits = 0;
> +	qm_port->cached_dir_credits = 0;
> +	/* CQs with depth < 8 use an 8-entry queue, but withhold credits so
> +	 * the effective depth is smaller.
> +	 */
> +	qm_port->cq_depth = cfg.cq_depth <= 8 ? 8 : cfg.cq_depth;
> +	qm_port->cq_idx = 0;
> +	qm_port->cq_idx_unmasked = 0;
> +
> +	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE)
> +		qm_port->cq_depth_mask = (cfg.cq_depth * 4) - 1;
> +	else
> +		qm_port->cq_depth_mask = cfg.cq_depth - 1;
> +
> +	qm_port->gen_bit_shift = __builtin_popcount(qm_port-
> >cq_depth_mask);
> +	/* starting value of gen bit - it toggles at wrap time */
> +	qm_port->gen_bit = 1;
> +
> +	qm_port->int_armed = false;
> +
> +	/* Save off for later use in info and lookup APIs. */
> +	qm_port->qid_mappings = &dlb2->qm_dir_to_ev_queue_id[0];
> +
> +	qm_port->dequeue_depth = dequeue_depth;
> +
> +	/* Directed ports are auto-pop, by default. */
> +	qm_port->token_pop_mode = AUTO_POP;
> +	qm_port->owed_tokens = 0;
> +	qm_port->issued_releases = 0;
> +
> +	/* Save config message too. */
> +	rte_memcpy(&qm_port->cfg.dir, &cfg, sizeof(cfg));

(See sizeof() comment above)

> +
> +	/* update state */
> +	qm_port->state = PORT_STARTED; /* enabled at create time */
> +	qm_port->config_state = DLB2_CONFIGURED;
> +
> +	qm_port->dir_credits = dir_credit_high_watermark;
> +	qm_port->ldb_credits = ldb_credit_high_watermark;
> +	qm_port->credit_pool[DLB2_DIR_QUEUE] = &dlb2->dir_credit_pool;
> +	qm_port->credit_pool[DLB2_LDB_QUEUE] = &dlb2->ldb_credit_pool;
> +
> +	DLB2_LOG_DBG("dlb2: created dir port %d, depth = %d cr=%d,%d\n",
> +		     qm_port_id,
> +		     dequeue_depth,
> +		     dir_credit_high_watermark,
> +		     ldb_credit_high_watermark);
> +
> +	rte_spinlock_unlock(&handle->resource_lock);
> +
> +	return 0;
> +
> +error_exit:
> +
> +	if (qm_port)
> +		dlb2_free_qe_mem(qm_port);
> +
> +	rte_spinlock_unlock(&handle->resource_lock);
> +
> +	DLB2_LOG_ERR("dlb2: create dir port failed!\n");
> +
> +	return ret;
> +}
> +
> +static int
> +dlb2_eventdev_port_setup(struct rte_eventdev *dev,
> +			 uint8_t ev_port_id,
> +			 const struct rte_event_port_conf *port_conf)
> +{
> +	struct dlb2_eventdev *dlb2;
> +	struct dlb2_eventdev_port *ev_port;
> +	int ret;
> +
> +	if (dev == NULL || port_conf == NULL) {
> +		DLB2_LOG_ERR("Null parameter\n");
> +		return -EINVAL;
> +	}
> +
> +	dlb2 = dlb2_pmd_priv(dev);
> +
> +	if (ev_port_id >= DLB2_MAX_NUM_PORTS)
> +		return -EINVAL;
> +
> +	if (port_conf->dequeue_depth >
> +		evdev_dlb2_default_info.max_event_port_dequeue_depth ||
> +	    port_conf->enqueue_depth >
> +		evdev_dlb2_default_info.max_event_port_enqueue_depth)
> +		return -EINVAL;
> +
> +	ev_port = &dlb2->ev_ports[ev_port_id];
> +	/* configured? */
> +	if (ev_port->setup_done) {
> +		DLB2_LOG_ERR("evport %d is already configured\n",
> ev_port_id);
> +		return -EINVAL;
> +	}
> +
> +	/* The reserved token interrupt arming scheme requires that one or
> more
> +	 * CQ tokens be reserved by the PMD. This limits the amount of CQ space
> +	 * usable by the DLB, so in order to give an *effective* CQ depth equal
> +	 * to the user-requested value, we double CQ depth and reserve half of
> +	 * its tokens. If the user requests the max CQ depth (256) then we
> +	 * cannot double it, so we reserve one token and give an effective
> +	 * depth of 255 entries.
> +	 */

I don't think this comment applies to the 2.0 device.

> +
> +	ev_port->qm_port.is_directed = port_conf->event_port_cfg &
> +		RTE_EVENT_PORT_CFG_SINGLE_LINK;
> +
> +	if (!ev_port->qm_port.is_directed) {
> +		ret = dlb2_hw_create_ldb_port(dlb2,
> +					      ev_port,
> +					      port_conf->dequeue_depth,
> +					      port_conf->enqueue_depth);
> +		if (ret < 0) {
> +			DLB2_LOG_ERR("Failed to create the lB port ve
> portId=%d\n",
> +				     ev_port_id);
> +
> +			return ret;
> +		}
> +	} else {
> +		ret = dlb2_hw_create_dir_port(dlb2,
> +					      ev_port,
> +					      port_conf->dequeue_depth,
> +					      port_conf->enqueue_depth);
> +		if (ret < 0) {
> +			DLB2_LOG_ERR("Failed to create the DIR port\n");
> +			return ret;
> +		}
> +	}
> +
> +	/* Save off port config for reconfig */
> +	dlb2->ev_ports[ev_port_id].conf = *port_conf;

Nit: 'ev_port' is assigned to &dlb2->ev_ports[ev_port_id] above, use it here and below?

> +
> +	dlb2->ev_ports[ev_port_id].id = ev_port_id;
> +	dlb2->ev_ports[ev_port_id].enq_configured = true;
> +	dlb2->ev_ports[ev_port_id].setup_done = true;
> +	dlb2->ev_ports[ev_port_id].inflight_max =
> +		port_conf->new_event_threshold;
> +	dlb2->ev_ports[ev_port_id].implicit_release =
> +		!(port_conf->event_port_cfg &
> +		  RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL);
> +	dlb2->ev_ports[ev_port_id].outstanding_releases = 0;
> +	dlb2->ev_ports[ev_port_id].inflight_credits = 0;
> +	dlb2->ev_ports[ev_port_id].credit_update_quanta =
> +		RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA;
> +	dlb2->ev_ports[ev_port_id].dlb2 = dlb2; /* reverse link */
> +
> +	/* Tear down pre-existing port->queue links */
> +	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
> +		dlb2_port_link_teardown(dlb2, &dlb2->ev_ports[ev_port_id]);
> +
> +	dev->data->ports[ev_port_id] = &dlb2->ev_ports[ev_port_id];
> +
> +	return 0;
> +}

[...]

> diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
> index dea70e6..a6824b1 100644
> --- a/drivers/event/dlb2/pf/dlb2_pf.c
> +++ b/drivers/event/dlb2/pf/dlb2_pf.c
> @@ -234,6 +234,183 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw_dev
> *handle,
>  	return ret;
>  }
> 
> +static void *
> +dlb2_alloc_coherent_aligned(uintptr_t *phys, size_t size, int align)
> +{
> +	const struct rte_memzone *mz;
> +	char mz_name[RTE_MEMZONE_NAMESIZE];
> +	uint32_t core_id = rte_lcore_id();
> +	unsigned int socket_id;
> +
> +	snprintf(mz_name, sizeof(mz_name) - 1, "%lx",
> +		 (unsigned long)rte_get_timer_cycles());

For debug purposes, it would be better if this name can trace the mz back to
this driver. How about something like event_dlb2_pf_name + ldb/dir_port + port ID?

> +	if (core_id == (unsigned int)LCORE_ID_ANY)
> +		core_id = rte_get_master_lcore();
> +	socket_id = rte_lcore_to_socket_id(core_id);

Should this use the socket ID devarg (and perhaps fall back to the core's socket if unspecified)?

> +	mz = rte_memzone_reserve_aligned(mz_name, size, socket_id,
> +					 RTE_MEMZONE_IOVA_CONTIG, align);
> +	if (!mz) {
> +		DLB2_LOG_DBG("Unable to allocate DMA memory of size %zu
> bytes - %s\n",
> +			     size, rte_strerror(rte_errno));
> +		*phys = 0;
> +		return NULL;
> +	}
> +	*phys = mz->iova;
> +	return mz->addr;
> +}
> +
> +static int
> +dlb2_pf_ldb_port_create(struct dlb2_hw_dev *handle,
> +			struct dlb2_create_ldb_port_args *cfg,
> +			enum dlb2_cq_poll_modes poll_mode)
> +{
> +	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> +	struct dlb2_cmd_response response = {0};
> +	struct dlb2_port_memory port_memory;
> +	int ret, cq_alloc_depth;
> +	uint8_t *port_base;
> +	int alloc_sz, qe_sz;
> +	phys_addr_t cq_base;
> +	phys_addr_t pp_base;
> +	int is_dir = false;
> +
> +	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
> +
> +	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
> +		qe_sz = sizeof(struct dlb2_dequeue_qe);
> +	else
> +		qe_sz = RTE_CACHE_LINE_SIZE;
> +
> +	/* Calculate the port memory required, and round up to the nearest
> +	 * cache line.
> +	 */
> +	cq_alloc_depth = RTE_MAX(cfg->cq_depth,
> DLB2_MIN_HARDWARE_CQ_DEPTH);
> +	alloc_sz = cq_alloc_depth * qe_sz;
> +	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
> +
> +	port_base = dlb2_alloc_coherent_aligned(&cq_base,
> +						alloc_sz,
> +						PAGE_SIZE);

This can fit on one line

> +	if (port_base == NULL)
> +		return -ENOMEM;
> +
> +	/* Lock the page in memory */
> +	ret = rte_mem_lock_page(port_base);
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device
> i/o\n");
> +		goto create_port_err;
> +	}
> +
> +
> +	memset(port_base, 0, alloc_sz);
> +
> +	ret = dlb2_pf_create_ldb_port(&dlb2_dev->hw,
> +				      handle->domain_id,
> +				      cfg,
> +				      cq_base,
> +				      &response);
> +	if (ret)
> +		goto create_port_err;
> +
> +	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);
> +	dlb2_port[response.id][DLB2_LDB_PORT].pp_addr =
> +		(void *)(uintptr_t)(pp_base + (PAGE_SIZE * response.id));

If pp_base is defined as a uintptr_t, I think you can avoid some of the explicit
casts.

> +
> +	dlb2_port[response.id][DLB2_LDB_PORT].cq_base =
> +		(void *)(uintptr_t)(port_base);

Since port_base is a uint8_t*, the uintptr_t cast shouldn't be necessary

> +	memset(&port_memory, 0, sizeof(port_memory));
> +	dlb2_list_init_head(&port_memory.list);
> +
> +	/* Fill out the per-port memory tracking structure */
> +	dlb2_dev->ldb_port_pages[response.id].valid = true;
> +	dlb2_list_splice(&port_memory.list,
> +			 &dlb2_dev->ldb_port_pages[response.id].list);

Does the list serve any purpose? Looks like port_memory is zero-initialized, then it becomes
the sole entry on a per-port list.

> +
> +	cfg->response = response;
> +
> +	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
> +		  __func__, ret);
> +
> +create_port_err:

Need to free the memzone in this case.

> +
> +	return ret;
> +}
> +
> +static int
> +dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
> +			struct dlb2_create_dir_port_args *cfg,
> +			enum dlb2_cq_poll_modes poll_mode)
> +{
> +	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
> +	struct dlb2_cmd_response response = {0};
> +	struct dlb2_port_memory port_memory;
> +	int ret;
> +	uint8_t *port_base;
> +	int alloc_sz, qe_sz;
> +	phys_addr_t cq_base;
> +	phys_addr_t pp_base;
> +	int is_dir = true;
> +
> +	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
> +
> +	if (poll_mode == DLB2_CQ_POLL_MODE_STD)
> +		qe_sz = sizeof(struct dlb2_dequeue_qe);
> +	else
> +		qe_sz = RTE_CACHE_LINE_SIZE;
> +
> +	/* Calculate the port memory required, and round up to the nearest
> +	 * cache line.
> +	 */
> +	alloc_sz = cfg->cq_depth * qe_sz;
> +	alloc_sz = RTE_CACHE_LINE_ROUNDUP(alloc_sz);
> +
> +	port_base = dlb2_alloc_coherent_aligned(&cq_base,
> +						alloc_sz,
> +						PAGE_SIZE);

This can fit on one line

> +	if (port_base == NULL)
> +		return -ENOMEM;
> +
> +	/* Lock the page in memory */
> +	ret = rte_mem_lock_page(port_base);
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2 pf pmd could not lock page for device
> i/o\n");
> +		goto create_port_err;
> +	}
> +
> +	memset(port_base, 0, alloc_sz);
> +
> +	ret = dlb2_pf_create_dir_port(&dlb2_dev->hw,
> +				      handle->domain_id,
> +				      cfg,
> +				      cq_base,
> +				      &response);
> +	if (ret)
> +		goto create_port_err;
> +
> +	pp_base = (uintptr_t)dlb2_dev->hw.func_kva + PP_BASE(is_dir);

(See uintptr_t comment above)

> +	dlb2_port[response.id][DLB2_DIR_PORT].pp_addr =
> +		(void *)(uintptr_t)(pp_base + (PAGE_SIZE * response.id));
> +
> +	dlb2_port[response.id][DLB2_DIR_PORT].cq_base =
> +		(void *)(uintptr_t)(port_base);

(See port_base comment above)

> +	memset(&port_memory, 0, sizeof(port_memory));
> +	dlb2_list_init_head(&port_memory.list);
> +
> +	/* Fill out the per-port memory tracking structure */
> +	dlb2_dev->dir_port_pages[response.id].valid = true;
> +	dlb2_list_splice(&port_memory.list,
> +			 &dlb2_dev->dir_port_pages[response.id].list);
> +

(See list comment above)

> +	cfg->response = response;
> +
> +	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
> +		  __func__, ret);
> +
> +create_port_err:

Need to free the memzone in this case

> +
> +	return ret;
> +}
> +
>  static void
>  dlb2_pf_iface_fn_ptrs_init(void)
>  {
> @@ -247,6 +424,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
>  	dlb2_iface_get_cq_poll_mode = dlb2_pf_get_cq_poll_mode;
>  	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
>  	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
> +	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
> +	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
>  	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
>  	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
>  	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
> --
> 2.6.4

I don't see the port memzones getting freed anywhere, e.g. if the event device is reset.
Looks like a possible memory leak.

Thanks,
Gage

[dpdk-dev] [PATCH 12/22] event/dlb2: add port link

[Timothy McDaniel]

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 305 ++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 633 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +
 drivers/event/dlb2/pf/dlb2_pf.c            |  50 +++
 6 files changed, 1010 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index a4c8833..bf50758 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1569,6 +1569,310 @@ dlb2_eventdev_port_setup(struct rte_eventdev *dev,
 	return 0;
 }
 
+static int16_t
+dlb2_hw_map_ldb_qid_to_port(struct dlb2_hw_dev *handle,
+			    uint32_t qm_port_id,
+			    uint16_t qm_qid,
+			    uint8_t priority)
+{
+	struct dlb2_map_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	/* Build message */
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+	cfg.priority = EV_TO_DLB2_PRIO(priority);
+
+	ret = dlb2_iface_map_qid(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: map qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		DLB2_LOG_ERR("dlb2: device_id=%d grp=%d, qm_port=%d, qm_qid=%d prio=%d\n",
+			     handle->device_id,
+			     handle->domain_id, cfg.port_id,
+			     cfg.qid,
+			     cfg.priority);
+	} else {
+		DLB2_LOG_DBG("dlb2: mapped queue %d to qm_port %d\n",
+			     qm_qid, qm_port_id);
+	}
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port,
+			  struct dlb2_eventdev_queue *ev_queue,
+			  uint8_t priority)
+{
+	int first_avail = -1;
+	int ret, i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid) {
+			if (ev_port->link[i].queue_id == ev_queue->id &&
+			    ev_port->link[i].priority == priority) {
+				if (ev_port->link[i].mapped)
+					return 0; /* already mapped */
+				first_avail = i;
+			}
+		} else if (first_avail == -1) {
+			first_avail = i;
+		}
+	}
+	if (first_avail == -1) {
+		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
+			     ev_port->qm_port.id);
+		return -EINVAL;
+	}
+
+	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
+					  ev_port->qm_port.id,
+					  ev_queue->qm_queue.id,
+					  priority);
+
+	if (!ret)
+		ev_port->link[first_avail].mapped = true;
+
+	return ret;
+}
+
+static int32_t
+dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *ev_queue,
+			 int32_t qm_port_id)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_create_dir_queue_args cfg;
+	int32_t ret;
+
+	/* The directed port is always configured before its queue */
+	cfg.port_id = qm_port_id;
+
+	if (ev_queue->depth_threshold == 0) {
+		cfg.depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+		ev_queue->depth_threshold = RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
+	} else {
+		cfg.depth_threshold = ev_queue->depth_threshold;
+	}
+
+	ret = dlb2_iface_dir_queue_create(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return -EINVAL;
+	}
+
+	return cfg.response.id;
+}
+
+static int
+dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_queue *ev_queue,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	int32_t qm_qid;
+
+	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port->qm_port.id);
+
+	if (qm_qid < 0) {
+		DLB2_LOG_ERR("Failed to create the DIR queue\n");
+		return qm_qid;
+	}
+
+	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
+
+	ev_queue->qm_queue.id = qm_qid;
+
+	return 0;
+}
+
+static int
+dlb2_do_port_link(struct rte_eventdev *dev,
+		  struct dlb2_eventdev_queue *ev_queue,
+		  struct dlb2_eventdev_port *ev_port,
+		  uint8_t prio)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int err;
+
+	/* Don't link until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	if (ev_queue->qm_queue.is_directed)
+		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue, ev_port);
+	else
+		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue, prio);
+
+	if (err) {
+		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
+			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
+			     ev_queue->id, ev_port->id);
+
+		rte_errno = err;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
+			uint8_t queue_id,
+			bool link_exists,
+			int index)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	bool port_is_dir, queue_is_dir;
+
+	if (queue_id > dlb2->num_queues) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	ev_queue = &dlb2->ev_queues[queue_id];
+
+	if (!ev_queue->setup_done &&
+	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	port_is_dir = ev_port->qm_port.is_directed;
+	queue_is_dir = ev_queue->qm_queue.is_directed;
+
+	if (port_is_dir != queue_is_dir) {
+		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
+			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
+			     port_is_dir ? "DIR" : "LDB", ev_port->id);
+
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if there is space for the requested link */
+	if (!link_exists && index == -1) {
+		DLB2_LOG_ERR("no space for new link\n");
+		rte_errno = -ENOSPC;
+		return -1;
+	}
+
+	/* Check if the directed port is already linked */
+	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	/* Check if the directed queue is already linked */
+	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
+	    !link_exists) {
+		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
+			     ev_queue->id);
+		rte_errno = -EINVAL;
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
+			const uint8_t queues[], const uint8_t priorities[],
+			uint16_t nb_links)
+
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i, j;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!ev_port->setup_done &&
+	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
+		DLB2_LOG_ERR("dlb2: evport not setup\n");
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
+	 * queues pointer.
+	 */
+	if (nb_links == 0) {
+		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
+		     nb_links,
+		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
+		     ev_port->id);
+
+	for (i = 0; i < nb_links; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		uint8_t queue_id, prio;
+		bool found = false;
+		int index = -1;
+
+		queue_id = queues[i];
+		prio = priorities[i];
+
+		/* Check if the link already exists. */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].valid) {
+				if (ev_port->link[j].queue_id == queue_id) {
+					found = true;
+					index = j;
+					break;
+				}
+			} else if (index == -1) {
+				index = j;
+			}
+
+		/* could not link */
+		if (index == -1)
+			break;
+
+		/* Check if already linked at the requested priority */
+		if (found && ev_port->link[j].priority == prio)
+			continue;
+
+		if (dlb2_validate_port_link(ev_port, queue_id, found, index))
+			break; /* return index of offending queue */
+
+		ev_queue = &dlb2->ev_queues[queue_id];
+
+		if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+			break; /* return index of offending queue */
+
+		ev_queue->num_links++;
+
+		ev_port->link[index].queue_id = queue_id;
+		ev_port->link[index].priority = prio;
+		ev_port->link[index].valid = true;
+		/* Entry already exists?  If so, then must be prio change */
+		if (!found)
+			ev_port->num_links++;
+	}
+	return i;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1580,6 +1884,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_link        = dlb2_eventdev_port_link,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 84cbf25..79f939f 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -66,3 +66,9 @@ int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 				  struct dlb2_create_dir_port_args *cfg,
 				  enum dlb2_cq_poll_modes poll_mode);
 
+int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+				   struct dlb2_create_dir_queue_args *cfg);
+
+int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+			  struct dlb2_map_qid_args *cfg);
+
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index a090a54..90f7b8a 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -51,4 +51,10 @@ extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 struct dlb2_create_dir_port_args *cfg,
 					 enum dlb2_cq_poll_modes poll_mode);
 
+extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
+					struct dlb2_create_dir_queue_args *cfg);
+
+extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
+				 struct dlb2_map_qid_args *cfg);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index bb8683c..45abda1 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -4891,3 +4891,636 @@ int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+				     struct dlb2_hw_domain *domain,
+				     struct dlb2_dir_pq_pair *queue,
+				     struct dlb2_create_dir_queue_args *args,
+				     bool vdev_req,
+				     unsigned int vdev_id)
+{
+	union dlb2_sys_dir_vasqid_v r0 = { {0} };
+	union dlb2_sys_dir_qid_its r1 = { {0} };
+	union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+	union dlb2_sys_dir_qid_v r5 = { {0} };
+
+	unsigned int offs;
+
+	/* QID write permissions are turned on when the domain is started */
+	r0.field.vasqid_v = 0;
+
+	offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+	/* Don't timestamp QEs that pass through this queue */
+	r1.field.qid_its = 0;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+		    r1.val);
+
+	r2.field.thresh = args->depth_threshold;
+
+	DLB2_CSR_WR(hw,
+		    DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+		    r2.val);
+
+	if (vdev_req) {
+		union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+		union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+		offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		r3.field.vqid_v = 1;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+		r4.field.qid = queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+	}
+
+	r5.field.qid_v = 1;
+
+	DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+	queue->queue_configured = true;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_create_dir_queue_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  struct dlb2_create_dir_queue_args *args,
+				  struct dlb2_cmd_response *resp,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		struct dlb2_dir_pq_pair *port;
+
+		port = dlb2_get_domain_used_dir_pq(args->port_id,
+						   vdev_req,
+						   domain);
+
+		if (!port || port->domain_id.phys_id != domain->id.phys_id ||
+		    !port->port_configured) {
+			resp->status = DLB2_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	}
+
+	/*
+	 * If the queue's port is not configured, validate that a free
+	 * port-queue pair is available.
+	 */
+	if (args->port_id == -1 &&
+	    dlb2_list_empty(&domain->avail_dir_pq_pairs)) {
+		resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_create_dir_queue_args *args,
+			     struct dlb2_cmd_response *resp,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+
+	dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_create_dir_queue_args(hw,
+						domain_id,
+						args,
+						resp,
+						vdev_req,
+						vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (!domain) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	if (args->port_id != -1)
+		queue = dlb2_get_domain_used_dir_pq(args->port_id,
+						    vdev_req,
+						    domain);
+	else
+		queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+					   typeof(*queue));
+	if (!queue) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: no available dir queues\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		dlb2_list_del(&domain->avail_dir_pq_pairs,
+			      &queue->domain_list);
+
+		dlb2_list_add(&domain->used_dir_pq_pairs,
+			      &queue->domain_list);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+					   struct dlb2_ldb_queue *queue,
+					   int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+					      struct dlb2_ldb_port *port,
+					      int slot,
+					      struct dlb2_map_qid_args *args)
+{
+	union dlb2_lsp_cq2priov r0;
+
+	/* Read-modify-write the priority and valid bit register */
+	r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+	r0.field.v |= 1 << slot;
+	r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+	DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+	dlb2_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+					      struct dlb2_ldb_queue *queue,
+					      struct dlb2_cmd_response *resp)
+{
+	enum dlb2_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB2_QUEUE_UNMAPPED;
+	if (dlb2_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+			  bool vdev_req,
+			  struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_queue *queue;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue, iter)
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+
+	return NULL;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+			      bool vdev_req,
+			      struct dlb2_hw_domain *domain)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_ldb_port *port;
+	int i;
+	RTE_SET_USED(iter);
+
+	if (id >= DLB2_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+		DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+
+		DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port, iter)
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+	}
+
+	return NULL;
+}
+
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+				    u32 domain_id,
+				    struct dlb2_map_qid_args *args,
+				    struct dlb2_cmd_response *resp,
+				    bool vdev_req,
+				    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+	struct dlb2_ldb_queue *queue;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (!port || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB2_QID_PRIORITIES) {
+		resp->status = DLB2_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (!queue || !queue->configured) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+			     u32 domain_id,
+			     struct dlb2_map_qid_args *args,
+			     bool vdev_req,
+			     unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	DLB2_HW_DBG(hw, "\tPriority:  %d\n",
+		    args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+		    u32 domain_id,
+		    struct dlb2_map_qid_args *args,
+		    struct dlb2_cmd_response *resp,
+		    bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	int ret, i, id;
+	u8 prio;
+
+	dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_map_qid_args(hw,
+				       domain_id,
+				       args,
+				       resp,
+				       vdev_req,
+				       vdev_id);
+	if (ret)
+		return ret;
+
+	prio = args->priority;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (!domain) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (!port) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (!queue) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	if (port->num_pending_removals)
+		dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb2_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		if (prio != port->qid_map[i].priority) {
+			dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+		}
+
+		st = DLB2_QUEUE_MAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		port->qid_map[i].pending_priority = prio;
+
+		DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+		if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb2_qid_map_state st;
+
+			if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+				DLB2_HW_ERR(hw,
+					    "[%s():%d] Internal error: port slot tracking failed\n",
+					    __func__, __LINE__);
+				return -EFAULT;
+			}
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb2_port_slot_state_transition(hw, port, queue,
+							      i, st);
+			if (ret)
+				return ret;
+
+			DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb2_ldb_port_cq_enable(hw, port);
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index 582ef53..efa7c1c 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -670,3 +670,13 @@ dlb2_pf_create_dir_port(struct dlb2_hw *hw,
 				       NOT_VF_REQ,
 				       PF_ID_ZERO);
 }
+
+int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+			 u32 id,
+			 struct dlb2_create_dir_queue_args *args,
+			 struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
+					PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a6824b1..2541842 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -411,6 +411,54 @@ dlb2_pf_dir_port_create(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_dir_queue_create(struct dlb2_hw_dev *handle,
+			 struct dlb2_create_dir_queue_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_create_dir_queue(&dlb2_dev->hw,
+				       handle->domain_id,
+				       cfg,
+				       &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
+		struct dlb2_map_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_map_qid(&dlb2_dev->hw,
+			      handle->domain_id,
+			      cfg,
+			      &response,
+			      false,
+			      0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -425,7 +473,9 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_sched_domain_create = dlb2_pf_sched_domain_create;
 	dlb2_iface_ldb_queue_create = dlb2_pf_ldb_queue_create;
 	dlb2_iface_ldb_port_create = dlb2_pf_ldb_port_create;
+	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
+	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

Re: [dpdk-dev] [PATCH 12/22] event/dlb2: add port link

[Eads, Gage]

> +static int
> +dlb2_event_queue_join_ldb(struct dlb2_eventdev *dlb2,
> +			  struct dlb2_eventdev_port *ev_port,
> +			  struct dlb2_eventdev_queue *ev_queue,
> +			  uint8_t priority)
> +{
> +	int first_avail = -1;
> +	int ret, i;
> +
> +	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
> +		if (ev_port->link[i].valid) {
> +			if (ev_port->link[i].queue_id == ev_queue->id &&
> +			    ev_port->link[i].priority == priority) {
> +				if (ev_port->link[i].mapped)
> +					return 0; /* already mapped */
> +				first_avail = i;
> +			}
> +		} else if (first_avail == -1) {
> +			first_avail = i;
> +		}

Nit: braces discouraged on a single-statement conditionals

> +	}
> +	if (first_avail == -1) {
> +		DLB2_LOG_ERR("dlb2: qm_port %d has no available QID slots.\n",
> +			     ev_port->qm_port.id);
> +		return -EINVAL;
> +	}
> +
> +	ret = dlb2_hw_map_ldb_qid_to_port(&dlb2->qm_instance,
> +					  ev_port->qm_port.id,
> +					  ev_queue->qm_queue.id,
> +					  priority);
> +
> +	if (!ret)
> +		ev_port->link[first_avail].mapped = true;
> +
> +	return ret;
> +}
> +
> +static int32_t
> +dlb2_hw_create_dir_queue(struct dlb2_eventdev *dlb2,
> +			 struct dlb2_eventdev_queue *ev_queue,
> +			 int32_t qm_port_id)
> +{
> +	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
> +	struct dlb2_create_dir_queue_args cfg;
> +	int32_t ret;
> +
> +	/* The directed port is always configured before its queue */
> +	cfg.port_id = qm_port_id;
> +
> +	if (ev_queue->depth_threshold == 0) {
> +		cfg.depth_threshold =
> RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
> +		ev_queue->depth_threshold =
> RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH;
> +	} else {
> +		cfg.depth_threshold = ev_queue->depth_threshold;
> +	}

Nit: braces discouraged on a single-statement conditionals

> +
> +	ret = dlb2_iface_dir_queue_create(handle, &cfg);
> +	if (ret < 0) {
> +		DLB2_LOG_ERR("dlb2: create DIR event queue error, ret=%d
> (driver status: %s)\n",
> +			     ret, dlb2_error_strings[cfg.response.status]);
> +		return -EINVAL;
> +	}
> +
> +	return cfg.response.id;
> +}
> +
> +static int
> +dlb2_eventdev_dir_queue_setup(struct dlb2_eventdev *dlb2,
> +			      struct dlb2_eventdev_queue *ev_queue,
> +			      struct dlb2_eventdev_port *ev_port)
> +{
> +	int32_t qm_qid;
> +
> +	qm_qid = dlb2_hw_create_dir_queue(dlb2, ev_queue, ev_port-
> >qm_port.id);
> +
> +	if (qm_qid < 0) {
> +		DLB2_LOG_ERR("Failed to create the DIR queue\n");
> +		return qm_qid;
> +	}
> +
> +	dlb2->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id;
> +
> +	ev_queue->qm_queue.id = qm_qid;
> +
> +	return 0;
> +}
> +
> +static int
> +dlb2_do_port_link(struct rte_eventdev *dev,
> +		  struct dlb2_eventdev_queue *ev_queue,
> +		  struct dlb2_eventdev_port *ev_port,
> +		  uint8_t prio)
> +{
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	int err;
> +
> +	/* Don't link until start time. */
> +	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
> +		return 0;
> +
> +	if (ev_queue->qm_queue.is_directed)
> +		err = dlb2_eventdev_dir_queue_setup(dlb2, ev_queue,
> ev_port);
> +	else
> +		err = dlb2_event_queue_join_ldb(dlb2, ev_port, ev_queue,
> prio);
> +
> +	if (err) {
> +		DLB2_LOG_ERR("port link failure for %s ev_q %d, ev_port %d\n",
> +			     ev_queue->qm_queue.is_directed ? "DIR" : "LDB",
> +			     ev_queue->id, ev_port->id);
> +
> +		rte_errno = err;
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +dlb2_validate_port_link(struct dlb2_eventdev_port *ev_port,
> +			uint8_t queue_id,
> +			bool link_exists,
> +			int index)
> +{
> +	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
> +	struct dlb2_eventdev_queue *ev_queue;
> +	bool port_is_dir, queue_is_dir;
> +
> +	if (queue_id > dlb2->num_queues) {
> +		rte_errno = -EINVAL;
> +		return -1;
> +	}
> +
> +	ev_queue = &dlb2->ev_queues[queue_id];
> +
> +	if (!ev_queue->setup_done &&
> +	    ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED) {
> +		rte_errno = -EINVAL;
> +		return -1;
> +	}
> +
> +	port_is_dir = ev_port->qm_port.is_directed;
> +	queue_is_dir = ev_queue->qm_queue.is_directed;
> +
> +	if (port_is_dir != queue_is_dir) {
> +		DLB2_LOG_ERR("%s queue %u can't link to %s port %u\n",
> +			     queue_is_dir ? "DIR" : "LDB", ev_queue->id,
> +			     port_is_dir ? "DIR" : "LDB", ev_port->id);
> +
> +		rte_errno = -EINVAL;
> +		return -1;
> +	}
> +
> +	/* Check if there is space for the requested link */
> +	if (!link_exists && index == -1) {
> +		DLB2_LOG_ERR("no space for new link\n");
> +		rte_errno = -ENOSPC;
> +		return -1;
> +	}
> +
> +	/* Check if the directed port is already linked */
> +	if (ev_port->qm_port.is_directed && ev_port->num_links > 0 &&
> +	    !link_exists) {
> +		DLB2_LOG_ERR("Can't link DIR port %d to >1 queues\n",
> +			     ev_port->id);
> +		rte_errno = -EINVAL;
> +		return -1;
> +	}
> +
> +	/* Check if the directed queue is already linked */
> +	if (ev_queue->qm_queue.is_directed && ev_queue->num_links > 0 &&
> +	    !link_exists) {
> +		DLB2_LOG_ERR("Can't link DIR queue %d to >1 ports\n",
> +			     ev_queue->id);
> +		rte_errno = -EINVAL;
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
> +			const uint8_t queues[], const uint8_t priorities[],
> +			uint16_t nb_links)
> +
> +{
> +	struct dlb2_eventdev_port *ev_port = event_port;
> +	struct dlb2_eventdev *dlb2;
> +	int i, j;
> +
> +	RTE_SET_USED(dev);
> +
> +	if (!ev_port) {
> +		DLB2_LOG_ERR("dlb2: evport not setup\n");
> +		rte_errno = -EINVAL;
> +		return 0;
> +	}
> +
> +	if (!ev_port->setup_done &&
> +	    ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED) {
> +		DLB2_LOG_ERR("dlb2: evport not setup\n");
> +		rte_errno = -EINVAL;
> +		return 0;
> +	}
> +
> +	/* Note: rte_event_port_link() ensures the PMD won't receive a NULL
> +	 * queues pointer.
> +	 */
> +	if (nb_links == 0) {
> +		DLB2_LOG_DBG("dlb2: nb_links is 0\n");
> +		return 0; /* Ignore and return success */
> +	}
> +
> +	dlb2 = ev_port->dlb2;
> +
> +	DLB2_LOG_DBG("Linking %u queues to %s port %d\n",
> +		     nb_links,
> +		     ev_port->qm_port.is_directed ? "DIR" : "LDB",
> +		     ev_port->id);
> +
> +	for (i = 0; i < nb_links; i++) {
> +		struct dlb2_eventdev_queue *ev_queue;
> +		uint8_t queue_id, prio;
> +		bool found = false;
> +		int index = -1;
> +
> +		queue_id = queues[i];
> +		prio = priorities[i];
> +
> +		/* Check if the link already exists. */
> +		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
> +			if (ev_port->link[j].valid) {
> +				if (ev_port->link[j].queue_id == queue_id) {
> +					found = true;
> +					index = j;
> +					break;
> +				}
> +			} else if (index == -1) {
> +				index = j;
> +			}

Nit: braces discouraged on a single-statement conditionals

Thanks,
Gage

[dpdk-dev] [PATCH 13/22] event/dlb2: add port unlink and port unlinks in progress

[Timothy McDaniel]

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 163 +++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   5 +
 drivers/event/dlb2/dlb2_iface.h            |   8 +-
 drivers/event/dlb2/pf/base/dlb2_resource.c | 283 +++++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  51 ++++++
 5 files changed, 509 insertions(+), 1 deletion(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index bf50758..7ac47d9 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -1873,6 +1873,166 @@ dlb2_eventdev_port_link(struct rte_eventdev *dev, void *event_port,
 	return i;
 }
 
+static int16_t
+dlb2_hw_unmap_ldb_qid_from_port(struct dlb2_hw_dev *handle,
+				uint32_t qm_port_id,
+				uint16_t qm_qid)
+{
+	struct dlb2_unmap_qid_args cfg;
+	int32_t ret;
+
+	if (handle == NULL)
+		return -EINVAL;
+
+	cfg.port_id = qm_port_id;
+	cfg.qid = qm_qid;
+
+	ret = dlb2_iface_unmap_qid(handle, &cfg);
+	if (ret < 0)
+		DLB2_LOG_ERR("dlb2: unmap qid error, ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+
+	return ret;
+}
+
+static int
+dlb2_event_queue_detach_ldb(struct dlb2_eventdev *dlb2,
+			    struct dlb2_eventdev_port *ev_port,
+			    struct dlb2_eventdev_queue *ev_queue)
+{
+	int ret, i;
+
+	/* Don't unlink until start time. */
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED)
+		return 0;
+
+	for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (ev_port->link[i].valid &&
+		    ev_port->link[i].queue_id == ev_queue->id)
+			break; /* found */
+	}
+
+	/* This is expected with eventdev API!
+	 * It blindly attemmpts to unmap all queues.
+	 */
+	if (i == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_LOG_DBG("dlb2: ignoring LB QID %d not mapped for qm_port %d.\n",
+			     ev_queue->qm_queue.id,
+			     ev_port->qm_port.id);
+		return 0;
+	}
+
+	ret = dlb2_hw_unmap_ldb_qid_from_port(&dlb2->qm_instance,
+					      ev_port->qm_port.id,
+					      ev_queue->qm_queue.id);
+	if (!ret)
+		ev_port->link[i].mapped = false;
+
+	return ret;
+}
+
+static int
+dlb2_eventdev_port_unlink(struct rte_eventdev *dev, void *event_port,
+			  uint8_t queues[], uint16_t nb_unlinks)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	int i;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	if (!queues || nb_unlinks == 0) {
+		DLB2_LOG_DBG("dlb2: queues is NULL or nb_unlinks is 0\n");
+		return 0; /* Ignore and return success */
+	}
+
+	/* FIXME: How to handle unlink on a directed port? */
+	if (ev_port->qm_port.is_directed) {
+		DLB2_LOG_DBG("dlb2: ignore unlink from dir port %d\n",
+			     ev_port->id);
+		rte_errno = 0;
+		return nb_unlinks; /* as if success */
+	}
+
+	dlb2 = ev_port->dlb2;
+
+	for (i = 0; i < nb_unlinks; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+		int ret, j;
+
+		if (queues[i] >= dlb2->num_queues) {
+			DLB2_LOG_ERR("dlb2: invalid queue id %d\n", queues[i]);
+			rte_errno = -EINVAL;
+			return i; /* return index of offending queue */
+		}
+
+		ev_queue = &dlb2->ev_queues[queues[i]];
+
+		/* Does a link exist? */
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+			if (ev_port->link[j].queue_id == queues[i] &&
+			    ev_port->link[j].valid)
+				break;
+
+		if (j == DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+			continue;
+
+		ret = dlb2_event_queue_detach_ldb(dlb2, ev_port, ev_queue);
+		if (ret) {
+			DLB2_LOG_ERR("unlink err=%d for port %d queue %d\n",
+				     ret, ev_port->id, queues[i]);
+			rte_errno = -ENOENT;
+			return i; /* return index of offending queue */
+		}
+
+		ev_port->link[j].valid = false;
+		ev_port->num_links--;
+		ev_queue->num_links--;
+	}
+
+	return nb_unlinks;
+}
+
+static int
+dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
+				       void *event_port)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_eventdev *dlb2;
+	struct dlb2_hw_dev *handle;
+	struct dlb2_pending_port_unmaps_args cfg;
+	int ret;
+
+	RTE_SET_USED(dev);
+
+	if (!ev_port->setup_done) {
+		DLB2_LOG_ERR("dlb2: evport %d is not configured\n",
+			     ev_port->id);
+		rte_errno = -EINVAL;
+		return 0;
+	}
+
+	cfg.port_id = ev_port->qm_port.id;
+	dlb2 = ev_port->dlb2;
+	handle = &dlb2->qm_instance;
+	ret = dlb2_iface_pending_port_unmaps(handle, &cfg);
+
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: num_unlinks_in_progress ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -1885,6 +2045,9 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
 		.port_link        = dlb2_eventdev_port_link,
+		.port_unlink      = dlb2_eventdev_port_unlink,
+		.port_unlinks_in_progress =
+				    dlb2_eventdev_port_unlinks_in_progress,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 79f939f..8034165 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -72,3 +72,8 @@ int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
 int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 			  struct dlb2_map_qid_args *cfg);
 
+int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+			    struct dlb2_unmap_qid_args *cfg);
+
+int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 90f7b8a..182a5bf 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -52,9 +52,15 @@ extern int (*dlb2_iface_dir_port_create)(struct dlb2_hw_dev *handle,
 					 enum dlb2_cq_poll_modes poll_mode);
 
 extern int (*dlb2_iface_dir_queue_create)(struct dlb2_hw_dev *handle,
-					struct dlb2_create_dir_queue_args *cfg);
+				  struct dlb2_create_dir_queue_args *cfg);
 
 extern int (*dlb2_iface_map_qid)(struct dlb2_hw_dev *handle,
 				 struct dlb2_map_qid_args *cfg);
 
+extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
+				   struct dlb2_unmap_qid_args *cfg);
+
+extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
+				struct dlb2_pending_port_unmaps_args *args);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 45abda1..a5b2be1 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5524,3 +5524,286 @@ int dlb2_hw_map_qid(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+			       u32 domain_id,
+			       struct dlb2_unmap_qid_args *args,
+			       bool vdev_req,
+			       unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+		    domain_id);
+	DLB2_HW_DBG(hw, "\tPort ID:   %d\n",
+		    args->port_id);
+	DLB2_HW_DBG(hw, "\tQueue ID:  %d\n",
+		    args->qid);
+	if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+		DLB2_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			    hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+				      u32 domain_id,
+				      struct dlb2_unmap_qid_args *args,
+				      struct dlb2_cmd_response *resp,
+				      bool vdev_req,
+				      unsigned int vdev_id)
+{
+	enum dlb2_qid_map_state state;
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	struct dlb2_ldb_port *port;
+	int slot;
+	int id;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (!port || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (!queue || !queue->configured) {
+		DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			    __func__, args->qid);
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB2_QUEUE_MAPPED;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	state = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+		return 0;
+
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		return 0;
+
+	resp->status = DLB2_ST_INVALID_QID;
+	return -EINVAL;
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+		      u32 domain_id,
+		      struct dlb2_unmap_qid_args *args,
+		      struct dlb2_cmd_response *resp,
+		      bool vdev_req,
+		      unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+	enum dlb2_qid_map_state st;
+	struct dlb2_ldb_port *port;
+	bool unmap_complete;
+	int i, ret, id;
+
+	dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb2_verify_unmap_qid_args(hw,
+					 domain_id,
+					 args,
+					 resp,
+					 vdev_req,
+					 vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (!domain) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	id = args->port_id;
+
+	port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+	if (!port) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+	if (!queue) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: queue not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB2_QUEUE_MAP_IN_PROG;
+	if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+		st = DLB2_QUEUE_UNMAPPED;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+			DLB2_HW_ERR(hw,
+				    "[%s():%d] Internal error: port slot tracking failed\n",
+				    __func__, __LINE__);
+			return -EFAULT;
+		}
+
+		st = DLB2_QUEUE_UNMAP_IN_PROG;
+		ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB2_QUEUE_MAPPED;
+	if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+		DLB2_HW_ERR(hw,
+			    "[%s()] Internal error: no available CQ slots\n",
+			    __func__);
+		return -EFAULT;
+	}
+
+	if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: port slot tracking failed\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asynchronous procedure. It requires
+	 * stopping the DLB2 from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb2_ldb_port_cq_disable(hw, port);
+
+	st = DLB2_QUEUE_UNMAP_IN_PROG;
+	ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete && !os_worker_active(hw))
+		os_schedule_work(hw);
+
+unmap_qid_done:
+	resp->status = 0;
+
+	return 0;
+}
+
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+				  struct dlb2_pending_port_unmaps_args *args,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_pending_port_unmaps_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_port *port;
+
+	dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (!port || !port->configured) {
+		resp->status = DLB2_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 2541842..a3ee198 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -459,6 +459,55 @@ dlb2_pf_map_qid(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw_dev *handle,
+		  struct dlb2_unmap_qid_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_unmap_qid(&dlb2_dev->hw,
+				handle->domain_id,
+				cfg,
+				&response,
+				false,
+				0);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
+			    struct dlb2_pending_port_unmaps_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_pending_port_unmaps(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -476,6 +525,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_queue_create = dlb2_pf_dir_queue_create;
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
+	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
 	dlb2_iface_get_sn_occupancy = dlb2_pf_get_sn_occupancy;
-- 
2.6.4

Re: [dpdk-dev] [PATCH 13/22] event/dlb2: add port unlink and port unlinks in progress

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 13/22] event/dlb2: add port unlink and port unlinks in progress
> 
> Add supports for the port unlink(s) eventdev entry points.
> The unlink operation is an asynchronous operation executed by
> a control thread, and the unlinks-in-progress function reads
> a counter shared with the control thread.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH 14/22] event/dlb2: add eventdev start

[Timothy McDaniel]

Add support for the eventdev start entry point.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 133 +++++++++++++++++++++++++++++
 drivers/event/dlb2/dlb2_iface.c            |   3 +
 drivers/event/dlb2/dlb2_iface.h            |   3 +
 drivers/event/dlb2/pf/base/dlb2_resource.c | 123 ++++++++++++++++++++++++++
 drivers/event/dlb2/pf/dlb2_main.c          |  10 +++
 drivers/event/dlb2/pf/dlb2_pf.c            |  25 ++++++
 6 files changed, 297 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 7ac47d9..ef23def 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2033,6 +2033,138 @@ dlb2_eventdev_port_unlinks_in_progress(struct rte_eventdev *dev,
 	return cfg.response.id;
 }
 
+static int
+dlb2_eventdev_reapply_configuration(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int ret, i;
+
+	/* If an event queue or port was previously configured, but hasn't been
+	 * reconfigured, reapply its original configuration.
+	 */
+	for (i = 0; i < dlb2->num_queues; i++) {
+		struct dlb2_eventdev_queue *ev_queue;
+
+		ev_queue = &dlb2->ev_queues[i];
+
+		if (ev_queue->qm_queue.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_queue_setup(dev, i, &ev_queue->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure queue %d", i);
+			return ret;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+
+		if (ev_port->qm_port.config_state != DLB2_PREV_CONFIGURED)
+			continue;
+
+		ret = dlb2_eventdev_port_setup(dev, i, &ev_port->conf);
+		if (ret < 0) {
+			DLB2_LOG_ERR("dlb2: failed to reconfigure ev_port %d",
+				     i);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_apply_port_links(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	int i;
+
+	/* Perform requested port->queue links */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		struct dlb2_eventdev_port *ev_port = &dlb2->ev_ports[i];
+		int j;
+
+		for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++) {
+			struct dlb2_eventdev_queue *ev_queue;
+			uint8_t prio, queue_id;
+
+			if (!ev_port->link[j].valid)
+				continue;
+
+			prio = ev_port->link[j].priority;
+			queue_id = ev_port->link[j].queue_id;
+
+			if (dlb2_validate_port_link(ev_port, queue_id, true, j))
+				return -EINVAL;
+
+			ev_queue = &dlb2->ev_queues[queue_id];
+
+			if (dlb2_do_port_link(dev, ev_queue, ev_port, prio))
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int
+dlb2_eventdev_start(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_start_domain_args cfg;
+	int ret, i;
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_ERR("bad state %d for dev_start\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return -EINVAL;
+	}
+	dlb2->run_state	= DLB2_RUN_STATE_STARTING;
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	/* If the device was configured more than once, some event ports and/or
+	 * queues may need to be reconfigured.
+	 */
+	ret = dlb2_eventdev_reapply_configuration(dev);
+	if (ret)
+		return ret;
+
+	/* The DLB PMD delays port links until the device is started. */
+	ret = dlb2_eventdev_apply_port_links(dev);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < dlb2->num_ports; i++) {
+		if (!dlb2->ev_ports[i].setup_done) {
+			DLB2_LOG_ERR("dlb2: port %d not setup", i);
+			return -ESTALE;
+		}
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0) {
+			DLB2_LOG_ERR("dlb2: queue %d is not linked", i);
+			return -ENOLINK;
+		}
+	}
+
+	ret = dlb2_iface_sched_domain_start(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: sched_domain_start ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STARTED;
+	DLB2_LOG_DBG("dlb2: sched_domain_start completed OK\n");
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2040,6 +2172,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
+		.dev_start        = dlb2_eventdev_start,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 8034165..8228bb3 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -77,3 +77,6 @@ int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
+
+int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				     struct dlb2_start_domain_args *cfg);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index 182a5bf..d5bb6be 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -63,4 +63,7 @@ extern int (*dlb2_iface_unmap_qid)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 				struct dlb2_pending_port_unmaps_args *args);
 
+extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
+				struct dlb2_start_domain_args *cfg);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index a5b2be1..4c4c0b4 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5807,3 +5807,126 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 struct dlb2_cmd_response *resp,
+					 bool vdev_req,
+					 unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB2_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void dlb2_log_start_domain(struct dlb2_hw *hw,
+				  u32 domain_id,
+				  bool vdev_req,
+				  unsigned int vdev_id)
+{
+	DLB2_HW_DBG(hw, "DLB2 start domain arguments:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_hw_start_domain() - Lock the domain configuration
+ * @hw:	Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @arg: User-provided arguments (unused, here for ioctl callback template).
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int
+dlb2_hw_start_domain(struct dlb2_hw *hw,
+		     u32 domain_id,
+		     __attribute((unused)) struct dlb2_start_domain_args *arg,
+		     struct dlb2_cmd_response *resp,
+		     bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb2_list_entry *iter;
+	struct dlb2_dir_pq_pair *dir_queue;
+	struct dlb2_ldb_queue *ldb_queue;
+	struct dlb2_hw_domain *domain;
+	int ret;
+	RTE_SET_USED(arg);
+	RTE_SET_USED(iter);
+
+	dlb2_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb2_verify_start_domain_args(hw,
+					    domain_id,
+					    resp,
+					    vdev_req,
+					    vdev_id);
+	if (ret)
+		return ret;
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (!domain) {
+		DLB2_HW_ERR(hw,
+			    "[%s():%d] Internal error: domain not found\n",
+			    __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB2 will drop all
+	 * incoming traffic to those queues.
+	 */
+	DLB2_DOM_LIST_FOR(domain->used_ldb_queues, ldb_queue, iter) {
+		union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r0.val);
+	}
+
+	DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_queue, iter) {
+		union dlb2_sys_dir_vasqid_v r0 = { {0} };
+		unsigned int offs;
+
+		r0.field.vasqid_v = 1;
+
+		offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+	}
+
+	dlb2_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_main.c b/drivers/event/dlb2/pf/dlb2_main.c
index efa7c1c..ba1c95c 100644
--- a/drivers/event/dlb2/pf/dlb2_main.c
+++ b/drivers/event/dlb2/pf/dlb2_main.c
@@ -680,3 +680,13 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
 	return dlb2_hw_create_dir_queue(hw, id, args, resp, NOT_VF_REQ,
 					PF_ID_ZERO);
 }
+
+int
+dlb2_pf_start_domain(struct dlb2_hw *hw,
+		     u32 id,
+		     struct dlb2_start_domain_args *args,
+		     struct dlb2_cmd_response *resp)
+{
+	return dlb2_hw_start_domain(hw, id, args, resp, NOT_VF_REQ,
+				    PF_ID_ZERO);
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index a3ee198..084ba64 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -508,6 +508,30 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw_dev *handle,
 
 	return ret;
 }
+
+static int
+dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
+			   struct dlb2_start_domain_args *cfg)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_pf_start_domain(&dlb2_dev->hw,
+				   handle->domain_id,
+				   cfg,
+				   &response);
+
+	cfg->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -526,6 +550,7 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
 	dlb2_iface_set_sn_allocation = dlb2_pf_set_sn_allocation;
-- 
2.6.4

Re: [dpdk-dev] [PATCH 14/22] event/dlb2: add eventdev start

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 14/22] event/dlb2: add eventdev start
> 
> Add support for the eventdev start entry point.
> 

Code looks fine (one whitespace nit below), but it could use a more detailed
commit message -- e.g. describing how the PMD delays port links until
eventdev start time, how the configuration will be reapplied at this time if
this is a reconfiguration scenario, etc.

[...]

> +static int
> +dlb2_eventdev_start(struct rte_eventdev *dev)
> +{
> +	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
> +	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
> +	struct dlb2_start_domain_args cfg;
> +	int ret, i;
> +
> +	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
> +	if (dlb2->run_state != DLB2_RUN_STATE_STOPPED) {
> +		DLB2_LOG_ERR("bad state %d for dev_start\n",
> +			     (int)dlb2->run_state);
> +		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
> +		return -EINVAL;
> +	}
> +	dlb2->run_state	= DLB2_RUN_STATE_STARTING;

Looks like there's a tab between run_state and the equals sign.

Thanks,
Gage

[dpdk-dev] [PATCH 15/22] event/dlb2: add enqueue and its burst variants

[Timothy McDaniel]

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c | 592 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 592 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index ef23def..ac4cf19 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2165,6 +2165,592 @@ dlb2_eventdev_start(struct rte_eventdev *dev)
 	return 0;
 }
 
+static uint8_t cmd_byte_map[DLB2_NUM_PORT_TYPES][DLB2_NUM_HW_SCHED_TYPES] = {
+	{
+		/* Load-balanced cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_FWD_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_COMP_CMD_BYTE,
+	},
+	{
+		/* Directed cmd bytes */
+		[RTE_EVENT_OP_NEW] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_FORWARD] = DLB2_NEW_CMD_BYTE,
+		[RTE_EVENT_OP_RELEASE] = DLB2_NOOP_CMD_BYTE,
+	},
+};
+
+static inline uint32_t
+dlb2_port_credits_get(struct dlb2_port *qm_port,
+		      enum dlb2_hw_queue_types type)
+{
+	uint32_t credits = *qm_port->credit_pool[type];
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	if (unlikely(credits < batch_size))
+		batch_size = credits;
+
+	if (likely(credits &&
+		   __atomic_compare_exchange_n(
+			qm_port->credit_pool[type],
+			&credits, credits - batch_size, false,
+			__ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)))
+		return batch_size;
+	else
+		return 0;
+}
+
+static inline void
+dlb2_replenish_sw_credits(struct dlb2_eventdev *dlb2,
+			  struct dlb2_eventdev_port *ev_port)
+{
+	uint16_t quanta = ev_port->credit_update_quanta;
+
+	if (ev_port->inflight_credits >= quanta * 2) {
+		/* Replenish credits, saving one quanta for enqueues */
+		uint16_t val = ev_port->inflight_credits - quanta;
+
+		__atomic_fetch_sub(&dlb2->inflights, val, __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits -= val;
+	}
+}
+
+static inline int
+dlb2_check_enqueue_sw_credits(struct dlb2_eventdev *dlb2,
+			      struct dlb2_eventdev_port *ev_port)
+{
+	uint32_t sw_inflights = __atomic_load_n(&dlb2->inflights,
+						__ATOMIC_SEQ_CST);
+	const int num = 1;
+
+	if (unlikely(ev_port->inflight_max < sw_inflights)) {
+		DLB2_INC_STAT(ev_port->stats.traffic.tx_nospc_inflight_max, 1);
+		rte_errno = -ENOSPC;
+		return 1;
+	}
+
+	if (ev_port->inflight_credits < num) {
+		/* check if event enqueue brings ev_port over max threshold */
+		uint32_t credit_update_quanta = ev_port->credit_update_quanta;
+
+		if (sw_inflights + credit_update_quanta >
+				dlb2->new_event_limit) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_new_event_limit,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+
+		__atomic_fetch_add(&dlb2->inflights, credit_update_quanta,
+				   __ATOMIC_SEQ_CST);
+		ev_port->inflight_credits += (credit_update_quanta);
+
+		if (ev_port->inflight_credits < num) {
+			DLB2_INC_STAT(
+			ev_port->stats.traffic.tx_nospc_inflight_credits,
+			1);
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_ldb_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_ldb_credits == 0)) {
+		qm_port->cached_ldb_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_LDB_QUEUE);
+		if (unlikely(qm_port->cached_ldb_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits,
+			1);
+			DLB2_LOG_DBG("ldb credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_check_enqueue_hw_dir_credits(struct dlb2_port *qm_port)
+{
+	if (unlikely(qm_port->cached_dir_credits == 0)) {
+		qm_port->cached_dir_credits =
+			dlb2_port_credits_get(qm_port,
+					      DLB2_DIR_QUEUE);
+		if (unlikely(qm_port->cached_dir_credits == 0)) {
+			DLB2_INC_STAT(
+			qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits,
+			1);
+			DLB2_LOG_DBG("dir credits exhausted\n");
+			return 1; /* credits exhausted */
+		}
+	}
+
+	return 0;
+}
+
+static __rte_always_inline void
+dlb2_pp_write(struct dlb2_enqueue_qe *qe4,
+	      struct process_local_port_data *port_data)
+{
+	qm_mmio_fns.pp_enqueue_four(qe4, port_data->pp_addr);
+}
+
+static inline int
+dlb2_consume_qe_immediate(struct dlb2_port *qm_port, int num)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_cq_pop_qe *qe;
+
+	RTE_ASSERT(qm_port->config_state == DLB2_CONFIGURED);
+
+	qe = qm_port->consume_qe;
+
+	qe->tokens = num - 1;
+
+	/* No store fence needed since no pointer is being sent, and CQ token
+	 * pops can be safely reordered with other HCWs.
+	 */
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	dlb2_movntdq_single(qe, port_data->pp_addr);
+
+	DLB2_LOG_DBG("dlb2: consume immediate - %d QEs\n", num);
+
+	qm_port->owed_tokens = 0;
+
+	return 0;
+}
+
+static inline void
+dlb2_hw_do_enqueue(struct dlb2_port *qm_port,
+		   bool do_sfence,
+		   struct process_local_port_data *port_data)
+{
+	/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that
+	 * application writes complete before enqueueing the QE.
+	 */
+	if (do_sfence)
+		rte_wmb();
+
+	dlb2_pp_write(qm_port->qe4, port_data);
+}
+
+static inline void
+dlb2_construct_token_pop_qe(struct dlb2_port *qm_port, int idx)
+{
+	struct dlb2_cq_pop_qe *qe = (void *)qm_port->qe4;
+	int num = qm_port->owed_tokens;
+
+	qe[idx].cmd_byte = DLB2_POP_CMD_BYTE;
+	qe[idx].tokens = num - 1;
+
+	qm_port->owed_tokens = 0;
+}
+
+static inline void
+dlb2_event_build_hcws(struct dlb2_port *qm_port,
+		      const struct rte_event ev[],
+		      int num,
+		      uint8_t *sched_type,
+		      uint8_t *queue_id)
+{
+	struct dlb2_enqueue_qe *qe;
+	uint16_t sched_word[4];
+	__m128i sse_qe[2];
+	int i;
+
+	qe = qm_port->qe4;
+
+	sse_qe[0] = _mm_setzero_si128();
+	sse_qe[1] = _mm_setzero_si128();
+
+	switch (num) {
+	case 4:
+		/* Construct the metadata portion of two HCWs in one 128b SSE
+		 * register. HCW metadata is constructed in the SSE registers
+		 * like so:
+		 * sse_qe[0][63:0]:   qe[0]'s metadata
+		 * sse_qe[0][127:64]: qe[1]'s metadata
+		 * sse_qe[1][63:0]:   qe[2]'s metadata
+		 * sse_qe[1][127:64]: qe[3]'s metadata
+		 */
+
+		/* Convert the event operation into a command byte and store it
+		 * in the metadata:
+		 * sse_qe[0][63:56]   = cmd_byte_map[is_directed][ev[0].op]
+		 * sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op]
+		 * sse_qe[1][63:56]   = cmd_byte_map[is_directed][ev[2].op]
+		 * sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op]
+		 */
+#define DLB2_QE_CMD_BYTE 7
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[0].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[0] = _mm_insert_epi8(sse_qe[0],
+				cmd_byte_map[qm_port->is_directed][ev[1].op],
+				DLB2_QE_CMD_BYTE + 8);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[2].op],
+				DLB2_QE_CMD_BYTE);
+		sse_qe[1] = _mm_insert_epi8(sse_qe[1],
+				cmd_byte_map[qm_port->is_directed][ev[3].op],
+				DLB2_QE_CMD_BYTE + 8);
+
+		/* Store priority, scheduling type, and queue ID in the sched
+		 * word array because these values are re-used when the
+		 * destination is a directed queue.
+		 */
+		sched_word[0] = EV_TO_DLB2_PRIO(ev[0].priority) << 10 |
+				sched_type[0] << 8 |
+				queue_id[0];
+		sched_word[1] = EV_TO_DLB2_PRIO(ev[1].priority) << 10 |
+				sched_type[1] << 8 |
+				queue_id[1];
+		sched_word[2] = EV_TO_DLB2_PRIO(ev[2].priority) << 10 |
+				sched_type[2] << 8 |
+				queue_id[2];
+		sched_word[3] = EV_TO_DLB2_PRIO(ev[3].priority) << 10 |
+				sched_type[3] << 8 |
+				queue_id[3];
+
+		/* Store the event priority, scheduling type, and queue ID in
+		 * the metadata:
+		 * sse_qe[0][31:16] = sched_word[0]
+		 * sse_qe[0][95:80] = sched_word[1]
+		 * sse_qe[1][31:16] = sched_word[2]
+		 * sse_qe[1][95:80] = sched_word[3]
+		 */
+#define DLB2_QE_QID_SCHED_WORD 1
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[0],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     sched_word[1],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[2],
+					     DLB2_QE_QID_SCHED_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     sched_word[3],
+					     DLB2_QE_QID_SCHED_WORD + 4);
+
+		/* If the destination is a load-balanced queue, store the lock
+		 * ID. If it is a directed queue, DLB places this field in
+		 * bytes 10-11 of the received QE, so we format it accordingly:
+		 * sse_qe[0][47:32]  = dir queue ? sched_word[0] : flow_id[0]
+		 * sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1]
+		 * sse_qe[1][47:32]  = dir queue ? sched_word[2] : flow_id[2]
+		 * sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3]
+		 */
+#define DLB2_QE_LOCK_ID_WORD 2
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[0] == DLB2_SCHED_DIRECTED) ?
+					sched_word[0] : ev[0].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+				(sched_type[1] == DLB2_SCHED_DIRECTED) ?
+					sched_word[1] : ev[1].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[2] == DLB2_SCHED_DIRECTED) ?
+					sched_word[2] : ev[2].flow_id,
+				DLB2_QE_LOCK_ID_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+				(sched_type[3] == DLB2_SCHED_DIRECTED) ?
+					sched_word[3] : ev[3].flow_id,
+				DLB2_QE_LOCK_ID_WORD + 4);
+
+		/* Store the event type and sub event type in the metadata:
+		 * sse_qe[0][15:0]  = flow_id[0]
+		 * sse_qe[0][79:64] = flow_id[1]
+		 * sse_qe[1][15:0]  = flow_id[2]
+		 * sse_qe[1][79:64] = flow_id[3]
+		 */
+#define DLB2_QE_EV_TYPE_WORD 0
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[0].sub_event_type << 8 |
+						ev[0].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[0] = _mm_insert_epi16(sse_qe[0],
+					     ev[1].sub_event_type << 8 |
+						ev[1].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[2].sub_event_type << 8 |
+						ev[2].event_type,
+					     DLB2_QE_EV_TYPE_WORD);
+		sse_qe[1] = _mm_insert_epi16(sse_qe[1],
+					     ev[3].sub_event_type << 8 |
+						ev[3].event_type,
+					     DLB2_QE_EV_TYPE_WORD + 4);
+
+		/* Store the metadata to memory (use the double-precision
+		 * _mm_storeh_pd because there is no integer function for
+		 * storing the upper 64b):
+		 * qe[0] metadata = sse_qe[0][63:0]
+		 * qe[1] metadata = sse_qe[0][127:64]
+		 * qe[2] metadata = sse_qe[1][63:0]
+		 * qe[3] metadata = sse_qe[1][127:64]
+		 */
+		_mm_storel_epi64((__m128i *)&qe[0].u.opaque_data, sse_qe[0]);
+		_mm_storeh_pd((double *)&qe[1].u.opaque_data,
+			      (__m128d)sse_qe[0]);
+		_mm_storel_epi64((__m128i *)&qe[2].u.opaque_data, sse_qe[1]);
+		_mm_storeh_pd((double *)&qe[3].u.opaque_data,
+			      (__m128d)sse_qe[1]);
+
+		qe[0].data = ev[0].u64;
+		qe[1].data = ev[1].u64;
+		qe[2].data = ev[2].u64;
+		qe[3].data = ev[3].u64;
+
+		break;
+	case 3:
+	case 2:
+	case 1:
+		/* At least one QE will be valid, so only zero out three */
+		qe[1].cmd_byte = 0;
+		qe[2].cmd_byte = 0;
+		qe[3].cmd_byte = 0;
+
+		for (i = 0; i < num; i++) {
+			qe[i].cmd_byte =
+				cmd_byte_map[qm_port->is_directed][ev[i].op];
+			qe[i].sched_type = sched_type[i];
+			qe[i].data = ev[i].u64;
+			qe[i].qid = queue_id[i];
+			qe[i].priority = EV_TO_DLB2_PRIO(ev[i].priority);
+			qe[i].lock_id = ev[i].flow_id;
+			if (sched_type[i] == DLB2_SCHED_DIRECTED) {
+				struct dlb2_msg_info *info =
+					(struct dlb2_msg_info *)&qe[i].lock_id;
+
+				info->qid = queue_id[i];
+				info->sched_type = DLB2_SCHED_DIRECTED;
+				info->priority = qe[i].priority;
+			}
+			qe[i].u.event_type.major = ev[i].event_type;
+			qe[i].u.event_type.sub = ev[i].sub_event_type;
+		}
+		break;
+	}
+}
+
+static inline int
+dlb2_event_enqueue_prep(struct dlb2_eventdev_port *ev_port,
+			struct dlb2_port *qm_port,
+			const struct rte_event ev[],
+			uint8_t *sched_type,
+			uint8_t *queue_id)
+{
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	struct dlb2_eventdev_queue *ev_queue;
+	uint16_t *cached_credits = NULL;
+	struct dlb2_queue *qm_queue;
+
+	ev_queue = &dlb2->ev_queues[ev->queue_id];
+	qm_queue = &ev_queue->qm_queue;
+	*queue_id = qm_queue->id;
+
+	/* Ignore sched_type and hardware credits on release events */
+	if (ev->op == RTE_EVENT_OP_RELEASE)
+		goto op_check;
+
+	if (!qm_queue->is_directed) {
+		/* Load balanced destination queue */
+
+		if (dlb2_check_enqueue_hw_ldb_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_ldb_credits;
+
+		switch (ev->sched_type) {
+		case RTE_SCHED_TYPE_ORDERED:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ORDERED\n");
+			if (qm_queue->sched_type != RTE_SCHED_TYPE_ORDERED) {
+				DLB2_LOG_ERR("dlb2: tried to send ordered event to unordered queue %d\n",
+					     *queue_id);
+				rte_errno = -EINVAL;
+				return 1;
+			}
+			*sched_type = DLB2_SCHED_ORDERED;
+			break;
+		case RTE_SCHED_TYPE_ATOMIC:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_ATOMIC\n");
+			*sched_type = DLB2_SCHED_ATOMIC;
+			break;
+		case RTE_SCHED_TYPE_PARALLEL:
+			DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_PARALLEL\n");
+			if (qm_queue->sched_type == RTE_SCHED_TYPE_ORDERED)
+				*sched_type = DLB2_SCHED_ORDERED;
+			else
+				*sched_type = DLB2_SCHED_UNORDERED;
+			break;
+		default:
+			DLB2_LOG_ERR("Unsupported LDB sched type in put_qe\n");
+			DLB2_INC_STAT(ev_port->stats.tx_invalid, 1);
+			rte_errno = -EINVAL;
+			return 1;
+		}
+	} else {
+		/* Directed destination queue */
+
+		if (dlb2_check_enqueue_hw_dir_credits(qm_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		cached_credits = &qm_port->cached_dir_credits;
+
+		DLB2_LOG_DBG("dlb2: put_qe: RTE_SCHED_TYPE_DIRECTED\n");
+
+		*sched_type = DLB2_SCHED_DIRECTED;
+	}
+
+op_check:
+	switch (ev->op) {
+	case RTE_EVENT_OP_NEW:
+		/* Check that a sw credit is available */
+		if (dlb2_check_enqueue_sw_credits(dlb2, ev_port)) {
+			rte_errno = -ENOSPC;
+			return 1;
+		}
+		ev_port->inflight_credits--;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_FORWARD:
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+		(*cached_credits)--;
+		break;
+	case RTE_EVENT_OP_RELEASE:
+		ev_port->inflight_credits++;
+		/* Check for outstanding_releases underflow. If this occurs,
+		 * the application is not using the EVENT_OPs correctly; for
+		 * example, forwarding or releasing events that were not
+		 * dequeued.
+		 */
+		RTE_ASSERT(ev_port->outstanding_releases > 0);
+		ev_port->outstanding_releases--;
+		qm_port->issued_releases++;
+
+		/* Replenish s/w credits if enough are cached */
+		dlb2_replenish_sw_credits(dlb2, ev_port);
+		break;
+	}
+
+	DLB2_INC_STAT(ev_port->stats.tx_op_cnt[ev->op], 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.tx_ok, 1);
+
+#ifndef RTE_LIBRTE_PMD_DLB2_QUELL_STATS
+	if (ev->op != RTE_EVENT_OP_RELEASE) {
+		DLB2_INC_STAT(ev_port->stats.queue[ev->queue_id].enq_ok, 1);
+		DLB2_INC_STAT(ev_port->stats.tx_sched_cnt[*sched_type], 1);
+	}
+#endif
+
+	return 0;
+}
+
+static inline uint16_t
+dlb2_event_enqueue_burst(void *event_port,
+			 const struct rte_event events[],
+			 uint16_t num)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct process_local_port_data *port_data;
+	int i, cnt;
+
+	RTE_ASSERT(ev_port->enq_configured);
+	RTE_ASSERT(events != NULL);
+
+	cnt = 0;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	for (i = 0; i < num; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		uint8_t sched_types[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t queue_ids[DLB2_NUM_QES_PER_CACHE_LINE];
+		int j = 0;
+
+		for (; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) {
+			const struct rte_event *ev = &events[i + j];
+
+			if (dlb2_event_enqueue_prep(ev_port, qm_port, ev,
+						    &sched_types[j],
+						    &queue_ids[j]))
+				break;
+		}
+
+		if (j == 0)
+			break;
+
+		dlb2_event_build_hcws(qm_port, &events[i], j,
+				      sched_types, queue_ids);
+
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+
+		if (j < DLB2_NUM_QES_PER_CACHE_LINE)
+			break;
+	}
+
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
+	return cnt;
+}
+
+static inline uint16_t
+dlb2_event_enqueue(void *event_port,
+		   const struct rte_event events[])
+{
+	return dlb2_event_enqueue_burst(event_port, events, 1);
+}
+
+static uint16_t
+dlb2_event_enqueue_new_burst(void *event_port,
+			     const struct rte_event events[],
+			     uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
+static uint16_t
+dlb2_event_enqueue_forward_burst(void *event_port,
+				 const struct rte_event events[],
+				 uint16_t num)
+{
+	return dlb2_event_enqueue_burst(event_port, events, num);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -2188,7 +2774,13 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
 	};
 
+	/* Expose PMD's eventdev interface */
+
 	dev->dev_ops = &dlb2_eventdev_entry_ops;
+	dev->enqueue = dlb2_event_enqueue;
+	dev->enqueue_burst = dlb2_event_enqueue_burst;
+	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
+	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
 }
 
 int
-- 
2.6.4

Re: [dpdk-dev] [PATCH 15/22] event/dlb2: add enqueue and its burst variants

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 15/22] event/dlb2: add enqueue and its burst variants
> 
> Add support for enqueue and its variants.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH 16/22] event/dlb2: add dequeue and its burst variants

[Timothy McDaniel]

Add support for dequeue, dequeue_burst, ...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c | 786 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 786 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index ac4cf19..8911c55 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -2751,9 +2751,786 @@ dlb2_event_enqueue_forward_burst(void *event_port,
 	return dlb2_event_enqueue_burst(event_port, events, num);
 }
 
+static inline void
+dlb2_port_credits_inc(struct dlb2_port *qm_port, int num)
+{
+	uint32_t batch_size = DLB2_SW_CREDIT_BATCH_SZ;
+
+	/* increment port credits, and return to pool if exceeds threshold */
+	if (!qm_port->is_directed) {
+		qm_port->cached_ldb_credits += num;
+		if (qm_port->cached_ldb_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_LDB_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_ldb_credits -= batch_size;
+		}
+	} else {
+		qm_port->cached_dir_credits += num;
+		if (qm_port->cached_dir_credits >= 2 * batch_size) {
+			__atomic_fetch_add(
+				qm_port->credit_pool[DLB2_DIR_QUEUE],
+				batch_size, __ATOMIC_SEQ_CST);
+			qm_port->cached_dir_credits -= batch_size;
+		}
+	}
+}
+
+static inline bool
+dlb2_cq_is_empty(struct dlb2_port *qm_port)
+{
+	volatile struct dlb2_dequeue_qe *qe_ptr;
+	struct dlb2_dequeue_qe qe;
+
+	qe_ptr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	qe = qe_ptr[qm_port->cq_idx];
+
+	return (qe.cq_gen != qm_port->gen_bit);
+}
+
+static inline int
+dlb2_dequeue_wait(struct dlb2_eventdev *dlb2,
+		  struct dlb2_eventdev_port *ev_port,
+		  struct dlb2_port *qm_port,
+		  uint64_t timeout,
+		  uint64_t start_ticks)
+{
+	struct process_local_port_data *port_data;
+	uint64_t elapsed_ticks;
+
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	elapsed_ticks = rte_get_timer_cycles() - start_ticks;
+
+	/* Wait/poll time expired */
+	if (elapsed_ticks >= timeout) {
+		return 1;
+	} else if (dlb2->umwait_allowed) {
+		volatile struct dlb2_dequeue_qe *cq_base;
+
+		cq_base = port_data->cq_base;
+
+		/* Block on cache line write to CQ. Note: it's
+		 * safe to access the per-process cq_base
+		 * address here, since the PMD has already
+		 * attempted at least one CQ dequeue.
+		 */
+		dlb2_umonitor(&cq_base[qm_port->cq_idx]);
+
+		/* Avoid race condition. Check if still empty */
+		if (dlb2_cq_is_empty(qm_port)) {
+			dlb2_umwait(RTE_LIBRTE_PMD_DLB2_UMWAIT_CTL_STATE,
+				    timeout + start_ticks);
+			DLB2_INC_STAT(
+				ev_port->stats.traffic.rx_umonitor_umwait, 1);
+		}
+	} else {
+		uint64_t poll_interval = RTE_LIBRTE_PMD_DLB2_POLL_INTERVAL;
+		uint64_t curr_ticks = rte_get_timer_cycles();
+		uint64_t init_ticks = curr_ticks;
+
+		while ((curr_ticks - start_ticks < timeout) &&
+		       (curr_ticks - init_ticks < poll_interval))
+			curr_ticks = rte_get_timer_cycles();
+	}
+
+	return 0;
+}
+
+static inline int
+dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
+			 struct dlb2_port *qm_port,
+			 struct rte_event *events,
+			 struct dlb2_dequeue_qe *qes,
+			 int cnt)
+{
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	int i, num, evq_id;
+
+	RTE_SET_USED(ev_port);  /* avoids unused variable error if stats off */
+
+	for (i = 0, num = 0; i < cnt; i++) {
+		struct dlb2_dequeue_qe *qe = &qes[i];
+		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
+			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+			[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+		};
+
+		/* Fill in event information.
+		 * Note that flow_id must be embedded in the data by
+		 * the app, such as the mbuf RSS hash field if the data
+		 * buffer is a mbuf.
+		 */
+		if (unlikely(qe->error)) {
+			DLB2_LOG_ERR("QE error bit ON\n");
+			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
+			dlb2_consume_qe_immediate(qm_port, 1);
+			continue; /* Ignore */
+		}
+
+		events[num].u64 = qe->data;
+		events[num].flow_id = qe->flow_id;
+		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
+		events[num].event_type = qe->u.event_type.major;
+		events[num].sub_event_type = qe->u.event_type.sub;
+		events[num].sched_type = sched_type_map[qe->sched_type];
+		events[num].impl_opaque = qe->qid_depth;
+
+		/* qid not preserved for directed queues */
+		if (qm_port->is_directed)
+			evq_id = ev_port->link[0].queue_id;
+		else
+			evq_id = qid_mappings[qe->qid];
+
+		events[num].queue_id = evq_id;
+		DLB2_INC_STAT(
+			ev_port->stats.queue[evq_id].qid_depth[qe->qid_depth],
+			1);
+		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type], 1);
+		DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, 1);
+		num++;
+	}
+
+	return num;
+}
+
+static inline int
+dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
+			      struct dlb2_port *qm_port,
+			      struct rte_event *events,
+			      struct dlb2_dequeue_qe *qes)
+{
+	int sched_type_map[] = {
+		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
+		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
+		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
+		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
+	};
+	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
+	uint8_t *qid_mappings = qm_port->qid_mappings;
+	__m128i sse_evt[2];
+
+	RTE_SET_USED(ev_port);  /* avoids unused variable error, if stats off */
+
+	/* In the unlikely case that any of the QE error bits are set, process
+	 * them one at a time.
+	 */
+	if (unlikely(qes[0].error || qes[1].error ||
+		     qes[2].error || qes[3].error))
+		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
+						 qes, num_events);
+
+	events[0].u64 = qes[0].data;
+	events[1].u64 = qes[1].data;
+	events[2].u64 = qes[2].data;
+	events[3].u64 = qes[3].data;
+
+	/* Construct the metadata portion of two struct rte_events
+	 * in one 128b SSE register. Event metadata is constructed in the SSE
+	 * registers like so:
+	 * sse_evt[0][63:0]:   event[0]'s metadata
+	 * sse_evt[0][127:64]: event[1]'s metadata
+	 * sse_evt[1][63:0]:   event[2]'s metadata
+	 * sse_evt[1][127:64]: event[3]'s metadata
+	 */
+	sse_evt[0] = _mm_setzero_si128();
+	sse_evt[1] = _mm_setzero_si128();
+
+	/* Convert the hardware queue ID to an event queue ID and store it in
+	 * the metadata:
+	 * sse_evt[0][47:40]   = qid_mappings[qes[0].qid]
+	 * sse_evt[0][111:104] = qid_mappings[qes[1].qid]
+	 * sse_evt[1][47:40]   = qid_mappings[qes[2].qid]
+	 * sse_evt[1][111:104] = qid_mappings[qes[3].qid]
+	 */
+#define RTE_EVENT_QUEUE_ID_BYTE 5
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[0].qid],
+				     RTE_EVENT_QUEUE_ID_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+				     qid_mappings[qes[1].qid],
+				     RTE_EVENT_QUEUE_ID_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[2].qid],
+				     RTE_EVENT_QUEUE_ID_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+				     qid_mappings[qes[3].qid],
+				     RTE_EVENT_QUEUE_ID_BYTE + 8);
+
+	/* Convert the hardware priority to an event priority and store it in
+	 * the metadata, while also returning the queue depth status
+	 * value captured by the hardware, storing it in impl_opaque, which can
+	 * be read by the application but not modified
+	 * sse_evt[0][55:48]   = DLB2_TO_EV_PRIO(qes[0].priority)
+	 * sse_evt[0][63:56]   = qes[0].qid_depth
+	 * sse_evt[0][119:112] = DLB2_TO_EV_PRIO(qes[1].priority)
+	 * sse_evt[0][127:120] = qes[1].qid_depth
+	 * sse_evt[1][55:48]   = DLB2_TO_EV_PRIO(qes[2].priority)
+	 * sse_evt[1][63:56]   = qes[2].qid_depth
+	 * sse_evt[1][119:112] = DLB2_TO_EV_PRIO(qes[3].priority)
+	 * sse_evt[1][127:120] = qes[3].qid_depth
+	 */
+#define RTE_EVENT_PRIO_IMPL_OPAQUE_WORD 3
+#define RTE_BYTE_SHIFT 8
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[0].priority) |
+			(qes[0].qid_depth << RTE_BYTE_SHIFT),
+			RTE_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[0] =
+		_mm_insert_epi16(sse_evt[0],
+			DLB2_TO_EV_PRIO((uint8_t)qes[1].priority) |
+			(qes[1].qid_depth << RTE_BYTE_SHIFT),
+			RTE_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[2].priority) |
+			(qes[2].qid_depth << RTE_BYTE_SHIFT),
+			RTE_EVENT_PRIO_IMPL_OPAQUE_WORD);
+	sse_evt[1] =
+		_mm_insert_epi16(sse_evt[1],
+			DLB2_TO_EV_PRIO((uint8_t)qes[3].priority) |
+			(qes[3].qid_depth << RTE_BYTE_SHIFT),
+			RTE_EVENT_PRIO_IMPL_OPAQUE_WORD + 4);
+
+	/* Write the event type, sub event type, and flow_id to the event
+	 * metadata.
+	 * sse_evt[0][31:0]   = qes[0].flow_id |
+	 *			qes[0].u.event_type.major << 28 |
+	 *			qes[0].u.event_type.sub << 20;
+	 * sse_evt[0][95:64]  = qes[1].flow_id |
+	 *			qes[1].u.event_type.major << 28 |
+	 *			qes[1].u.event_type.sub << 20;
+	 * sse_evt[1][31:0]   = qes[2].flow_id |
+	 *			qes[2].u.event_type.major << 28 |
+	 *			qes[2].u.event_type.sub << 20;
+	 * sse_evt[1][95:64]  = qes[3].flow_id |
+	 *			qes[3].u.event_type.major << 28 |
+	 *			qes[3].u.event_type.sub << 20;
+	 */
+#define RTE_EVENT_EV_TYPE_DW 0
+#define RTE_EVENT_EV_TYPE_SHIFT 28
+#define RTE_EVENT_SUB_EV_TYPE_SHIFT 20
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[0].flow_id |
+			qes[0].u.event_type.major << RTE_EVENT_EV_TYPE_SHIFT |
+			qes[0].u.event_type.sub <<  RTE_EVENT_SUB_EV_TYPE_SHIFT,
+			RTE_EVENT_EV_TYPE_DW);
+	sse_evt[0] = _mm_insert_epi32(sse_evt[0],
+			qes[1].flow_id |
+			qes[1].u.event_type.major << RTE_EVENT_EV_TYPE_SHIFT |
+			qes[1].u.event_type.sub <<  RTE_EVENT_SUB_EV_TYPE_SHIFT,
+			RTE_EVENT_EV_TYPE_DW + 2);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[2].flow_id |
+			qes[2].u.event_type.major << RTE_EVENT_EV_TYPE_SHIFT |
+			qes[2].u.event_type.sub <<  RTE_EVENT_SUB_EV_TYPE_SHIFT,
+			RTE_EVENT_EV_TYPE_DW);
+	sse_evt[1] = _mm_insert_epi32(sse_evt[1],
+			qes[3].flow_id |
+			qes[3].u.event_type.major << RTE_EVENT_EV_TYPE_SHIFT  |
+			qes[3].u.event_type.sub << RTE_EVENT_SUB_EV_TYPE_SHIFT,
+			RTE_EVENT_EV_TYPE_DW + 2);
+
+	/* Write the sched type to the event metadata. 'op' and 'rsvd' are not
+	 * set:
+	 * sse_evt[0][39:32]  = sched_type_map[qes[0].sched_type] << 6
+	 * sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6
+	 * sse_evt[1][39:32]  = sched_type_map[qes[2].sched_type] << 6
+	 * sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6
+	 */
+#define RTE_EVENT_SCHED_TYPE_BYTE 4
+#define RTE_EVENT_SCHED_TYPE_SHIFT 6
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[0].sched_type] << RTE_EVENT_SCHED_TYPE_SHIFT,
+		RTE_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[0] = _mm_insert_epi8(sse_evt[0],
+		sched_type_map[qes[1].sched_type] << RTE_EVENT_SCHED_TYPE_SHIFT,
+		RTE_EVENT_SCHED_TYPE_BYTE + 8);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[2].sched_type] << RTE_EVENT_SCHED_TYPE_SHIFT,
+		RTE_EVENT_SCHED_TYPE_BYTE);
+	sse_evt[1] = _mm_insert_epi8(sse_evt[1],
+		sched_type_map[qes[3].sched_type] << RTE_EVENT_SCHED_TYPE_SHIFT,
+		RTE_EVENT_SCHED_TYPE_BYTE + 8);
+
+	/* Store the metadata to the event (use the double-precision
+	 * _mm_storeh_pd because there is no integer function for storing the
+	 * upper 64b):
+	 * events[0].event = sse_evt[0][63:0]
+	 * events[1].event = sse_evt[0][127:64]
+	 * events[2].event = sse_evt[1][63:0]
+	 * events[3].event = sse_evt[1][127:64]
+	 */
+	_mm_storel_epi64((__m128i *)&events[0].event, sse_evt[0]);
+	_mm_storeh_pd((double *)&events[1].event, (__m128d) sse_evt[0]);
+	_mm_storel_epi64((__m128i *)&events[2].event, sse_evt[1]);
+	_mm_storeh_pd((double *)&events[3].event, (__m128d) sse_evt[1]);
+
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[0].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[1].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[2].sched_type], 1);
+	DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qes[3].sched_type], 1);
+
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[0].queue_id].
+			qid_depth[qes[0].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[1].queue_id].
+			qid_depth[qes[1].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[2].queue_id].
+			qid_depth[qes[2].qid_depth],
+		1);
+	DLB2_INC_STAT(
+		ev_port->stats.queue[events[3].queue_id].
+			qid_depth[qes[3].qid_depth],
+		1);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, num_events);
+
+	return num_events;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe_sparse(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe)
+{
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	uint8_t xor_mask[2] = {0x0F, 0x00};
+	const uint8_t and_mask = 0x0F;
+	__m128i *qes = (__m128i *)qe;
+	uint8_t gen_bits, gen_bit;
+	uintptr_t addr[4];
+	uint16_t idx;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+
+	idx = qm_port->cq_idx;
+
+	/* Load the next 4 QEs */
+	addr[0] = (uintptr_t)&cq_addr[idx];
+	addr[1] = (uintptr_t)&cq_addr[(idx +  4) & qm_port->cq_depth_mask];
+	addr[2] = (uintptr_t)&cq_addr[(idx +  8) & qm_port->cq_depth_mask];
+	addr[3] = (uintptr_t)&cq_addr[(idx + 12) & qm_port->cq_depth_mask];
+
+	/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */
+	rte_prefetch0(&cq_addr[(idx + 16) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 20) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 24) & qm_port->cq_depth_mask]);
+	rte_prefetch0(&cq_addr[(idx + 28) & qm_port->cq_depth_mask]);
+
+	/* Correct the xor_mask for wrap-around QEs */
+	gen_bit = qm_port->gen_bit;
+	xor_mask[gen_bit] ^= !!((idx +  4) > qm_port->cq_depth_mask) << 1;
+	xor_mask[gen_bit] ^= !!((idx +  8) > qm_port->cq_depth_mask) << 2;
+	xor_mask[gen_bit] ^= !!((idx + 12) > qm_port->cq_depth_mask) << 3;
+
+	/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is
+	 * valid, then QEs[0:N-1] are too.
+	 */
+	qes[3] = _mm_load_si128((__m128i *)(void *)addr[3]);
+	rte_compiler_barrier();
+	qes[2] = _mm_load_si128((__m128i *)(void *)addr[2]);
+	rte_compiler_barrier();
+	qes[1] = _mm_load_si128((__m128i *)(void *)addr[1]);
+	rte_compiler_barrier();
+	qes[0] = _mm_load_si128((__m128i *)(void *)addr[0]);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[gen_bit];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask;
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline void
+dlb2_inc_cq_idx(struct dlb2_port *qm_port, int cnt)
+{
+	uint16_t idx = qm_port->cq_idx_unmasked + cnt;
+
+	qm_port->cq_idx_unmasked = idx;
+	qm_port->cq_idx = idx & qm_port->cq_depth_mask;
+	qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) & 0x1;
+}
+
+static int
+dlb2_event_release(struct dlb2_eventdev *dlb2,
+		   uint8_t port_id,
+		   int n)
+{
+	struct process_local_port_data *port_data;
+	struct dlb2_eventdev_port *ev_port;
+	struct dlb2_port *qm_port;
+	int i, cnt;
+
+	if (port_id > dlb2->num_ports) {
+		DLB2_LOG_ERR("Invalid port id %d in dlb2-event_release\n",
+			     port_id);
+		rte_errno = -EINVAL;
+		return rte_errno;
+	}
+
+	ev_port = &dlb2->ev_ports[port_id];
+	qm_port = &ev_port->qm_port;
+	port_data = &dlb2_port[qm_port->id][PORT_TYPE(qm_port)];
+
+	cnt = 0;
+
+	if (qm_port->is_directed) {
+		cnt = n;
+		goto sw_credit_update;
+	}
+
+	for (i = 0; i < n; i += DLB2_NUM_QES_PER_CACHE_LINE) {
+		int j;
+
+		/* Zero-out QEs */
+		qm_port->qe4[0].cmd_byte = 0;
+		qm_port->qe4[1].cmd_byte = 0;
+		qm_port->qe4[2].cmd_byte = 0;
+		qm_port->qe4[3].cmd_byte = 0;
+
+		for (j = 0; j < DLB2_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++)
+			qm_port->qe4[j].cmd_byte = DLB2_COMP_CMD_BYTE;
+
+		qm_port->issued_releases += j;
+
+		if (j == 0)
+			break;
+
+		if (qm_port->token_pop_mode == DELAYED_POP && j < 4 &&
+		    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+			dlb2_construct_token_pop_qe(qm_port, j);
+
+			/* Reset the releases counter for the next QE batch */
+			qm_port->issued_releases -= qm_port->token_pop_thresh;
+		}
+
+		dlb2_hw_do_enqueue(qm_port, i == 0, port_data);
+
+		cnt += j;
+	}
+
+	if (qm_port->token_pop_mode == DELAYED_POP &&
+	    qm_port->issued_releases >= qm_port->token_pop_thresh - 1) {
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+		qm_port->issued_releases -= qm_port->token_pop_thresh;
+	}
+
+sw_credit_update:
+	/* each release returns one credit */
+	if (!ev_port->outstanding_releases) {
+		DLB2_LOG_ERR("Unrecoverable application error. Outstanding releases underflowed.\n");
+		rte_errno = -ENOTRECOVERABLE;
+		return rte_errno;
+	}
+
+	ev_port->outstanding_releases -= cnt;
+	ev_port->inflight_credits += cnt;
+
+	/* Replenish s/w credits if enough releases are performed */
+	dlb2_replenish_sw_credits(dlb2, ev_port);
+	return 0;
+}
+
+static inline int16_t
+dlb2_hw_dequeue_sparse(struct dlb2_eventdev *dlb2,
+		       struct dlb2_eventdev_port *ev_port,
+		       struct rte_event *events,
+		       uint16_t max_num,
+		       uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe_sparse(qm_port, qes);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail << 2);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							      qm_port,
+							      &events[num],
+							      &qes[0]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							 qm_port,
+							 &events[num],
+							 &qes[0],
+							 num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static __rte_always_inline int
+dlb2_recv_qe(struct dlb2_port *qm_port, struct dlb2_dequeue_qe *qe,
+	     uint8_t *offset)
+{
+	uint8_t xor_mask[2][4] = { {0x0F, 0x0E, 0x0C, 0x08},
+				   {0x00, 0x01, 0x03, 0x07} };
+	uint8_t and_mask[4] = {0x0F, 0x0E, 0x0C, 0x08};
+	volatile struct dlb2_dequeue_qe *cq_addr;
+	__m128i *qes = (__m128i *)qe;
+	uint64_t *cache_line_base;
+	uint8_t gen_bits;
+
+	cq_addr = dlb2_port[qm_port->id][PORT_TYPE(qm_port)].cq_base;
+	cq_addr = &cq_addr[qm_port->cq_idx];
+
+	cache_line_base = (void *)(((uintptr_t)cq_addr) & ~0x3F);
+	*offset = ((uintptr_t)cq_addr & 0x30) >> 4;
+
+	/* Load the next CQ cache line from memory. Pack these reads as tight
+	 * as possible to reduce the chance that DLB invalidates the line while
+	 * the CPU is reading it. Read the cache line backwards to ensure that
+	 * if QE[N] (N > 0) is valid, then QEs[0:N-1] are too.
+	 *
+	 * (Valid QEs start at &qe[offset])
+	 */
+	qes[3] = _mm_load_si128((__m128i *)&cache_line_base[6]);
+	qes[2] = _mm_load_si128((__m128i *)&cache_line_base[4]);
+	qes[1] = _mm_load_si128((__m128i *)&cache_line_base[2]);
+	qes[0] = _mm_load_si128((__m128i *)&cache_line_base[0]);
+
+	/* Evict the cache line ASAP */
+	dlb2_cldemote(cache_line_base);
+
+	/* Extract and combine the gen bits */
+	gen_bits = ((_mm_extract_epi8(qes[0], 15) & 0x1) << 0) |
+		   ((_mm_extract_epi8(qes[1], 15) & 0x1) << 1) |
+		   ((_mm_extract_epi8(qes[2], 15) & 0x1) << 2) |
+		   ((_mm_extract_epi8(qes[3], 15) & 0x1) << 3);
+
+	/* XOR the combined bits such that a 1 represents a valid QE */
+	gen_bits ^= xor_mask[qm_port->gen_bit][*offset];
+
+	/* Mask off gen bits we don't care about */
+	gen_bits &= and_mask[*offset];
+
+	return __builtin_popcount(gen_bits);
+}
+
+static inline int16_t
+dlb2_hw_dequeue(struct dlb2_eventdev *dlb2,
+		struct dlb2_eventdev_port *ev_port,
+		struct rte_event *events,
+		uint16_t max_num,
+		uint64_t dequeue_timeout_ticks)
+{
+	uint64_t timeout;
+	uint64_t start_ticks = 0ULL;
+	struct dlb2_port *qm_port;
+	int num = 0;
+
+	qm_port = &ev_port->qm_port;
+
+	/* We have a special implementation for waiting. Wait can be:
+	 * 1) no waiting at all
+	 * 2) busy poll only
+	 * 3) wait for interrupt. If wakeup and poll time
+	 * has expired, then return to caller
+	 * 4) umonitor/umwait repeatedly up to poll time
+	 */
+
+	/* If configured for per dequeue wait, then use wait value provided
+	 * to this API. Otherwise we must use the global
+	 * value from eventdev config time.
+	 */
+	if (!dlb2->global_dequeue_wait)
+		timeout = dequeue_timeout_ticks;
+	else
+		timeout = dlb2->global_dequeue_wait_ticks;
+
+	start_ticks = rte_get_timer_cycles();
+
+	while (num < max_num) {
+		struct dlb2_dequeue_qe qes[DLB2_NUM_QES_PER_CACHE_LINE];
+		uint8_t offset;
+		int num_avail;
+
+		/* Copy up to 4 QEs from the current cache line into qes */
+		num_avail = dlb2_recv_qe(qm_port, qes, &offset);
+
+		/* But don't process more than the user requested */
+		num_avail = RTE_MIN(num_avail, max_num - num);
+
+		dlb2_inc_cq_idx(qm_port, num_avail);
+
+		if (num_avail == DLB2_NUM_QES_PER_CACHE_LINE)
+			num += dlb2_process_dequeue_four_qes(ev_port,
+							     qm_port,
+							     &events[num],
+							     &qes[offset]);
+		else if (num_avail)
+			num += dlb2_process_dequeue_qes(ev_port,
+							qm_port,
+							&events[num],
+							&qes[offset],
+							num_avail);
+		else if ((timeout == 0) || (num > 0))
+			/* Not waiting in any form, or 1+ events received? */
+			break;
+		else if (dlb2_dequeue_wait(dlb2, ev_port, qm_port,
+					   timeout, start_ticks))
+			break;
+	}
+
+	qm_port->owed_tokens += num;
+
+	if (num) {
+		if (qm_port->token_pop_mode == AUTO_POP)
+			dlb2_consume_qe_immediate(qm_port, num);
+
+		ev_port->outstanding_releases += num;
+
+		dlb2_port_credits_inc(qm_port, num);
+	}
+
+	return num;
+}
+
+static uint16_t
+dlb2_event_dequeue_burst(void *event_port, struct rte_event *ev, uint16_t num,
+			 uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
+	cnt = dlb2_hw_dequeue(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue(void *event_port, struct rte_event *ev, uint64_t wait)
+{
+	return dlb2_event_dequeue_burst(event_port, ev, 1, wait);
+}
+
+static uint16_t
+dlb2_event_dequeue_burst_sparse(void *event_port, struct rte_event *ev,
+				uint16_t num, uint64_t wait)
+{
+	struct dlb2_eventdev_port *ev_port = event_port;
+	struct dlb2_port *qm_port = &ev_port->qm_port;
+	struct dlb2_eventdev *dlb2 = ev_port->dlb2;
+	uint16_t cnt;
+
+	RTE_ASSERT(ev_port->setup_done);
+	RTE_ASSERT(ev != NULL);
+
+	if (ev_port->implicit_release && ev_port->outstanding_releases > 0) {
+		uint16_t out_rels = ev_port->outstanding_releases;
+
+		if (dlb2_event_release(dlb2, ev_port->id, out_rels))
+			return 0; /* rte_errno is set */
+
+		DLB2_INC_STAT(ev_port->stats.tx_implicit_rel, out_rels);
+	}
+
+	if (qm_port->token_pop_mode == DEFERRED_POP && qm_port->owed_tokens)
+		dlb2_consume_qe_immediate(qm_port, qm_port->owed_tokens);
+
+	cnt = dlb2_hw_dequeue_sparse(dlb2, ev_port, ev, num, wait);
+
+	DLB2_INC_STAT(ev_port->stats.traffic.total_polls, 1);
+	DLB2_INC_STAT(ev_port->stats.traffic.zero_polls, ((cnt == 0) ? 1 : 0));
+	return cnt;
+}
+
+static uint16_t
+dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
+			  uint64_t wait)
+{
+	return dlb2_event_dequeue_burst_sparse(event_port, ev, 1, wait);
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
+	struct dlb2_eventdev *dlb2;
+
 	/* Expose PMD's eventdev interface */
 	static struct rte_eventdev_ops dlb2_eventdev_entry_ops = {
 		.dev_infos_get    = dlb2_eventdev_info_get,
@@ -2781,6 +3558,15 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 	dev->enqueue_burst = dlb2_event_enqueue_burst;
 	dev->enqueue_new_burst = dlb2_event_enqueue_new_burst;
 	dev->enqueue_forward_burst = dlb2_event_enqueue_forward_burst;
+
+	dlb2 = dev->data->dev_private;
+	if (dlb2->poll_mode == DLB2_CQ_POLL_MODE_SPARSE) {
+		dev->dequeue = dlb2_event_dequeue_sparse;
+		dev->dequeue_burst = dlb2_event_dequeue_burst_sparse;
+	} else {
+		dev->dequeue = dlb2_event_dequeue;
+		dev->dequeue_burst = dlb2_event_dequeue_burst;
+	}
 }
 
 int
-- 
2.6.4

Re: [dpdk-dev] [PATCH 16/22] event/dlb2: add dequeue and its burst variants

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 16/22] event/dlb2: add dequeue and its burst variants
> 
> Add support for dequeue, dequeue_burst, ...

Please elaborate -- this commit message doesn't mention (e.g.) the use of
umonitor/umwait or the "sparse" dequeue function variants. I think the
average reviewer needs some more context in order to understand this change.

[...]

> +
> +static inline int
> +dlb2_process_dequeue_qes(struct dlb2_eventdev_port *ev_port,
> +			 struct dlb2_port *qm_port,
> +			 struct rte_event *events,
> +			 struct dlb2_dequeue_qe *qes,
> +			 int cnt)
> +{
> +	uint8_t *qid_mappings = qm_port->qid_mappings;
> +	int i, num, evq_id;
> +
> +	RTE_SET_USED(ev_port);  /* avoids unused variable error if stats off */

Looks like ev_port is used unconditionally later on: "evq_id = ev_port->link[0].queue_id;"

> +
> +	for (i = 0, num = 0; i < cnt; i++) {
> +		struct dlb2_dequeue_qe *qe = &qes[i];
> +		int sched_type_map[DLB2_NUM_HW_SCHED_TYPES] = {
> +			[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
> +			[DLB2_SCHED_UNORDERED] =
> RTE_SCHED_TYPE_PARALLEL,
> +			[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
> +			[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
> +		};
> +
> +		/* Fill in event information.
> +		 * Note that flow_id must be embedded in the data by
> +		 * the app, such as the mbuf RSS hash field if the data
> +		 * buffer is a mbuf.
> +		 */
> +		if (unlikely(qe->error)) {
> +			DLB2_LOG_ERR("QE error bit ON\n");
> +			DLB2_INC_STAT(ev_port->stats.traffic.rx_drop, 1);
> +			dlb2_consume_qe_immediate(qm_port, 1);
> +			continue; /* Ignore */
> +		}
> +
> +		events[num].u64 = qe->data;
> +		events[num].flow_id = qe->flow_id;
> +		events[num].priority = DLB2_TO_EV_PRIO((uint8_t)qe->priority);
> +		events[num].event_type = qe->u.event_type.major;
> +		events[num].sub_event_type = qe->u.event_type.sub;
> +		events[num].sched_type = sched_type_map[qe->sched_type];
> +		events[num].impl_opaque = qe->qid_depth;
> +
> +		/* qid not preserved for directed queues */
> +		if (qm_port->is_directed)
> +			evq_id = ev_port->link[0].queue_id;
> +		else
> +			evq_id = qid_mappings[qe->qid];
> +
> +		events[num].queue_id = evq_id;
> +		DLB2_INC_STAT(
> +			ev_port->stats.queue[evq_id].qid_depth[qe-
> >qid_depth],
> +			1);
> +		DLB2_INC_STAT(ev_port->stats.rx_sched_cnt[qe->sched_type],
> 1);
> +		DLB2_INC_STAT(ev_port->stats.traffic.rx_ok, 1);

Move this outside the loop and increment by 'num' rather than '1'?

> +		num++;
> +	}
> +
> +	return num;
> +}
> +
> +static inline int
> +dlb2_process_dequeue_four_qes(struct dlb2_eventdev_port *ev_port,
> +			      struct dlb2_port *qm_port,
> +			      struct rte_event *events,
> +			      struct dlb2_dequeue_qe *qes)
> +{
> +	int sched_type_map[] = {
> +		[DLB2_SCHED_ATOMIC] = RTE_SCHED_TYPE_ATOMIC,
> +		[DLB2_SCHED_UNORDERED] = RTE_SCHED_TYPE_PARALLEL,
> +		[DLB2_SCHED_ORDERED] = RTE_SCHED_TYPE_ORDERED,
> +		[DLB2_SCHED_DIRECTED] = RTE_SCHED_TYPE_ATOMIC,
> +	};
> +	const int num_events = DLB2_NUM_QES_PER_CACHE_LINE;
> +	uint8_t *qid_mappings = qm_port->qid_mappings;
> +	__m128i sse_evt[2];
> +
> +	RTE_SET_USED(ev_port);  /* avoids unused variable error, if stats off */

ev_port gets passed to dlb2_process_dequeue_qes, I don't think this line is necessary.

> +
> +	/* In the unlikely case that any of the QE error bits are set, process
> +	 * them one at a time.
> +	 */
> +	if (unlikely(qes[0].error || qes[1].error ||
> +		     qes[2].error || qes[3].error))
> +		return dlb2_process_dequeue_qes(ev_port, qm_port, events,
> +						 qes, num_events);

Thanks,
Gage

[dpdk-dev] [PATCH 17/22] event/dlb2: add eventdev stop and close

[Timothy McDaniel]

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c                  | 261 +++++++++++++++++++++++++++--
 drivers/event/dlb2/dlb2_iface.c            |   6 +
 drivers/event/dlb2/dlb2_iface.h            |   6 +
 drivers/event/dlb2/pf/base/dlb2_resource.c |  91 ++++++++++
 drivers/event/dlb2/pf/dlb2_pf.c            |  52 ++++++
 5 files changed, 401 insertions(+), 15 deletions(-)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 8911c55..43b85d7 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -77,21 +77,6 @@ static struct dlb2_port_low_level_io_functions qm_mmio_fns;
 struct process_local_port_data
 dlb2_port[DLB2_MAX_NUM_PORTS][DLB2_NUM_PORT_TYPES];
 
-/*
- * DUMMY - added so that xstats path will compile/link.
- * Will be replaced by real version in a subsequent
- * patch.
- */
-uint32_t
-dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
-		     struct dlb2_eventdev_queue *queue)
-{
-	RTE_SET_USED(dlb2);
-	RTE_SET_USED(queue);
-
-	return 0;
-}
-
 static void
 dlb2_free_qe_mem(struct dlb2_port *qm_port)
 {
@@ -3527,6 +3512,250 @@ dlb2_event_dequeue_sparse(void *event_port, struct rte_event *ev,
 }
 
 static void
+dlb2_flush_port(struct rte_eventdev *dev, int port_id)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	eventdev_stop_flush_t flush;
+	struct rte_event ev;
+	uint8_t dev_id;
+	void *arg;
+	int i;
+
+	flush = dev->dev_ops->dev_stop_flush;
+	dev_id = dev->data->dev_id;
+	arg = dev->data->dev_stop_flush_arg;
+
+	while (rte_event_dequeue_burst(dev_id, port_id, &ev, 1, 0)) {
+		if (flush)
+			flush(dev_id, ev, arg);
+
+		if (dlb2->ev_ports[port_id].qm_port.is_directed)
+			continue;
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+	}
+
+	/* Enqueue any additional outstanding releases */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = dlb2->ev_ports[port_id].outstanding_releases; i > 0; i--)
+		rte_event_enqueue_burst(dev_id, port_id, &ev, 1);
+}
+
+static uint32_t
+dlb2_get_ldb_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_ldb_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_ldb_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_ldb_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+static uint32_t
+dlb2_get_dir_queue_depth(struct dlb2_eventdev *dlb2,
+			 struct dlb2_eventdev_queue *queue)
+{
+	struct dlb2_hw_dev *handle = &dlb2->qm_instance;
+	struct dlb2_get_dir_queue_depth_args cfg;
+	int ret;
+
+	cfg.queue_id = queue->qm_queue.id;
+
+	ret = dlb2_iface_get_dir_queue_depth(handle, &cfg);
+	if (ret < 0) {
+		DLB2_LOG_ERR("dlb2: get_dir_queue_depth ret=%d (driver status: %s)\n",
+			     ret, dlb2_error_strings[cfg.response.status]);
+		return ret;
+	}
+
+	return cfg.response.id;
+}
+
+uint32_t
+dlb2_get_queue_depth(struct dlb2_eventdev *dlb2,
+		     struct dlb2_eventdev_queue *queue)
+{
+	if (queue->qm_queue.is_directed)
+		return dlb2_get_dir_queue_depth(dlb2, queue);
+	else
+		return dlb2_get_ldb_queue_depth(dlb2, queue);
+}
+
+static bool
+dlb2_queue_is_empty(struct dlb2_eventdev *dlb2,
+		    struct dlb2_eventdev_queue *queue)
+{
+	return dlb2_get_queue_depth(dlb2, queue) == 0;
+}
+
+static bool
+dlb2_linked_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (dlb2->ev_queues[i].num_links == 0)
+			continue;
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static bool
+dlb2_queues_empty(struct dlb2_eventdev *dlb2)
+{
+	int i;
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		if (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			return false;
+	}
+
+	return true;
+}
+
+static void
+dlb2_drain(struct rte_eventdev *dev)
+{
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+	struct dlb2_eventdev_port *ev_port = NULL;
+	uint8_t dev_id;
+	int i;
+
+	dev_id = dev->data->dev_id;
+
+	while (!dlb2_linked_queues_empty(dlb2)) {
+		/* Flush all the ev_ports, which will drain all their connected
+		 * queues.
+		 */
+		for (i = 0; i < dlb2->num_ports; i++)
+			dlb2_flush_port(dev, i);
+	}
+
+	/* The queues are empty, but there may be events left in the ports. */
+	for (i = 0; i < dlb2->num_ports; i++)
+		dlb2_flush_port(dev, i);
+
+	/* If the domain's queues are empty, we're done. */
+	if (dlb2_queues_empty(dlb2))
+		return;
+
+	/* Else, there must be at least one unlinked load-balanced queue.
+	 * Select a load-balanced port with which to drain the unlinked
+	 * queue(s).
+	 */
+	for (i = 0; i < dlb2->num_ports; i++) {
+		ev_port = &dlb2->ev_ports[i];
+
+		if (!ev_port->qm_port.is_directed)
+			break;
+	}
+
+	if (i == dlb2->num_ports) {
+		DLB2_LOG_ERR("internal error: no LDB ev_ports\n");
+		return;
+	}
+
+	rte_errno = 0;
+	rte_event_port_unlink(dev_id, ev_port->id, NULL, 0);
+
+	if (rte_errno) {
+		DLB2_LOG_ERR("internal error: failed to unlink ev_port %d\n",
+			     ev_port->id);
+		return;
+	}
+
+	for (i = 0; i < dlb2->num_queues; i++) {
+		uint8_t qid, prio;
+		int ret;
+
+		if (dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			continue;
+
+		qid = i;
+		prio = 0;
+
+		/* Link the ev_port to the queue */
+		ret = rte_event_port_link(dev_id, ev_port->id, &qid, &prio, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to link ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+
+		/* Flush the queue */
+		while (!dlb2_queue_is_empty(dlb2, &dlb2->ev_queues[i]))
+			dlb2_flush_port(dev, ev_port->id);
+
+		/* Drain any extant events in the ev_port. */
+		dlb2_flush_port(dev, ev_port->id);
+
+		/* Unlink the ev_port from the queue */
+		ret = rte_event_port_unlink(dev_id, ev_port->id, &qid, 1);
+		if (ret != 1) {
+			DLB2_LOG_ERR("internal error: failed to unlink ev_port %d to queue %d\n",
+				     ev_port->id, qid);
+			return;
+		}
+	}
+}
+
+static void
+dlb2_eventdev_stop(struct rte_eventdev *dev)
+{
+	/* FIXME: Handle the case that app threads are waiting in
+	 * rte_event_dequeue_burst() (either blocked on interrupt or in umwait,
+	 * or waiting with a timeout). Looking into proposing a new function to
+	 * wake blocked threads that the app must call before stopping a device.
+	 */
+	struct dlb2_eventdev *dlb2 = dlb2_pmd_priv(dev);
+
+	rte_spinlock_lock(&dlb2->qm_instance.resource_lock);
+
+	if (dlb2->run_state == DLB2_RUN_STATE_STOPPED) {
+		DLB2_LOG_DBG("Internal error: already stopped\n");
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	} else if (dlb2->run_state != DLB2_RUN_STATE_STARTED) {
+		DLB2_LOG_ERR("Internal error: bad state %d for dev_stop\n",
+			     (int)dlb2->run_state);
+		rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+		return;
+	}
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPING;
+
+	rte_spinlock_unlock(&dlb2->qm_instance.resource_lock);
+
+	dlb2_drain(dev);
+
+	dlb2->run_state = DLB2_RUN_STATE_STOPPED;
+}
+
+static int
+dlb2_eventdev_close(struct rte_eventdev *dev)
+{
+	dlb2_hw_reset_sched_domain(dev, false);
+
+	return 0;
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3536,6 +3765,8 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_infos_get    = dlb2_eventdev_info_get,
 		.dev_configure    = dlb2_eventdev_configure,
 		.dev_start        = dlb2_eventdev_start,
+		.dev_stop         = dlb2_eventdev_stop,
+		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
diff --git a/drivers/event/dlb2/dlb2_iface.c b/drivers/event/dlb2/dlb2_iface.c
index 8228bb3..eac2ea2 100644
--- a/drivers/event/dlb2/dlb2_iface.c
+++ b/drivers/event/dlb2/dlb2_iface.c
@@ -80,3 +80,9 @@ int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 
 int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				     struct dlb2_start_domain_args *cfg);
+
+int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
diff --git a/drivers/event/dlb2/dlb2_iface.h b/drivers/event/dlb2/dlb2_iface.h
index d5bb6be..bcd9446 100644
--- a/drivers/event/dlb2/dlb2_iface.h
+++ b/drivers/event/dlb2/dlb2_iface.h
@@ -66,4 +66,10 @@ extern int (*dlb2_iface_pending_port_unmaps)(struct dlb2_hw_dev *handle,
 extern int (*dlb2_iface_sched_domain_start)(struct dlb2_hw_dev *handle,
 				struct dlb2_start_domain_args *cfg);
 
+extern int (*dlb2_iface_get_ldb_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_ldb_queue_depth_args *args);
+
+extern int (*dlb2_iface_get_dir_queue_depth)(struct dlb2_hw_dev *handle,
+				struct dlb2_get_dir_queue_depth_args *args);
+
 #endif /* _DLB2_IFACE_H_ */
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 4c4c0b4..a835e97 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -5930,3 +5930,94 @@ dlb2_hw_start_domain(struct dlb2_hw *hw,
 
 	return 0;
 }
+
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_dir_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_dir_pq_pair *queue;
+	struct dlb2_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (!domain) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (!queue) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_dir_queue_depth(hw, queue);
+
+	return 0;
+}
+
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+					 u32 domain_id,
+					 u32 queue_id,
+					 bool vdev_req,
+					 unsigned int vf_id)
+{
+	DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+				u32 domain_id,
+				struct dlb2_get_ldb_queue_depth_args *args,
+				struct dlb2_cmd_response *resp,
+				bool vdev_req,
+				unsigned int vdev_id)
+{
+	struct dlb2_hw_domain *domain;
+	struct dlb2_ldb_queue *queue;
+
+	dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				     vdev_req, vdev_id);
+
+	domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (!domain) {
+		resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (!queue) {
+		resp->status = DLB2_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/pf/dlb2_pf.c b/drivers/event/dlb2/pf/dlb2_pf.c
index 084ba64..a901a0e 100644
--- a/drivers/event/dlb2/pf/dlb2_pf.c
+++ b/drivers/event/dlb2/pf/dlb2_pf.c
@@ -532,6 +532,56 @@ dlb2_pf_sched_domain_start(struct dlb2_hw_dev *handle,
 	return ret;
 }
 
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_ldb_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_ldb_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw_dev *handle,
+			    struct dlb2_get_dir_queue_depth_args *args)
+{
+	struct dlb2_dev *dlb2_dev = (struct dlb2_dev *)handle->pf_dev;
+	struct dlb2_cmd_response response = {0};
+	int ret = 0;
+
+	DLB2_INFO(dev->dlb2_device, "Entering %s()\n", __func__);
+
+	ret = dlb2_hw_get_dir_queue_depth(&dlb2_dev->hw,
+					  handle->domain_id,
+					  args,
+					  &response,
+					  false,
+					  0);
+
+	args->response = response;
+
+	DLB2_INFO(dev->dlb2_device, "Exiting %s() with ret=%d\n",
+		  __func__, ret);
+
+	return ret;
+}
+
 static void
 dlb2_pf_iface_fn_ptrs_init(void)
 {
@@ -550,6 +600,8 @@ dlb2_pf_iface_fn_ptrs_init(void)
 	dlb2_iface_dir_port_create = dlb2_pf_dir_port_create;
 	dlb2_iface_map_qid = dlb2_pf_map_qid;
 	dlb2_iface_unmap_qid = dlb2_pf_unmap_qid;
+	dlb2_iface_get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth;
+	dlb2_iface_get_dir_queue_depth = dlb2_pf_get_dir_queue_depth;
 	dlb2_iface_sched_domain_start = dlb2_pf_sched_domain_start;
 	dlb2_iface_pending_port_unmaps = dlb2_pf_pending_port_unmaps;
 	dlb2_iface_get_sn_allocation = dlb2_pf_get_sn_allocation;
-- 
2.6.4

Re: [dpdk-dev] [PATCH 17/22] event/dlb2: add eventdev stop and close

[Eads, Gage]

> +static void
> +dlb2_eventdev_stop(struct rte_eventdev *dev)
> +{
> +	/* FIXME: Handle the case that app threads are waiting in
> +	 * rte_event_dequeue_burst() (either blocked on interrupt or in umwait,
> +	 * or waiting with a timeout). Looking into proposing a new function to
> +	 * wake blocked threads that the app must call before stopping a device.
> +	 */

Leftover FIXME -- otherwise looks good.

Thanks,
Gage

[dpdk-dev] [PATCH 18/22] event/dlb2: add PMD's token pop public interface

[Timothy McDaniel]

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/meson.build                    |  5 ++-
 drivers/event/dlb2/rte_pmd_dlb2.c                 | 39 +++++++++++++++++++++++
 drivers/event/dlb2/rte_pmd_dlb2_event_version.map |  6 ++++
 3 files changed, 49 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c

diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 492452e..4549a75 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -1,3 +1,4 @@
+
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
@@ -6,7 +7,9 @@ sources = files('dlb2.c',
 		'dlb2_xstats.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
-		'pf/base/dlb2_resource.c'
+		'pf/base/dlb2_resource.c',
+		'rte_pmd_dlb2.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'bus_vdev', 'pci', 'bus_pci']
+install_headers('rte_pmd_dlb2.h')
\ No newline at end of file
diff --git a/drivers/event/dlb2/rte_pmd_dlb2.c b/drivers/event/dlb2/rte_pmd_dlb2.c
new file mode 100644
index 0000000..b09b585
--- /dev/null
+++ b/drivers/event/dlb2/rte_pmd_dlb2.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include <rte_eventdev.h>
+#include <rte_eventdev_pmd.h>
+
+#include "rte_pmd_dlb2.h"
+#include "dlb2_priv.h"
+#include "dlb2_inline_fns.h"
+
+int
+rte_pmd_dlb2_set_token_pop_mode(uint8_t dev_id,
+				uint8_t port_id,
+				enum dlb2_token_pop_mode mode)
+{
+	struct dlb2_eventdev *dlb2;
+	struct rte_eventdev *dev;
+
+	RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+	dev = &rte_eventdevs[dev_id];
+
+	dlb2 = dlb2_pmd_priv(dev);
+
+	if (mode >= NUM_TOKEN_POP_MODES)
+		return -EINVAL;
+
+	/* The event device must be configured, but not yet started */
+	if (!dlb2->configured || dlb2->run_state != DLB2_RUN_STATE_STOPPED)
+		return -EINVAL;
+
+	/* The token pop mode must be set before configuring the port */
+	if (port_id >= dlb2->num_ports || dlb2->ev_ports[port_id].setup_done)
+		return -EINVAL;
+
+	dlb2->ev_ports[port_id].qm_port.token_pop_mode = mode;
+
+	return 0;
+}
diff --git a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
index 299ae63..84b81a4 100644
--- a/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
+++ b/drivers/event/dlb2/rte_pmd_dlb2_event_version.map
@@ -1,3 +1,9 @@
 DPDK_21.0 {
 	local: *;
 };
+
+EXPERIMENTAL {
+	global:
+
+	rte_pmd_dlb2_set_token_pop_mode;
+};
-- 
2.6.4

Re: [dpdk-dev] [PATCH 18/22] event/dlb2: add PMD's token pop public interface

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> To: Ray Kinsella <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 18/22] event/dlb2: add PMD's token pop public interface
> 
> The PMD uses a public interface to allow applications to
> control the token pop mode. Supported token pop modes are
> as follows, and they impact core scheduling affinity for
> ldb ports.
> 
> AUTO_POP: Pop the CQ tokens immediately after dequeueing.
> DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
>              are released. Supported on load-balanced ports
>              only.
> DEFERRED_POP: Pop the CQ tokens during next dequeue operation.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  drivers/event/dlb2/meson.build                    |  5 ++-
>  drivers/event/dlb2/rte_pmd_dlb2.c                 | 39 +++++++++++++++++++++++
>  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |  6 ++++

Should rte_pmd_dlb2.h be listed in doc/api/doxy-api-index.md, so it's included
in the doxygen docs?

Besides that, and the build error related to the experimental tag mentioned in
another review, this looks good to me.

Thanks,
Gage

[dpdk-dev] [PATCH 19/22] event/dlb2: add PMD self-tests

[Timothy McDaniel]

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 app/test/test_eventdev.c           |    9 +
 drivers/event/dlb2/dlb2.c          |    1 +
 drivers/event/dlb2/dlb2_selftest.c | 1570 ++++++++++++++++++++++++++++++++++++
 drivers/event/dlb2/meson.build     |    3 +-
 4 files changed, 1582 insertions(+), 1 deletion(-)
 create mode 100644 drivers/event/dlb2/dlb2_selftest.c

diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
index 43ccb1c..b8d8df8 100644
--- a/app/test/test_eventdev.c
+++ b/app/test/test_eventdev.c
@@ -1030,6 +1030,13 @@ test_eventdev_selftest_dpaa2(void)
 	return test_eventdev_selftest_impl("event_dpaa2", "");
 }
 
+static int
+test_eventdev_selftest_dlb2(void)
+{
+	return test_eventdev_selftest_impl("dlb2_event", "");
+}
+
+
 REGISTER_TEST_COMMAND(eventdev_common_autotest, test_eventdev_common);
 REGISTER_TEST_COMMAND(eventdev_selftest_sw, test_eventdev_selftest_sw);
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
@@ -1037,3 +1044,5 @@ REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
 REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
 		test_eventdev_selftest_octeontx2);
 REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2, test_eventdev_selftest_dpaa2);
+REGISTER_TEST_COMMAND(eventdev_selftest_dlb2, test_eventdev_selftest_dlb2);
+
diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 43b85d7..620a0a5 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3780,6 +3780,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
 		.xstats_get_by_name = dlb2_eventdev_xstats_get_by_name,
 		.xstats_reset	    = dlb2_eventdev_xstats_reset,
+		.dev_selftest     = test_dlb2_eventdev,
 	};
 
 	/* Expose PMD's eventdev interface */
diff --git a/drivers/event/dlb2/dlb2_selftest.c b/drivers/event/dlb2/dlb2_selftest.c
new file mode 100644
index 0000000..86d6344
--- /dev/null
+++ b/drivers/event/dlb2/dlb2_selftest.c
@@ -0,0 +1,1570 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <unistd.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_eventdev.h>
+#include <rte_pause.h>
+
+#include "dlb2_priv.h"
+#include "rte_pmd_dlb2.h"
+
+#define MAX_PORTS 32
+#define MAX_QIDS 32
+#define DEFAULT_NUM_SEQ_NUMS 64
+
+static struct rte_mempool *eventdev_func_mempool;
+static int evdev;
+
+struct test {
+	struct rte_mempool *mbuf_pool;
+	int nb_qids;
+};
+
+/* initialization and config */
+static inline int
+init(struct test *t, int nb_queues, int nb_ports)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	memset(t, 0, sizeof(*t));
+
+	t->mbuf_pool = eventdev_func_mempool;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = nb_queues;
+	config.nb_event_ports = nb_ports;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0)
+		printf("%d: Error configuring device\n", __LINE__);
+
+	return ret;
+}
+
+static inline int
+create_ports(int num_ports)
+{
+	int i;
+
+	if (num_ports > MAX_PORTS)
+		return -1;
+
+	for (i = 0; i < num_ports; i++) {
+		struct rte_event_port_conf conf;
+
+		if (rte_event_port_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default port conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		if (rte_event_port_setup(evdev, i, &conf) < 0) {
+			printf("%d: Error setting up port %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static inline int
+create_lb_qids(struct test *t, int num_qids, uint32_t flags)
+{
+	int i;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		struct rte_event_queue_conf conf;
+
+		if (rte_event_queue_default_conf_get(evdev, i, &conf)) {
+			printf("%d: Error querying default queue conf\n",
+			       __LINE__);
+			return -1;
+		}
+
+		conf.schedule_type = flags;
+
+		if (conf.schedule_type == RTE_SCHED_TYPE_PARALLEL)
+			conf.nb_atomic_order_sequences = 0;
+		else
+			conf.nb_atomic_order_sequences = DEFAULT_NUM_SEQ_NUMS;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+static inline int
+create_atomic_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ATOMIC);
+}
+
+static inline int
+create_ordered_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_ORDERED);
+}
+
+static inline int
+create_unordered_qids(struct test *t, int num_qids)
+{
+	return create_lb_qids(t, num_qids, RTE_SCHED_TYPE_PARALLEL);
+}
+
+static inline int
+create_directed_qids(struct test *t, int num_qids, const uint8_t ports[])
+{
+	static struct rte_event_queue_conf conf;
+	int i;
+
+	conf.priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
+	conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) {
+		uint8_t queue_id;
+
+		if (rte_event_queue_setup(evdev, i, &conf) < 0) {
+			printf("%d: error creating qid %d\n", __LINE__, i);
+			return -1;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, ports[i - t->nb_qids],
+					&queue_id, NULL, 1) != 1) {
+			printf("%d: error creating link for qid %d\n",
+			       __LINE__, i);
+			return -1;
+		}
+	}
+
+	t->nb_qids += num_qids;
+	if (t->nb_qids > MAX_QIDS)
+		return -1;
+
+	return 0;
+}
+
+/* destruction */
+static inline void
+cleanup(struct test *t __rte_unused)
+{
+	int ret = 0;
+
+	rte_event_dev_stop(evdev);
+	ret = rte_event_dev_close(evdev);
+
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+};
+
+static inline int
+enqueue_timeout(uint8_t port_id, struct rte_event *ev, uint64_t tmo_us)
+{
+	const uint64_t start = rte_get_timer_cycles();
+	const uint64_t ticks = (tmo_us * rte_get_timer_hz()) / 1E6;
+
+	while ((rte_get_timer_cycles() - start) < ticks) {
+		if (rte_event_enqueue_burst(evdev, port_id, ev, 1) == 1)
+			return 0;
+
+		if (rte_errno != -ENOSPC) {
+			printf("enqueue_burst returned rte_errno %d\n",
+			       rte_errno);
+			return -1;
+		}
+	}
+	printf("%s time out\n", __func__);
+	return -1;
+}
+
+static void
+flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused)
+{
+	rte_pktmbuf_free(event.mbuf);
+}
+
+static int
+test_stop_flush(struct test *t) /* test to check we can properly flush events */
+{
+	struct rte_event ev;
+	uint32_t dequeue_depth;
+	unsigned int i, count;
+	uint8_t queue_id;
+
+	ev.op = RTE_EVENT_OP_NEW;
+
+	if (init(t, 2, 1) < 0 ||
+	    create_ports(1) < 0 ||
+	    create_atomic_qids(t, 2) < 0) {
+		printf("%d: Error initializing device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_port_link(evdev, 0, NULL, NULL, 0) != 2) {
+		printf("%d: Error linking queues to the port\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: Error with start call\n", __LINE__);
+		goto err;
+	}
+
+	/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked
+	 * queue.
+	 */
+	queue_id = 1;
+
+	if (rte_event_port_unlink(evdev, 0, &queue_id, 1) != 1) {
+		printf("%d: Error unlinking queue 1 from port\n", __LINE__);
+		goto err;
+	}
+
+	count = rte_mempool_avail_count(t->mbuf_pool);
+
+	if (rte_event_port_attr_get(evdev,
+				    0,
+				    RTE_EVENT_PORT_ATTR_DEQ_DEPTH,
+				    &dequeue_depth)) {
+		printf("%d: Error retrieveing dequeue depth\n", __LINE__);
+		goto err;
+	}
+
+	/* Send QEs to queue 0 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 0;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Send QEs to queue 1 */
+	for (i = 0; i < dequeue_depth + 1; i++) {
+		ev.mbuf = rte_pktmbuf_alloc(t->mbuf_pool);
+		ev.queue_id = 1;
+		ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+
+		if (enqueue_timeout(0, &ev, 1000)) {
+			printf("%d: Error enqueuing events\n", __LINE__);
+			goto err;
+		}
+	}
+
+	/* Now the DLB is scheduling events from the port to the IQ, and at
+	 * least one event should be remaining in each queue.
+	 */
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, flush, NULL)) {
+		printf("%d: Error installing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	cleanup(t);
+
+	if (count != rte_mempool_avail_count(t->mbuf_pool)) {
+		printf("%d: Error executing the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_dev_stop_flush_callback_register(evdev, NULL, NULL)) {
+		printf("%d: Error uninstalling the flush callback\n", __LINE__);
+		goto err;
+	}
+
+	return 0;
+err:
+	cleanup(t);
+	return -1;
+}
+
+static int
+test_single_link(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Create a directed port */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) == 0) {
+		printf("%d: port 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Create a load-balanced port */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a directed queue */
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to create another directed queue */
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) == 0) {
+		printf("%d: queue 1 setup expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Create a load-balanced queue */
+	queue_conf.event_queue_cfg = 0;
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	/* Attempt to link directed and load-balanced resources */
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 0 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) == 1) {
+		printf("%d: port 1 link expected to fail\n", __LINE__);
+		goto err;
+	}
+
+	/* Link ports to queues */
+	queue_id = 0;
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	queue_id = 1;
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+#define NUM_LDB_PORTS 64
+#define NUM_LDB_QUEUES 32
+
+static int
+test_info_get(void)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_dev_info info;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	config.nb_event_ports = info.max_event_ports;
+	config.nb_event_queues = NUM_LDB_QUEUES + info.max_event_ports / 2;
+	config.nb_single_link_event_port_queues = info.max_event_ports / 2;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		goto err;
+	}
+
+	/* The DLB2 PMD only reports load-balanced ports and queues in its
+	 * info_get function. Confirm that these values don't include the
+	 * directed port or queue counts.
+	 */
+
+	if (info.max_event_ports != NUM_LDB_PORTS) {
+		printf("%d: Got %u ports, expected %u\n",
+		       __LINE__, info.max_event_ports, NUM_LDB_PORTS);
+		goto err;
+	}
+
+	if (info.max_event_queues != NUM_LDB_QUEUES) {
+		printf("%d: Got %u queues, expected %u\n",
+		       __LINE__, info.max_event_queues, NUM_LDB_QUEUES);
+		goto err;
+	}
+
+	ret = rte_event_dev_close(evdev);
+	if (ret) {
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+		return -1;
+	}
+	return 0;
+
+err:
+	ret = rte_event_dev_close(evdev);
+	if (ret)
+		printf("%d: rte_event_dev_close failed, ret = %d\n",
+			__LINE__, ret);
+
+	return -1;
+}
+
+static int
+test_reconfiguration_link(struct test *t)
+{
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 2;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 2 LDB queues */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 1, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+			printf("%d: P0->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P1->Q0 and Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+			printf("%d: P1->Q%d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Configure device with 2 DIR ports and 2 DIR queues */
+	config.nb_single_link_event_port_queues = 2;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	for (i = 0; i < 2; i++) {
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Link P0->Q0 and P1->Q1 */
+	for (i = 0; i < 2; i++) {
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: port %d link expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	/* Reconfigure device */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q1 */
+	queue_id = 1;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: P0->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Link P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: P1->Q%d link expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_queues = 5;
+	config.nb_event_ports = 5;
+	config.nb_single_link_event_port_queues = 1;
+
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < config.nb_event_queues - 1; i++) {
+		port_conf.event_port_cfg = 0;
+		queue_conf.event_queue_cfg = 0;
+
+		if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+			printf("%d: port %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+			printf("%d: queue %d setup expected to succeed\n",
+			       __LINE__, i);
+			goto err;
+		}
+
+		queue_id = i;
+
+		if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+			printf("%d: P%d->Q%d link expected to succeed\n",
+			       __LINE__, i, i);
+			goto err;
+		}
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_SINGLE_LINK;
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, i, &port_conf) < 0) {
+		printf("%d: port %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, i, &queue_conf) < 0) {
+		printf("%d: queue %d setup expected to succeed\n",
+		       __LINE__, i);
+		goto err;
+	}
+
+	queue_id = i;
+
+	if (rte_event_port_link(evdev, i, &queue_id, NULL, 1) != 1) {
+		printf("%d: P%d->Q%d link expected to succeed\n",
+		       __LINE__, i, i);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Stop the device */
+	rte_event_dev_stop(evdev);
+
+	config.nb_event_ports += 1;
+
+	/* Reconfigure device with 1 more load-balanced port */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error re-configuring device\n", __LINE__);
+		return -1;
+	}
+
+	port_conf.event_port_cfg = 0;
+
+	/* Configure the new port */
+	if (rte_event_port_setup(evdev, config.nb_event_ports - 1,
+				 &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	cleanup(t);
+	return 0;
+
+err:
+	cleanup(t);
+	return -1;
+}
+
+static int
+test_load_balanced_traffic(struct test *t)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup(t);
+	return 0;
+
+err:
+	cleanup(t);
+	return -1;
+}
+
+static int
+test_directed_traffic(struct test *t)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 1;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 DIR port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 1 NEW event */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 FORWARD event */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (ev.queue_id != 0) {
+		printf("%d: invalid dequeued event queue ID (%d)\n",
+		       __LINE__, ev.queue_id);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_FORWARD;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue and enqueue 1 RELEASE operation */
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: NEW enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup(t);
+	return 0;
+
+err:
+	cleanup(t);
+	return -1;
+}
+
+static int
+test_deferred_sched(struct test *t)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	const int num_events = 128;
+	struct rte_event ev;
+	uint8_t queue_id;
+	int ret, i;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 2;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 2 LDB ports and 1 queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 1, DEFERRED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.dequeue_depth = 1;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_setup(evdev, 1, &port_conf) < 0) {
+		printf("%d: port 1 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	queue_conf.schedule_type = RTE_SCHED_TYPE_PARALLEL;
+	queue_conf.nb_atomic_order_sequences = 0;
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 and P1->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_port_link(evdev, 1, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 1 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	/* Enqueue 128 NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_PARALLEL;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue one event from port 0 */
+	timeout = 0xFFFFFFFFF;
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+		printf("%d: event dequeue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Dequeue (and release) all other events from port 1. Deferred
+	 * scheduling ensures no other events are scheduled to port 0 without a
+	 * subsequent rte_event_dequeue_burst() call.
+	 */
+	for (i = 0; i < num_events - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 1, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+
+		ev.op = RTE_EVENT_OP_RELEASE;
+
+		if (rte_event_enqueue_burst(evdev, 1, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	cleanup(t);
+	return 0;
+
+err:
+	cleanup(t);
+	return -1;
+}
+
+static int
+test_delayed_pop(struct test *t)
+{
+	uint64_t timeout;
+	struct rte_event_dev_config config = {0};
+	struct rte_event_queue_conf queue_conf;
+	struct rte_event_port_conf port_conf;
+	struct rte_event_dev_info info;
+	int ret, i, num_events;
+	struct rte_event ev;
+	uint8_t queue_id;
+
+	if (rte_event_dev_info_get(evdev, &info)) {
+		printf("%d: Error querying device info\n", __LINE__);
+		return -1;
+	}
+
+	config.nb_event_queues = 1;
+	config.nb_event_ports = 1;
+	config.nb_single_link_event_port_queues = 0;
+	config.nb_event_queue_flows = info.max_event_queue_flows;
+	config.nb_events_limit = info.max_num_events;
+	config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth;
+	config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth;
+	config.dequeue_timeout_ns = info.max_dequeue_timeout_ns;
+	config.event_dev_cfg = RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT;
+
+	/* Configure the device with 1 LDB port and queue */
+	ret = rte_event_dev_configure(evdev, &config);
+	if (ret < 0) {
+		printf("%d: Error configuring device\n", __LINE__);
+		return -1;
+	}
+
+	ret = rte_pmd_dlb2_set_token_pop_mode(evdev, 0, DELAYED_POP);
+	if (ret < 0) {
+		printf("%d: Error setting deferred scheduling\n", __LINE__);
+		goto err;
+	}
+
+	/* Configure the ports and queues */
+	if (rte_event_port_default_conf_get(evdev, 0, &port_conf)) {
+		printf("%d: Error querying default port conf\n", __LINE__);
+		goto err;
+	}
+
+	port_conf.event_port_cfg = RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL;
+
+	if (rte_event_port_setup(evdev, 0, &port_conf) < 0) {
+		printf("%d: port 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_default_conf_get(evdev, 0, &queue_conf)) {
+		printf("%d: Error querying default queue conf\n", __LINE__);
+		goto err;
+	}
+
+	if (rte_event_queue_setup(evdev, 0, &queue_conf) < 0) {
+		printf("%d: queue 0 setup expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Link P0->Q0 */
+	queue_id = 0;
+
+	if (rte_event_port_link(evdev, 0, &queue_id, NULL, 1) != 1) {
+		printf("%d: port 0 link expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	/* Start the device */
+	if (rte_event_dev_start(evdev) < 0) {
+		printf("%d: device start failed\n", __LINE__);
+		goto err;
+	}
+
+	num_events = 2 * port_conf.dequeue_depth;
+
+	/* Enqueue 2 * dequeue_depth NEW events */
+	ev.op = RTE_EVENT_OP_NEW;
+	ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
+	ev.queue_id = 0;
+	ev.priority = 0;
+	ev.u64 = 0;
+
+	for (i = 0; i < num_events; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: NEW enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	/* Dequeue dequeue_depth events but only release dequeue_depth - 2.
+	 * Delayed pop won't perform the pop and no more events will be
+	 * scheduled.
+	 */
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	for (i = 0; i < port_conf.dequeue_depth - 2; i++) {
+		if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+			printf("%d: RELEASE enqueue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	ret = rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout);
+	if (ret != 0) {
+		printf("%d: event dequeue expected to fail (ret = %d)\n",
+		       __LINE__, ret);
+		goto err;
+	}
+
+	/* Release one more event. This will trigger the token pop, and
+	 * dequeue_depth - 1 more events will be scheduled to the device.
+	 */
+	ev.op = RTE_EVENT_OP_RELEASE;
+
+	if (rte_event_enqueue_burst(evdev, 0, &ev, 1) != 1) {
+		printf("%d: RELEASE enqueue expected to succeed\n",
+		       __LINE__);
+		goto err;
+	}
+
+	timeout = 0xFFFFFFFFF;
+
+	for (i = 0; i < port_conf.dequeue_depth - 1; i++) {
+		if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 1) {
+			printf("%d: event dequeue expected to succeed\n",
+			       __LINE__);
+			goto err;
+		}
+	}
+
+	timeout = 0x10000;
+
+	if (rte_event_dequeue_burst(evdev, 0, &ev, 1, timeout) != 0) {
+		printf("%d: event dequeue expected to fail\n",
+		       __LINE__);
+		goto err;
+	}
+
+	cleanup(t);
+	return 0;
+
+err:
+	cleanup(t);
+	return -1;
+}
+
+static int
+do_selftest(void)
+{
+	struct test t;
+	int ret;
+
+	/* Only create mbuf pool once, reuse for each test run */
+	if (!eventdev_func_mempool) {
+		eventdev_func_mempool =
+			rte_pktmbuf_pool_create("EVENTDEV_DLB2_ST_POOL",
+						(1 << 12), /* 4k buffers */
+						32 /*MBUF_CACHE_SIZE*/,
+						0,
+						512, /* use very small mbufs */
+						rte_socket_id());
+		if (!eventdev_func_mempool) {
+			printf("ERROR creating mempool\n");
+			goto test_fail;
+		}
+	}
+	t.mbuf_pool = eventdev_func_mempool;
+
+	printf("*** Running Stop Flush test...\n");
+	ret = test_stop_flush(&t);
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Single Link test...\n");
+	ret = test_single_link();
+	if (ret != 0) {
+		printf("ERROR - Single Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Info Get test...\n");
+	ret = test_info_get();
+	if (ret != 0) {
+		printf("ERROR - Stop Flush test FAILED.\n");
+		return ret;
+	}
+
+	printf("*** Running Reconfiguration Link test...\n");
+	ret = test_reconfiguration_link(&t);
+	if (ret != 0) {
+		printf("ERROR - Reconfiguration Link test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Load-Balanced Traffic test...\n");
+	ret = test_load_balanced_traffic(&t);
+	if (ret != 0) {
+		printf("ERROR - Load-Balanced Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Directed Traffic test...\n");
+	ret = test_directed_traffic(&t);
+	if (ret != 0) {
+		printf("ERROR - Directed Traffic test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Deferred Scheduling test...\n");
+	ret = test_deferred_sched(&t);
+	if (ret != 0) {
+		printf("ERROR - Deferred Scheduling test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	printf("*** Running Delayed Pop test...\n");
+	ret = test_delayed_pop(&t);
+	if (ret != 0) {
+		printf("ERROR - Delayed Pop test FAILED.\n");
+
+		goto test_fail;
+	}
+
+	return 0;
+
+test_fail:
+	return -1;
+}
+
+int
+test_dlb2_eventdev(void)
+{
+	const char *dlb2_eventdev_name = "dlb2_event";
+	uint8_t num_evdevs = rte_event_dev_count();
+	int i, ret = 0;
+	int found = 0, skipped = 0, passed = 0, failed = 0;
+	struct rte_event_dev_info info;
+
+	for (i = 0; found + skipped < num_evdevs && i < RTE_EVENT_MAX_DEVS;
+	     i++) {
+		ret = rte_event_dev_info_get(i, &info);
+		if (ret < 0)
+			continue;
+
+		/* skip non-dlb2 event devices */
+		if (strncmp(info.driver_name, dlb2_eventdev_name,
+			    sizeof(*info.driver_name)) != 0) {
+			skipped++;
+			continue;
+		}
+
+		evdev = rte_event_dev_get_dev_id(info.driver_name);
+		if (evdev < 0) {
+			printf("Could not get dev_id for eventdev with name %s, i=%d\n",
+			       info.driver_name, i);
+			skipped++;
+			continue;
+		}
+		found++;
+		printf("Running selftest on eventdev %s\n", info.driver_name);
+		ret = do_selftest();
+		if (ret == 0) {
+			passed++;
+			printf("Selftest passed for eventdev %s\n",
+			       info.driver_name);
+		} else {
+			failed++;
+			printf("Selftest failed for eventdev %s, err=%d\n",
+			       info.driver_name, ret);
+		}
+	}
+
+	printf("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n",
+	       found, skipped, passed, failed);
+	return ret;
+}
diff --git a/drivers/event/dlb2/meson.build b/drivers/event/dlb2/meson.build
index 4549a75..07a25dc 100644
--- a/drivers/event/dlb2/meson.build
+++ b/drivers/event/dlb2/meson.build
@@ -8,7 +8,8 @@ sources = files('dlb2.c',
 		'pf/dlb2_main.c',
 		'pf/dlb2_pf.c',
 		'pf/base/dlb2_resource.c',
-		'rte_pmd_dlb2.c'
+		'rte_pmd_dlb2.c',
+		'dlb2_selftest.c'
 )
 
 deps += ['mbuf', 'mempool', 'ring', 'bus_vdev', 'pci', 'bus_pci']
-- 
2.6.4

Re: [dpdk-dev] [PATCH 19/22] event/dlb2: add PMD self-tests

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> To: Jerin Jacob <jerinj@marvell.com>
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>
> Subject: [PATCH 19/22] event/dlb2: add PMD self-tests
> 
> Add a variety of self-tests for both ldb and directed
> ports/queues, as well as configure, start, stop, link, etc...
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  app/test/test_eventdev.c           |    9 +
>  drivers/event/dlb2/dlb2.c          |    1 +
>  drivers/event/dlb2/dlb2_selftest.c | 1570
> ++++++++++++++++++++++++++++++++++++
>  drivers/event/dlb2/meson.build     |    3 +-
>  4 files changed, 1582 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/event/dlb2/dlb2_selftest.c
> 
> diff --git a/app/test/test_eventdev.c b/app/test/test_eventdev.c
> index 43ccb1c..b8d8df8 100644
> --- a/app/test/test_eventdev.c
> +++ b/app/test/test_eventdev.c
> @@ -1030,6 +1030,13 @@ test_eventdev_selftest_dpaa2(void)
>  	return test_eventdev_selftest_impl("event_dpaa2", "");
>  }
> 
> +static int
> +test_eventdev_selftest_dlb2(void)
> +{
> +	return test_eventdev_selftest_impl("dlb2_event", "");
> +}
> +
> +

Nit: extra newline

>  REGISTER_TEST_COMMAND(eventdev_common_autotest,
> test_eventdev_common);
>  REGISTER_TEST_COMMAND(eventdev_selftest_sw,
> test_eventdev_selftest_sw);
>  REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
> @@ -1037,3 +1044,5 @@
> REGISTER_TEST_COMMAND(eventdev_selftest_octeontx,
>  REGISTER_TEST_COMMAND(eventdev_selftest_octeontx2,
>  		test_eventdev_selftest_octeontx2);
>  REGISTER_TEST_COMMAND(eventdev_selftest_dpaa2,
> test_eventdev_selftest_dpaa2);
> +REGISTER_TEST_COMMAND(eventdev_selftest_dlb2,
> test_eventdev_selftest_dlb2);
> +

Nit: newline at the end of the file

> diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
> index 43b85d7..620a0a5 100644
> --- a/drivers/event/dlb2/dlb2.c
> +++ b/drivers/event/dlb2/dlb2.c

[...]

> +/* destruction */
> +static inline void
> +cleanup(struct test *t __rte_unused)

Since 't' is unused, just drop the parameter.

With that and the whitespace issues fixed:
Reviewed-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

[dpdk-dev] [PATCH 20/22] event/dlb2: add queue and port release

[Timothy McDaniel]

These entry points are NO-OPS. DLB does not support
reconfiguring individual queues or ports. The entire device
must be reconfigured.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index 620a0a5..daf9b9f 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3756,6 +3756,29 @@ dlb2_eventdev_close(struct rte_eventdev *dev)
 }
 
 static void
+dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(id);
+
+	/* This function intentionally left blank. dlb2 does not support
+	 * reconfiguring individual queues or ports -- the entire device
+	 * must be reconfigured.
+	 */
+}
+
+static void
+dlb2_eventdev_port_release(void *port)
+{
+	RTE_SET_USED(port);
+
+	/* This function intentionally left blank. dlb2 does not support
+	 * reconfiguring individual queues or ports -- the entire device
+	 * must be reconfigured.
+	 */
+}
+
+static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
 	struct dlb2_eventdev *dlb2;
@@ -3769,8 +3792,10 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.dev_close        = dlb2_eventdev_close,
 		.queue_def_conf   = dlb2_eventdev_queue_default_conf_get,
 		.queue_setup      = dlb2_eventdev_queue_setup,
+		.queue_release    = dlb2_eventdev_queue_release,
 		.port_def_conf    = dlb2_eventdev_port_default_conf_get,
 		.port_setup       = dlb2_eventdev_port_setup,
+		.port_release     = dlb2_eventdev_port_release,
 		.port_link        = dlb2_eventdev_port_link,
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
-- 
2.6.4

Re: [dpdk-dev] [PATCH 20/22] event/dlb2: add queue and port release

[Eads, Gage]

>  static void
> +dlb2_eventdev_queue_release(struct rte_eventdev *dev, uint8_t id)
> +{
> +	RTE_SET_USED(dev);
> +	RTE_SET_USED(id);
> +
> +	/* This function intentionally left blank. dlb2 does not support
> +	 * reconfiguring individual queues or ports -- the entire device
> +	 * must be reconfigured.
> +	 */

I think the 2nd sentence in the comment need to be reconsidered (or just
removed). The release callbacks are called (for every port and queue) when
rte_event_dev_configure() is called a 2nd/3rd/etc. time to reconfigure the
device, and the callbacks give the PMD the opportunity clean-up the individual
port/queue. Implementing the logic can be useful for
entire-device-reconfiguration.

> +}
> +
> +static void
> +dlb2_eventdev_port_release(void *port)
> +{
> +	RTE_SET_USED(port);
> +
> +	/* This function intentionally left blank. dlb2 does not support
> +	 * reconfiguring individual queues or ports -- the entire device
> +	 * must be reconfigured.
> +	 */

This might be the appropriate place to free each port's memzone (see my
comments in patch 11).

Thanks,
Gage

[dpdk-dev] [PATCH 21/22] event/dlb2: add timeout ticks entry point

[Timothy McDaniel]

Adds the timeout ticks conversion function.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 drivers/event/dlb2/dlb2.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/event/dlb2/dlb2.c b/drivers/event/dlb2/dlb2.c
index daf9b9f..50fb4b9 100644
--- a/drivers/event/dlb2/dlb2.c
+++ b/drivers/event/dlb2/dlb2.c
@@ -3778,6 +3778,19 @@ dlb2_eventdev_port_release(void *port)
 	 */
 }
 
+static int
+dlb2_eventdev_timeout_ticks(struct rte_eventdev *dev, uint64_t ns,
+			    uint64_t *timeout_ticks)
+{
+	RTE_SET_USED(dev);
+	uint64_t cycles_per_ns;
+
+	cycles_per_ns = rte_get_timer_hz() / 1E9;
+	*timeout_ticks = ns * cycles_per_ns;
+
+	return 0;
+}
+
 static void
 dlb2_entry_points_init(struct rte_eventdev *dev)
 {
@@ -3800,6 +3813,7 @@ dlb2_entry_points_init(struct rte_eventdev *dev)
 		.port_unlink      = dlb2_eventdev_port_unlink,
 		.port_unlinks_in_progress =
 				    dlb2_eventdev_port_unlinks_in_progress,
+		.timeout_ticks    = dlb2_eventdev_timeout_ticks,
 		.dump             = dlb2_eventdev_dump,
 		.xstats_get       = dlb2_eventdev_xstats_get,
 		.xstats_get_names = dlb2_eventdev_xstats_get_names,
-- 
2.6.4

Re: [dpdk-dev] [PATCH 21/22] event/dlb2: add timeout ticks entry point

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 21/22] event/dlb2: add timeout ticks entry point
> 
> Adds the timeout ticks conversion function.
> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

Reviewed-by: Gage Eads <gage.eads@intel.com>

[dpdk-dev] [PATCH 22/22] doc: add new DLB2 eventdev driver to relnotes

[Timothy McDaniel]

Added announcement of availabililty for the new driver
for Intel Dynamic Load Balancer V2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
---
 doc/guides/rel_notes/release_20_11.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst
index df227a1..f80df21 100644
--- a/doc/guides/rel_notes/release_20_11.rst
+++ b/doc/guides/rel_notes/release_20_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
+
+  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
+  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.
+
 
 Removed Items
 -------------
-- 
2.6.4

Re: [dpdk-dev] [PATCH 22/22] doc: add new DLB2 eventdev driver to relnotes

[Eads, Gage]

> -----Original Message-----
> From: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Sent: Friday, September 11, 2020 3:26 PM
> To: Mcnamara, John <john.mcnamara@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>
> Cc: dev@dpdk.org; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads, Gage
> <gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>;
> jerinj@marvell.com
> Subject: [PATCH 22/22] doc: add new DLB2 eventdev driver to relnotes
> 
> Added announcement of availabililty for the new driver
> for Intel Dynamic Load Balancer V2.0 hardware.

There's some inconsistency in the version numbering (2.0, V2.0, v2.0) in this series 
-- I believe 2.0 is the marketing-approved one.

> 
> Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
> ---
>  doc/guides/rel_notes/release_20_11.rst | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/doc/guides/rel_notes/release_20_11.rst
> b/doc/guides/rel_notes/release_20_11.rst
> index df227a1..f80df21 100644
> --- a/doc/guides/rel_notes/release_20_11.rst
> +++ b/doc/guides/rel_notes/release_20_11.rst
> @@ -55,6 +55,11 @@ New Features
>       Also, make sure to start the actual text at the margin.
>       =======================================================
> 
> +* **Added a new driver for the Intel Dynamic Load Balancer v2.0 device.**
> +
> +  Added the new ``dlb2`` eventdev driver for the Intel DLB V2.0 device. See the
> +  :doc:`../eventdevs/dlb2` eventdev guide for more details on this new driver.

Looks like the guide didn't make it into this series ;)

Please also add an entry MAINTAINERS entry for the PMD.

Thanks,
Gage

Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD

[Jerin Jacob]

On Sat, Sep 12, 2020 at 2:00 AM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> The following patch series adds support for a new eventdev PMD. The DLB2
> PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
> hardware.
> The DLB2 is a PCIe device that provides load-balanced, prioritized
> scheduling of core-to-core communication. The device consists of
> queues and arbiters that connect producer and consumer cores, and
> implements load-balanced queueing features including:
> - Lock-free multi-producer/multi-consumer operation.
> - Multiple priority levels for varying traffic types.
> - 'Direct' traffic (i.e. multi-producer/single-consumer)
> - Simple unordered load-balanced distribution.
> - Atomic lock-free load balancing across multiple consumers.
> - Queue element reordering feature allowing ordered load-balanced
>   distribution.
>
> The DLB2 hardware supports both load balanced and directed ports and
> queues. Unlike other eventdev devices already in the repo,  not all
> DLB2 ports and queues are equally capable. In particular, directed
> ports are limited to a single link, and must be connected to a
> directed queue. Additionally, even though LDB ports may link multiple queues,
> the number of queues that may be linked is limited by hardware.
>
> While reviewing the code, please be aware that this PMD has full
> control over the DLB2 hardware. Intel will be extending the DLB2 PMD
> in the future (not as part of this first series) with a mode that we
> refer to as the bifurcated PMD. The bifurcated PMD communicates with a
> kernel driver to configure the device, ports, and queues, and memory
> maps device MMIO so datapath operations occur purely in user-space.
> Note that the DLB2 hardware is a successor of the DLB hardware, and
> as such is structured similarly, both in terms of code layout and
> implementation.
>
> The framework to support both the PF PMD and bifurcated PMD exists in
> this patchset, and is why the iface.[ch] layer is present.
>
> Depends-on: patch-77466 ("eventdev: add PCI probe named convenience function")
> Depends-on: series-12160 ("Eventdev ABI changes")
> Depends-on: patch-77460 ("eal: add umonitor umwait to x86 cpuflags")
>
> Timothy McDaniel (22):


There is two version of driver(DLB and DLB2)[1]
Both are needed?
Is two separate set of HW?

http://patches.dpdk.org/project/dpdk/list/?submitter=826


>   event/dlb2: add meson build infrastructure
>   event/dlb2: add dynamic logging
>   event/dlb2: add private data structures and constants
>   event/dlb2: add definitions shared with LKM or shared code
>   event/dlb2: add inline functions
>   event/dlb2: add probe
>   event/dlb2: add xstats
>   event/dlb2: add infos get and configure
>   event/dlb2: add queue and port default conf
>   event/dlb2: add queue setup
>   event/dlb2: add port setup
>   event/dlb2: add port link
>   event/dlb2: add port unlink and port unlinks in progress
>   event/dlb2: add eventdev start
>   event/dlb2: add enqueue and its burst variants
>   event/dlb2: add dequeue and its burst variants
>   event/dlb2: add eventdev stop and close
>   event/dlb2: add PMD's token pop public interface
>   event/dlb2: add PMD self-tests
>   event/dlb2: add queue and port release
>   event/dlb2: add timeout ticks entry point
>   doc: add new DLB2 eventdev driver to relnotes
>
>  app/test/test_eventdev.c                          |    9 +
>  config/rte_config.h                               |    7 +
>  doc/guides/rel_notes/release_20_11.rst            |    5 +
>  drivers/event/dlb2/dlb2.c                         | 4046 ++++++++++++++
>  drivers/event/dlb2/dlb2_iface.c                   |   88 +
>  drivers/event/dlb2/dlb2_iface.h                   |   75 +
>  drivers/event/dlb2/dlb2_inline_fns.h              |   85 +
>  drivers/event/dlb2/dlb2_log.h                     |   25 +
>  drivers/event/dlb2/dlb2_priv.h                    |  619 +++
>  drivers/event/dlb2/dlb2_selftest.c                | 1570 ++++++
>  drivers/event/dlb2/dlb2_user.h                    |  883 +++
>  drivers/event/dlb2/dlb2_xstats.c                  | 1269 +++++
>  drivers/event/dlb2/meson.build                    |   16 +
>  drivers/event/dlb2/pf/base/dlb2_hw_types.h        |  367 ++
>  drivers/event/dlb2/pf/base/dlb2_mbox.h            |  596 ++
>  drivers/event/dlb2/pf/base/dlb2_osdep.h           |  248 +
>  drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h    |  447 ++
>  drivers/event/dlb2/pf/base/dlb2_osdep_list.h      |  131 +
>  drivers/event/dlb2/pf/base/dlb2_osdep_types.h     |   31 +
>  drivers/event/dlb2/pf/base/dlb2_regs.h            | 2527 +++++++++
>  drivers/event/dlb2/pf/base/dlb2_resource.c        | 6023 +++++++++++++++++++++
>  drivers/event/dlb2/pf/base/dlb2_resource.h        | 1913 +++++++
>  drivers/event/dlb2/pf/dlb2_main.c                 |  692 +++
>  drivers/event/dlb2/pf/dlb2_main.h                 |  107 +
>  drivers/event/dlb2/pf/dlb2_pf.c                   |  734 +++
>  drivers/event/dlb2/rte_pmd_dlb2.c                 |   39 +
>  drivers/event/dlb2/rte_pmd_dlb2.h                 |   59 +
>  drivers/event/dlb2/rte_pmd_dlb2_event_version.map |    9 +
>  drivers/event/meson.build                         |    4 +
>  29 files changed, 22624 insertions(+)
>  create mode 100644 drivers/event/dlb2/dlb2.c
>  create mode 100644 drivers/event/dlb2/dlb2_iface.c
>  create mode 100644 drivers/event/dlb2/dlb2_iface.h
>  create mode 100644 drivers/event/dlb2/dlb2_inline_fns.h
>  create mode 100644 drivers/event/dlb2/dlb2_log.h
>  create mode 100644 drivers/event/dlb2/dlb2_priv.h
>  create mode 100644 drivers/event/dlb2/dlb2_selftest.c
>  create mode 100644 drivers/event/dlb2/dlb2_user.h
>  create mode 100644 drivers/event/dlb2/dlb2_xstats.c
>  create mode 100644 drivers/event/dlb2/meson.build
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_hw_types.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_mbox.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_bitmap.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_list.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_osdep_types.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_regs.h
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.c
>  create mode 100644 drivers/event/dlb2/pf/base/dlb2_resource.h
>  create mode 100644 drivers/event/dlb2/pf/dlb2_main.c
>  create mode 100644 drivers/event/dlb2/pf/dlb2_main.h
>  create mode 100644 drivers/event/dlb2/pf/dlb2_pf.c
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.c
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2.h
>  create mode 100644 drivers/event/dlb2/rte_pmd_dlb2_event_version.map
>
> --
> 2.6.4
>

Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Monday, September 21, 2020 12:12 PM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD
> 
> On Sat, Sep 12, 2020 at 2:00 AM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > The following patch series adds support for a new eventdev PMD. The DLB2
> > PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
> > hardware.
> > The DLB2 is a PCIe device that provides load-balanced, prioritized
> > scheduling of core-to-core communication. The device consists of
> > queues and arbiters that connect producer and consumer cores, and
> > implements load-balanced queueing features including:
> > - Lock-free multi-producer/multi-consumer operation.
> > - Multiple priority levels for varying traffic types.
> > - 'Direct' traffic (i.e. multi-producer/single-consumer)
> > - Simple unordered load-balanced distribution.
> > - Atomic lock-free load balancing across multiple consumers.
> > - Queue element reordering feature allowing ordered load-balanced
> >   distribution.
> >
> > The DLB2 hardware supports both load balanced and directed ports and
> > queues. Unlike other eventdev devices already in the repo,  not all
> > DLB2 ports and queues are equally capable. In particular, directed
> > ports are limited to a single link, and must be connected to a
> > directed queue. Additionally, even though LDB ports may link multiple queues,
> > the number of queues that may be linked is limited by hardware.
> >
> > While reviewing the code, please be aware that this PMD has full
> > control over the DLB2 hardware. Intel will be extending the DLB2 PMD
> > in the future (not as part of this first series) with a mode that we
> > refer to as the bifurcated PMD. The bifurcated PMD communicates with a
> > kernel driver to configure the device, ports, and queues, and memory
> > maps device MMIO so datapath operations occur purely in user-space.
> > Note that the DLB2 hardware is a successor of the DLB hardware, and
> > as such is structured similarly, both in terms of code layout and
> > implementation.
> >
> > The framework to support both the PF PMD and bifurcated PMD exists in
> > this patchset, and is why the iface.[ch] layer is present.
> >
> > Depends-on: patch-77466 ("eventdev: add PCI probe named convenience
> function")
> > Depends-on: series-12160 ("Eventdev ABI changes")
> > Depends-on: patch-77460 ("eal: add umonitor umwait to x86 cpuflags")
> >
> > Timothy McDaniel (22):
> 
> 
> There is two version of driver(DLB and DLB2)[1]
> Both are needed?
> Is two separate set of HW?
> 

Yes, the two PMDs, DLB and DLB2 are both needed.  They support two different hardware devices.

 

Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD

[Jerin Jacob]

On Mon, Sep 21, 2020 at 10:45 PM McDaniel, Timothy
<timothy.mcdaniel@intel.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Jerin Jacob <jerinjacobk@gmail.com>
> > Sent: Monday, September 21, 2020 12:12 PM
> > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> > Gage <gage.eads@intel.com>; Van Haaren, Harry
> > <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>
> > Subject: Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD
> >
> > On Sat, Sep 12, 2020 at 2:00 AM Timothy McDaniel
> > <timothy.mcdaniel@intel.com> wrote:
> > >
>
> > >
> > > Timothy McDaniel (22):
> >
> >
> > There is two version of driver(DLB and DLB2)[1]
> > Both are needed?
> > Is two separate set of HW?
> >
>
> Yes, the two PMDs, DLB and DLB2 are both needed.  They support two different hardware devices.

There are plenty of build errors. Please see
http://mails.dpdk.org/archives/test-report/2020-September/152143.html

Also, I would like close the eventdev spec first before we merge this
driver as the driver is depended on
http://patches.dpdk.org/patch/77465/



>
>

Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Tuesday, September 29, 2020 1:41 PM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD
> 
> On Mon, Sep 21, 2020 at 10:45 PM McDaniel, Timothy
> <timothy.mcdaniel@intel.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Jerin Jacob <jerinjacobk@gmail.com>
> > > Sent: Monday, September 21, 2020 12:12 PM
> > > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > > Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>;
> Eads,
> > > Gage <gage.eads@intel.com>; Van Haaren, Harry
> > > <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>
> > > Subject: Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD
> > >
> > > On Sat, Sep 12, 2020 at 2:00 AM Timothy McDaniel
> > > <timothy.mcdaniel@intel.com> wrote:
> > > >
> >
> > > >
> > > > Timothy McDaniel (22):
> > >
> > >
> > > There is two version of driver(DLB and DLB2)[1]
> > > Both are needed?
> > > Is two separate set of HW?
> > >
> >
> > Yes, the two PMDs, DLB and DLB2 are both needed.  They support two
> different hardware devices.
> 
> There are plenty of build errors. Please see
> http://mails.dpdk.org/archives/test-report/2020-September/152143.html
> 
> Also, I would like close the eventdev spec first before we merge this
> driver as the driver is depended on
> http://patches.dpdk.org/patch/77465/
> 
Thank you, Jerin.

I am working on addressing the build breaks and your other review comments. I hope to upload those changes tomorrow,
and that will include the performance comparison that you requested.

Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Tuesday, September 29, 2020 1:41 PM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <Erik.G.Carrillo@intel.com>;
> Eads, Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD
> 
> On Mon, Sep 21, 2020 at 10:45 PM McDaniel, Timothy
> <timothy.mcdaniel@intel.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Jerin Jacob <jerinjacobk@gmail.com>
> > > Sent: Monday, September 21, 2020 12:12 PM
> > > To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> > > Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>;
> Eads,
> > > Gage <gage.eads@intel.com>; Van Haaren, Harry
> > > <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>
> > > Subject: Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD
> > >
> > > On Sat, Sep 12, 2020 at 2:00 AM Timothy McDaniel
> > > <timothy.mcdaniel@intel.com> wrote:
> > > >
> >
> > > >
> > > > Timothy McDaniel (22):
> > >
> > >
> > > There is two version of driver(DLB and DLB2)[1]
> > > Both are needed?
> > > Is two separate set of HW?
> > >
> >
> > Yes, the two PMDs, DLB and DLB2 are both needed.  They support two
> different hardware devices.
> 
> There are plenty of build errors. Please see
> http://mails.dpdk.org/archives/test-report/2020-September/152143.html
> 

The build errors appear to be a result of the depends-on patches not having been applied.
Also, note that there is a 2-way dependency on the ABI changes patch and the test/apps patch.
They both depend on each other.

Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD

[Jerin Jacob]

On Sat, Sep 12, 2020 at 2:00 AM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> The following patch series adds support for a new eventdev PMD. The DLB2
> PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
> hardware.
> The DLB2 is a PCIe device that provides load-balanced, prioritized
> scheduling of core-to-core communication. The device consists of
> queues and arbiters that connect producer and consumer cores, and
> implements load-balanced queueing features including:
> - Lock-free multi-producer/multi-consumer operation.
> - Multiple priority levels for varying traffic types.
> - 'Direct' traffic (i.e. multi-producer/single-consumer)
> - Simple unordered load-balanced distribution.
> - Atomic lock-free load balancing across multiple consumers.
> - Queue element reordering feature allowing ordered load-balanced
>   distribution.
>
> The DLB2 hardware supports both load balanced and directed ports and
> queues. Unlike other eventdev devices already in the repo,  not all
> DLB2 ports and queues are equally capable. In particular, directed
> ports are limited to a single link, and must be connected to a
> directed queue. Additionally, even though LDB ports may link multiple queues,
> the number of queues that may be linked is limited by hardware.
>
> While reviewing the code, please be aware that this PMD has full
> control over the DLB2 hardware. Intel will be extending the DLB2 PMD
> in the future (not as part of this first series) with a mode that we
> refer to as the bifurcated PMD. The bifurcated PMD communicates with a
> kernel driver to configure the device, ports, and queues, and memory
> maps device MMIO so datapath operations occur purely in user-space.
> Note that the DLB2 hardware is a successor of the DLB hardware, and
> as such is structured similarly, both in terms of code layout and
> implementation.
>
> The framework to support both the PF PMD and bifurcated PMD exists in
> this patchset, and is why the iface.[ch] layer is present.
>
> Depends-on: patch-77466 ("eventdev: add PCI probe named convenience function")
> Depends-on: series-12160 ("Eventdev ABI changes")
> Depends-on: patch-77460 ("eal: add umonitor umwait to x86 cpuflags")

Cc: Thomas and David

There is a duplicate for patch-77460 as
https://patches.dpdk.org/patch/76554/ from different authors,
and both are open comments to be closed.
@McDaniel, Timothy and @Liang Ma , We need to have this eal patch on
master before I merge this
patch series in eventdev-next. Please check.


>

Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD

[McDaniel, Timothy]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Tuesday, September 29, 2020 1:46 PM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; David Marchand <david.marchand@redhat.com>;
> Ma, Liang J <liang.j.ma@intel.com>
> Cc: dpdk-dev <dev@dpdk.org>; Carrillo, Erik G <Erik.G.Carrillo@intel.com>;
> Eads, Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob <jerinj@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH 00/22] Add DLB2 PMD
> 
> On Sat, Sep 12, 2020 at 2:00 AM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > The following patch series adds support for a new eventdev PMD. The DLB2
> > PMD adds support for the Intel Dynamic Load Balancer 2.0 (DLB2)
> > hardware.
> > The DLB2 is a PCIe device that provides load-balanced, prioritized
> > scheduling of core-to-core communication. The device consists of
> > queues and arbiters that connect producer and consumer cores, and
> > implements load-balanced queueing features including:
> > - Lock-free multi-producer/multi-consumer operation.
> > - Multiple priority levels for varying traffic types.
> > - 'Direct' traffic (i.e. multi-producer/single-consumer)
> > - Simple unordered load-balanced distribution.
> > - Atomic lock-free load balancing across multiple consumers.
> > - Queue element reordering feature allowing ordered load-balanced
> >   distribution.
> >
> > The DLB2 hardware supports both load balanced and directed ports and
> > queues. Unlike other eventdev devices already in the repo,  not all
> > DLB2 ports and queues are equally capable. In particular, directed
> > ports are limited to a single link, and must be connected to a
> > directed queue. Additionally, even though LDB ports may link multiple queues,
> > the number of queues that may be linked is limited by hardware.
> >
> > While reviewing the code, please be aware that this PMD has full
> > control over the DLB2 hardware. Intel will be extending the DLB2 PMD
> > in the future (not as part of this first series) with a mode that we
> > refer to as the bifurcated PMD. The bifurcated PMD communicates with a
> > kernel driver to configure the device, ports, and queues, and memory
> > maps device MMIO so datapath operations occur purely in user-space.
> > Note that the DLB2 hardware is a successor of the DLB hardware, and
> > as such is structured similarly, both in terms of code layout and
> > implementation.
> >
> > The framework to support both the PF PMD and bifurcated PMD exists in
> > this patchset, and is why the iface.[ch] layer is present.
> >
> > Depends-on: patch-77466 ("eventdev: add PCI probe named convenience
> function")
> > Depends-on: series-12160 ("Eventdev ABI changes")
> > Depends-on: patch-77460 ("eal: add umonitor umwait to x86 cpuflags")
> 
> Cc: Thomas and David
> 
> There is a duplicate for patch-77460 as
> https://patches.dpdk.org/patch/76554/ from different authors,
> and both are open comments to be closed.
> @McDaniel, Timothy and @Liang Ma , We need to have this eal patch on
> master before I merge this
> patch series in eventdev-next. Please check.
> 
> 

We have discussed this duplication with @Liang Ma, who split out just the
bit definitions into its own patch. DLB/DLB2 have been updated to use his new patch going forward,
but will duplicate it here until it is merged.